Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/ci-i386.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ jobs:
- name: Checkout source
uses: actions/checkout@v6
with:
submodules: recursive
fetch-depth: 0 # required for version resolution

- name: Setup Alpine Linux environment
Expand Down
3 changes: 0 additions & 3 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ jobs:
- name: Checkout source
uses: actions/checkout@v6
with:
submodules: recursive
fetch-depth: 0 # required for version resolution

- name: Set up Python
Expand Down Expand Up @@ -75,7 +74,6 @@ jobs:
- name: Checkout source
uses: actions/checkout@v6
with:
submodules: recursive
fetch-depth: 0

- name: Set up Python
Expand Down Expand Up @@ -125,7 +123,6 @@ jobs:
- name: Checkout source
uses: actions/checkout@v6
with:
submodules: recursive
fetch-depth: 0 # required for version resolution

- name: Set up Python
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/wheel.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ jobs:
steps:
- uses: actions/checkout@v6
with:
submodules: true
fetch-depth: 0 # required for version resolution for nightly wheels

- uses: pypa/cibuildwheel@v3.4.1
Expand All @@ -48,7 +47,6 @@ jobs:
steps:
- uses: actions/checkout@v6
with:
submodules: true
fetch-depth: 0

- uses: actions/setup-python@v6
Expand Down
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -99,4 +99,8 @@ ENV/
.pixi/*
*.egg-info
pixi.lock
uv.lock
uv.lock
# meson subprojects: only wraps and our overlays are tracked
/subprojects/*
!/subprojects/*.wrap
!/subprojects/packagefiles/
3 changes: 0 additions & 3 deletions .gitmodules

This file was deleted.

3 changes: 0 additions & 3 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
version: 2

submodules:
include: all

build:
os: ubuntu-24.04
tools:
Expand Down
21 changes: 21 additions & 0 deletions LICENSES/BITSHUFFLE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
Bitshuffle - Filter for improving compression of typed binary data.

Copyright (c) 2014 Kiyoshi Masui (kiyo@physics.ubc.ca)

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
31 changes: 31 additions & 0 deletions LICENSES/BLOSC.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
BSD License

For Blosc - A blocking, shuffling and lossless compression library

Copyright (c) 2009-2018 Francesc Alted <francesc@blosc.org>
Copyright (c) 2019-present Blosc Development Team <blosc@blosc.org>

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

* Neither the name Francesc Alted nor the names of its contributors may be used
to endorse or promote products derived from this software without specific
prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
20 changes: 20 additions & 0 deletions LICENSES/FASTLZ.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
FastLZ - Byte-aligned LZ77 compression library
Copyright (C) 2005-2020 Ariya Hidayat <ariya.hidayat@gmail.com>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
24 changes: 24 additions & 0 deletions LICENSES/LZ4.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
LZ4 Library
Copyright (c) 2011-2020, Yann Collet
All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
22 changes: 22 additions & 0 deletions LICENSES/ZLIB.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
Copyright notice:

(C) 1995-2026 Jean-loup Gailly and Mark Adler

This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any damages
arising from the use of this software.

Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:

1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this software
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.

Jean-loup Gailly Mark Adler
jloup@gzip.org madler@alumni.caltech.edu
30 changes: 30 additions & 0 deletions LICENSES/ZSTD.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
BSD License

For Zstandard software

Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

* Neither the name Facebook, nor Meta, nor the names of its contributors may
be used to endorse or promote products derived from this software without
specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
1 change: 0 additions & 1 deletion c-blosc
Submodule c-blosc deleted from 04c06f
15 changes: 7 additions & 8 deletions docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,13 @@ You will need your own fork to work on the code. Go to the link above and hit
the "Fork" button. Then clone your fork to your local machine:

```
$ git clone --recursive git@github.com:your-user-name/numcodecs.git # with ssh
$ git clone git@github.com:your-user-name/numcodecs.git # with ssh
```

or:

```
$ git clone --recursive https://github.com/your-user-name/numcodecs.git # with https
$ git clone https://github.com/your-user-name/numcodecs.git # with https
```

Then `cd` into the clone and add the `upstream` remote:
Expand All @@ -78,12 +78,11 @@ $ cd numcodecs
$ git remote add upstream https://github.com/zarr-developers/numcodecs.git
```

Note the ``--recursive`` flag is required to clone the ``c-blosc`` git submodule. If you
forgot it, you can initialize it later with:

```
$ git submodule update --init --recursive
```
The C libraries numcodecs links against (c-blosc, zstd, lz4, zlib) are not stored in the
repository. They are pinned in `subprojects/*.wrap` and downloaded by meson the first time
you build, so the first build needs network access. To fetch them ahead of time, run
`meson subprojects download`. Numcodecs-specific build files and patches for c-blosc live in
`subprojects/packagefiles/c-blosc/`.

### Creating a development environment

Expand Down
5 changes: 1 addition & 4 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,14 +58,11 @@ To work with Numcodecs source code in development, see the
development environment with venv or uv.

```
$ git clone --recursive https://github.com/zarr-developers/numcodecs.git
$ git clone https://github.com/zarr-developers/numcodecs.git
$ cd numcodecs
$ pip install -e .[test,msgpack,zfpy]
```

Note: if you prefer to use the GitHub CLI `gh` you will need to append `-- --recurse-submodules`
to the clone command to everything works properly.

To verify that Numcodecs has been fully installed (including the Blosc
extension) run the test suite:

Expand Down
24 changes: 24 additions & 0 deletions docs/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,30 @@

## Unreleased

### Enhancements

* Blosc byte-shuffle and unshuffle now use NEON SIMD on aarch64 (Apple Silicon, AWS
Graviton, etc). c-blosc 1.x only ships SSE2/AVX2 kernels, so on ARM it previously fell
back to a scalar loop that dominated Blosc runtime. The NEON kernels are taken from
c-blosc2 and produce byte-identical output. On an Apple M5, decoding 1 MiB lz4+shuffle
chunks is ~5-6x faster and encoding ~2x faster. Controlled by the new `neon` meson
option (default `auto`).
By {user}`Davis Bennett <d-v-b>`.

### Maintenance

* **Replace the c-blosc git submodule with meson wrap subprojects.** zstd, lz4 and zlib
now come from [WrapDB](https://mesonbuild.com/Wrapdb-projects.html) and are pinned
independently of c-blosc (zstd 1.5.6 -> 1.5.7, zlib 1.3.1 -> 1.3.2). c-blosc is pinned
in `subprojects/c-blosc.wrap` to a trimmed source tarball generated by
`tools/make_c_blosc_tarball.py`, with its meson build and numcodecs patches in
`subprojects/packagefiles/c-blosc/`. Cloning no longer needs `--recursive`; the first
build downloads the sources, and the sdist includes them so it still builds offline.
Building from source now also requires a C++ compiler (zstd's meson build declares
C++). The `system_*` meson options are unchanged. License texts for the bundled
libraries, including the previously missing zstd license, are in `LICENSES/`.
By {user}`Davis Bennett <d-v-b>`.

(release_0.17.0)=

* Allow building against a system Zlib. Note, this is only required if blosc is still
Expand Down
2 changes: 2 additions & 0 deletions meson.options
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,5 @@ option('sse2', type: 'feature', value: 'auto',
description: 'Enable SSE2 SIMD optimizations')
option('avx2', type: 'feature', value: 'auto',
description: 'Enable AVX2 SIMD optimizations')
option('neon', type: 'feature', value: 'auto',
description: 'Enable NEON SIMD shuffle on aarch64')
10 changes: 8 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,7 @@ maintainers = [
license = "MIT"
license-files = [
"LICENSE.txt",
"c-blosc/LICENSE.txt",
"c-blosc/LICENSES/*",
"LICENSES/*",
]

[project.urls]
Expand Down Expand Up @@ -238,6 +237,13 @@ warn_redundant_casts = true
warn_unused_ignores = true
warn_unused_configs = true

[tool.meson-python.args]
# C libraries are built as static subprojects and linked into the extension
# modules; don't install their headers, static libraries or pkg-config files.
install = ["--skip-subprojects"]
# Ship the subproject sources in the sdist so it builds without network access.
dist = ["--include-subprojects"]

[tool.uv]
# meson-python editable installs require build deps (ninja, meson) at runtime for
# auto-rebuild on import. Skip build isolation so the venv's ninja is used.
Expand Down
Loading
Loading