Skip to content

ENH: Free-Threading (No-GIL) Support for Python 3.13+ #749

Open
Open
ENH: Free-Threading (No-GIL) Support for Python 3.13+#749

Description

@HaoZeke
HaoZeke
opened on May 26, 2025

xref #670

I took a quick look, and I'm not wholly sure on what parts of the locking, possibly synchronizing access to the c-bloc library. Worst case scenario, maybe each of the codecs need to be checked for support.

The standard TODOs for adding free-threading support are in two phases, the first of which is a baseline consisting of the following tasks:

  • Run the test suite with pytest-run-parallel to find potential issues, and fix them.
  • Run the test suite under ThreadSanitizer. If possible, depends on how many dependencies there are and if they run under TSan.
  • Add cp313t-* to CI to build free-threading wheels.

The next phase is a bit more invasive, and it involves a single task:

For more details, please see the suggested plan of attack in the py-free-threading guide. The two phase approach is discussed in explosion/cymem#47 (comment) and gorakhargosh/watchdog#1105 (comment).

Note that this is the first time I've looked at this repo, so I might be
missing known issues or code that needs closer inspection. Any suggestions here
will be very useful. I'll start with the build without warning PR as also discussed in #670.

Let me know if I ought to hold off for any reason. (including if @tomwhite is working on it).

Activity

  1. HaoZeke commented on May 26, 2025

    @HaoZeke
    HaoZeke
    on May 26, 2025
    Author

    One of the issues is zfpy

    DEBUG No compatible version found for: numcodecs[zfpy]
  × No solution found when resolving dependencies:
  ╰─▶ Because only the following versions of zfpy are available:
          zfpy<=1.0.0
          zfpy==1.0.0.post1
          zfpy==1.0.0.post3
          zfpy==1.0.1
      and zfpy==1.0.0 has no wheels with a matching Python implementation tag (e.g., `cp313`), we can conclude that zfpy>=1.0.0,<1.0.0.post1 cannot be used.
      And because zfpy>=1.0.0.post1,<=1.0.0.post3 was yanked (reason: arm64 wheels still broken), we can conclude that zfpy>=1.0.0.post1,<=1.0.0.post3 cannot
      be used.
      And because zfpy==1.0.1 has no wheels with a matching Python ABI tag (e.g., `cp313t`) and numcodecs[zfpy]==0.16.2.dev0+dirty depends on zfpy>=1.0.0, we
      can conclude that numcodecs[zfpy]==0.16.2.dev0+dirty cannot be used.
      And because only numcodecs[zfpy]==0.16.2.dev0+dirty is available and you require numcodecs[zfpy], we can conclude that your requirements are
      unsatisfiable.

      hint: You require CPython 3.13 (`cp313t`), but we only found wheels for `zfpy` (v1.0.1) with the following Python ABI tags: `cp39`, `cp310`, `cp311`,
      `cp312`, `cp313`

      hint: You require CPython 3.13 (`cp313`), but we only found wheels for `zfpy` (v1.0.0) with the following Python implementation tags: `cp37`, `cp38`,
      `cp39`, `cp310`, `cp311`, `cp312`

  2. HaoZeke commented on May 26, 2025

    @HaoZeke
    HaoZeke
    on May 26, 2025
    Author

    pcodec also seems to cause immediate segfaults...

    https://github.com/HaoZeke/numcodecs/actions/runs/15246098627/job/42873119696

    Test run 
    ============================= test session starts ==============================
platform darwin -- Python 3.13.3, pytest-8.3.5, pluggy-1.6.0 -- /Users/runner/work/numcodecs/numcodecs/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /Users/runner/work/numcodecs/numcodecs
configfile: pyproject.toml
testpaths: numcodecs/tests
plugins: timeout-2.4.0, run-parallel-0.4.2, cov-6.1.1
timeout: 600.0s
timeout method: signal
timeout func_only: False
Fatal Python error: Segmentation fault

Current thread 0x00000001e8d30f80 (most recent call first):
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1320 in create_module
  File "<frozen importlib._bootstrap>", line 813 in module_from_spec
  File "<frozen importlib._bootstrap>", line 921 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pcodec/__init__.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1026 in exec_module
collecting ... 
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/Users/runner/work/numcodecs/numcodecs/numcodecs/pcodec.py", line 5 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1026 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/Users/runner/work/numcodecs/numcodecs/numcodecs/__init__.py", line 144 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1026 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1310 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1310 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "/Users/runner/.local/share/uv/python/cpython-3.13.3+freethreaded-macos-aarch64-none/lib/python3.13t/importlib/__init__.py", line 88 in import_module
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/pathlib.py", line 587 in import_path
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 493 in importtestmodule
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 546 in _getobj
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 284 in obj
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 562 in _register_setup_module_fixture
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 549 in collect
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 389 in collect
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 341 in from_call
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 391 in pytest_make_collect_report
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 567 in collect_one_node
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 835 in _collect_one_node
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 970 in genitems
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 975 in genitems
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 809 in perform_collect
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 347 in pytest_collection
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 336 in _main
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/Users/runner/work/numcodecs/numcodecs/.venv/lib/python3.13t/site-packages/pytest/__main__.py", line 9 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numcodecs.compat_ext, numcodecs.blosc, numcodecs.zstd, numcodecs.lz4, numcodecs._shuffle, numcodecs.jenkins, crc32c._crc32c, numcodecs.vlen, numcodecs.fletcher32 (total: [11](https://github.com/HaoZeke/numcodecs/actions/runs/15246098627/job/42873119699#step:12:12))
Error: Process completed with exit code 139.

  3. HaoZeke commented on May 26, 2025

    @HaoZeke
    HaoZeke
    on May 26, 2025
    Author

    pcodec is an optional dependency, but the current blocker is blosc.

    https://github.com/HaoZeke/numcodecs/actions/runs/15246191211/job/42873362395

    Failure Log 
    numcodecs/__init__.py:43: in <module>
    from numcodecs import blosc
E   RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'numcodecs.blosc', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
________________ ERROR collecting numcodecs/tests/test_zarr3.py ________________
numcodecs/tests/test_zarr3.py:14: in <module>
    zarr = pytest.importorskip("zarr")
.venv/lib/python3.13t/site-packages/zarr/__init__.py:2: in <module>
    from zarr.api.synchronous import (
.venv/lib/python3.13t/site-packages/zarr/api/synchronous.py:7: in <module>
    import zarr.api.asynchronous as async_api
.venv/lib/python3.13t/site-packages/zarr/api/asynchronous.py:12: in <module>
    from zarr.core.array import (
.venv/lib/python3.13t/site-packages/zarr/core/__init__.py:9: in <module>
    from zarr.core.codec_pipeline import BatchedCodecPipeline  # noqa: F401
.venv/lib/python3.13t/site-packages/zarr/core/codec_pipeline.py:20: in <module>
    from zarr.core.metadata.v2 import _default_fill_value
.venv/lib/python3.13t/site-packages/zarr/core/metadata/__init__.py:3: in <module>
    from .v2 import ArrayV2Metadata, ArrayV2MetadataDict
.venv/lib/python3.13t/site-packages/zarr/core/metadata/v2.py:47: in <module>
    CompressorLikev2: TypeAlias = dict[str, JSON] | numcodecs.abc.Codec | None
E   AttributeError: module 'numcodecs' has no attribute 'abc'
************************** pytest-run-parallel report **************************
All tests were run in parallel! 🎉
=========================== short test summary info ============================
ERROR numcodecs/tests/test_astype.py - RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'numcodecs.blosc', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
ERROR numcodecs/tests/test_zarr3.py - AttributeError: module 'numcodecs' has no attribute 'abc'
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!

  4. tomwhite commented on May 27, 2025

    @tomwhite
    tomwhite
    on May 27, 2025
    Member

    Thanks for looking at this @HaoZeke! I'm not working on this, so I'm happy for you to continue.

    I think while investigating it's OK to force the GIL to be diabled. This would allow testing with Blosc.

    Building free-threaded wheels would be good to get working too, as it would allow people to experiment with free-threaded Python on Zarr and the wider stack. Now that Cython 3.1 is out, it should be straightforward to change the Numcodecs wheels build to include cp313t.

  5. crusaderky commented on Apr 2, 2026

    @crusaderky
    crusaderky
    on Apr 2, 2026

    Any update on this?
    This is effectively blocking Dask Arrays as most users rely on Zarr for I/O.

  6. d-v-b commented on Apr 2, 2026

    @d-v-b
    d-v-b
    on Apr 2, 2026
    Contributor

    hi @crusaderky I don't know if anyone is presently working on this. I'd be happy to chat more about the broader numcodecs maintenance (or lack thereof) situation, but in the context of this issue here, I would be happy to review a PR, but I don't have cycles to submit one.

  7. kumaraditya303 commented on Sep 9, 2026

    @kumaraditya303
    kumaraditya303
    on Sep 9, 2026
    Contributor

    @d-v-b I created #858 to add free-threading support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions