Repository navigation
FATURE REQUEST: allowing reinit PythonCall #700
Description
Is your feature request related to a problem? Please describe.
While writing tests for Zarr.jl, it's found that I cannot test Zarr.jl with both python-zar v2 and python-zarr v3.
It is caused that PythonCall has monolithic python session and they cannot be restarted after using PythonCall:
using CondaPkg
using PythonCall
# Use Zarr 3.x to test Zarr.jl
CondaPkg.add("zarr", "3.*")
zarr = pyimport("zarr")
CondaPkg.withenv() do
run(`python -c "import zarr; print(zarr.__version__)"`)
end # 3.1.3
println(zarr.__version__) # 3.1.3
# testing...
# Use Zarr 2.x to test Zarr.jl
CondaPkg.add("zarr", "2.*")
zarr = pyimport("zarr")
CondaPkg.withenv() do
run(`python -c "import zarr; print(zarr.__version__)"`)
end # 2.18.7
println(zarr.__version__) # 3.1.3
# testing failed due to importlib.reload does not useful because the v2 and v3 has different API so it returns error:
julia> importlib = pyimport("importlib")
Python: <module 'importlib' from '/home/.../.../Zarr.jl/test/.CondaPkg/.pixi/envs/default/lib/python3.13/importlib/__init__.py'>
julia> importlib.reload(zarr) # try v2 -> v3
ERROR: Python: ModuleNotFoundError: No module named 'zarr.core.array'; 'zarr.core' is not a package
Python stacktrace:
julia> importlib.reload(zarr) # try v3 -> v2
ERROR: Python: ImportError: cannot import name 'MetadataError' from 'zarr.errors' (/home/.../.../Zarr.jl/test/.CondaPkg/.pixi/envs/default/lib/python3.13/site-packages/zarr/errors.py)
Python stacktrace:Describe the solution you'd like
Supporting restart of python session will be helpful.
CondaPkg.add("zarr", "2.*")
zarr = pysession.pyimport("zarr")
# testing with zarr v2
PythonCall.reset()
CondaPkg.add("zarr", "3.*")
zarr = pysession.pyimport("zarr")
# testing with zarr v3
Describe alternatives you've considered
Manual starting and ending of python sesssion will also be useful. I'm thinking of
CondaPkg.add("zarr", "2.*")
PythonCall.start() do pysession
zarr = pysession.pyimport("zarr")
# do the rest
end
CondaPkg.add("zarr", "3.*")
PythonCall.start() do pysession
zarr = pysession.pyimport("zarr")
# do the rest
end
Activity
cjdoris commentedon Oct 19, 2025
I don't think it is really possible to do this. We'd have to be able to guarantee that every single Python object was freed by Julia's GC when the interpreter is restarted. And this is basically impossible to know because there will be Python objects cached in all sorts of places.
Anyway it's a bit of an XY problem. Wanting to perform integration tests with multiple versions of a package is quite common, and the usual way to do this is to run your test suite multiple times in different environments with different packages installed in them. You can branch on the package version in the test suite, say if there are some tests that are only relevant for zarr v3.
ehgus commentedon Oct 20, 2025
Thank you for the explanation about the GC issue.
Yeah, this is more of an XY problem. I will look for another solution following your suggestion.