venv base path does not resolve symlinks using realpath()

It seems there is a bug in venv (and a similar one in virtualenv) where the "base" path in pyvenv.cfg is set incorrectly. If Python is installed in a non-standard folder, e.g. /usr/local/python-X.Y.Z and then symlinked into /usr/local/bin/python3, the venv package does not work correctly. I believe the source of the trouble is the setting of "home" variable. Specifically, this line:

dirname, exename = os.path.split(os.path.abspath(executable))

The dirname result is used to set "home". If the executable path is /usr/local/bin/python3 (actually a symlink to /usr/local/python-X.Y.Z/bin/python3), the "home" should not be set to /usr/local. Changing the above line (in the ensure_directories()) function to:

dirname, exename = os.path.split(os.path.realpath(executable))

This fixes the problem. I believe this is consistent with what the getpath.py module in Python does.

I noticed with problem when running the most recent Debian OS, version 12. It includes Python 3.11 and therefore /usr/lib/python3.11 exists. With the above bug, the venv tries to use /usr/lib/python3.11 as the sys.path. Importing the struct module fails with the mysterious error:

>>> import struct
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/struct.py", line 13, in <module>
    from _struct import *
ModuleNotFoundError: No module named '_struct'

Linked PRs

In order not to break usage involving symlinks to a python install tree, rather than resolving the realpath, only executable symlinks can be checked. xref pypa/virtualenv#2770 (comment)

pythongh-106045: Fix ``venv`` creation from a python executable symlink

@FFY00

To summarize, the error only happens (editing this to 3.13 to be actionable on main) when you have a custom built python that is linked to exactly /usr/local/bin/python3, but there's also a system python of the same version installed in /usr installed that created /usr/lib/python3.13/lib-dynload

Yes, I've been tracking this down, and it boils down to sys.base_exec_prefix being wrong. In turn, there are docs at

cpython/Doc/c-api/init.rst

Line 653 in 6ea04da

Return the *exec-prefix* for installed platform-*dependent* files. This is

which state:

Return the exec-prefix for installed platform-dependent files. This is
derived through a number of complicated rules from the program name set with
:c:member:PyConfig.program_name and some environment variables;

I note also that setting home in pyvenv.cfg to f'{sys.base_prefix}/bin (instead of dirname(sys.executable)) also works-around this issue (IMO this is a little "less-wrong" than resolving the symlink to sys.executable, but still not correct).

I've spent a lot of time digging into the logic of getpath.py where the venv path work is done, and realised that the fundamental problem here is that having a symlinked executable outside of an installation prefix is not something that was considered by PEP-405. Specifically:

If a home key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the home key is the directory containing the Python executable used to create this virtual environment.
In this case, prefix-finding continues as normal using the value of the home key as the effective Python binary location, which finds the prefix of the base installation

Clearly if the executable (symlinked) is in a different location to the prefix, then the documented approach is going to result in a computed prefix from the symlink (if that prefix has the appropriate landmark "lib-dynlib" directory)

This explains why resolving the symlink to executable before using that as home does work in many cases, but that solution misses the subtlety of symlinked prefixes that getpath handles well for venv-less prefix resolution (which doesn't fully resolve symlinks, but instead stops descending as soon as it finds a suitable prefix).

Note that today, CPython has no problem with non-venv symlinked executables living outside of the prefix. Using exactly the same setup as above, we get the desired behaviour:

$ /usr/local/bin/python3 -c "import sys; print(f'{sys.executable=}\n{sys.prefix=}')"
sys.executable='/usr/local/bin/python3'
sys.prefix='/opt/cpython1'

Perhaps @FFY00 (given their recent changes to getpath) has some idea of whether we can use exactly the same logic for venv base_prefix calculation as we do for non-venv prefix calculation?

Fundamentally, my belief is that the definition of home in PEP-405 should really be updated if it were to properly support symlinked executables outside of their prefix - the reason is that we don't record the actual base executable, and have to guess it based on heuristics (and the original symlink could have been named arbitrarily) . The OP's proposal to automatically resolve the symlink would be inconsistent with what Python does for non-venvs, and in my opinion, would be the wrong thing to do (as mentioned by others in this issue - it would break certain established / reasonable usages).

Given that today it works well to run a symlinked executable which is outside of its prefix for non-venv cases, I would propose that the PEP is updated to replace home with a base_executable key (which is simply the value of sys.base_executable), upon which the existing getpath logic for determining its prefix and lib directories can be applied.

I'd be happy to turn this into a discussion on python.org, if this is the best way to establish a solid rationale and consensus (please confirm if this is the best approach).

To follow-up to my very long last message - I think following what CPython is doing for prefix resolution (for non-venvs) at startup would work well for all cases that I've seen. Namely, it uses a non-path segment recursive symlink resolve on the executable, which then is then used to search for the stdlib.

This would require no change in venv - it should continue to report dirname(executable) as the home in pyvenv.cfg, as it is the interpreter's responsibility to determine the correct location for the venv's prefix based on that executable location.

It turns out (after a lot of searching), that this is a two-line fix to getpath.py:

diff --git a/Modules/getpath.py b/Modules/getpath.py
index c34101e7208..e82ba23c4c2 100644
--- a/Modules/getpath.py
+++ b/Modules/getpath.py
@@ -412,6 +412,9 @@ def search_up(prefix, *landmarks, test=isfile):
                             if isfile(candidate):
                                 base_executable = candidate
                                 break
+                if base_executable:
+                    # Update the executable directory to be based on the resolved base executable
+                    executable_dir = basename(base_executable)
             break
     else:
         # We didn't find a 'home' key in pyvenv.cfg (no break), reset venv_prefix.

I'll dig into how to write a test for this, and submit.

One thing to note: I think the calculation of sys._base_executable is over-resolving symlinks today - this is a separate bug which I guess means that homebrew users can't take a venv from a venv and have it survive beyond the next patch update.

@FFY00

Perhaps @FFY00 (given their recent changes to getpath) has some idea of whether we can use exactly the same logic for venv base_prefix calculation as we do for non-venv prefix calculation?

Fundamentally, my belief is that the definition of home in PEP-405 should really be updated if it were to properly support symlinked executables outside of their prefix - the reason is that we don't record the actual base executable, and have to guess it based on heuristics (and the original symlink could have been named arbitrarily) . The OP's proposal to automatically resolve the symlink would be inconsistent with what Python does for non-venvs, and in my opinion, would be the wrong thing to do (as mentioned by others in this issue - it would break certain established / reasonable usages).

Given that today it works well to run a symlinked executable which is outside of its prefix for non-venv cases, I would propose that the PEP is updated to replace home with a base_executable key (which is simply the value of sys.base_executable), upon which the existing getpath logic for determining its prefix and lib directories can be applied.

Yeah, it's known that the home key is problematic. We could replace it with a base_executable key, as you propose, but we'd still be relying on the IMO faulty "search up until we find something 🤷" heuristic. I'd rather just go away with all heuristics and replace the home key with base-prefix and base-exec-prefix keys, which directly gives us the prefixes. I opened GH-127895 with this proposal.

It turns out (after a lot of searching), that this is a two-line fix to getpath.py:

Please submit it! I think we can treat it as a bug, so we can still backport the fix to 3.12 and 3.13.

It seems there is a bug in venv (and a similar one in virtualenv) where the "base" path in pyvenv.cfg is set incorrectly. If Python is installed in a non-standard folder, e.g. /usr/local/python-X.Y.Z and then symlinked into /usr/local/bin/python3, the venv package does not work correctly.

With GH-127972 and GH-127974, on POSIX builds with a shared libpython, this should now be mitigated by looking at the libpython directory before resorting to the base interpreter, which should be much more reliable.

pythongh-106045: Fix ``venv`` creation from a python executable symlink

on POSIX builds with a shared libpython, this should now be mitigated by looking at the libpython directory before resorting to the base interpreter, which should be much more reliable.

It does not seem to be helping in this case (on macOS):

mkdir -p ../temp-test/bin
mkdir -p ../temp-test/lib/python3.14/lib-dynload
touch ../temp-test/lib/python3.14/os.py
./configure --enable-shared --with-pydebug --prefix=$(pwd)/build/install  && make -j install
ln -sf $(pwd)/build/install/bin/python3 ../temp-test/bin/python
../temp-test/bin/python -m venv ../temp-test/venv   
Error: Command '['../temp-test/venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

With #115237 resolving home differently, the venv works with both shared & static builds.

@pelson

It turns out (after a lot of searching), that this is a two-line fix to getpath.py

@pelson, did you mean the following instead ?

diff --git a/Modules/getpath.py b/Modules/getpath.py
index be2210345a..6607aacb80 100644
--- a/Modules/getpath.py
+++ b/Modules/getpath.py
@@ -412,6 +412,9 @@ def search_up(prefix, *landmarks, test=isfile):
                             if isfile(candidate):
                                 base_executable = candidate
                                 break
+                if base_executable and isfile(base_executable):
+                    # Update the executable directory to be based on the resolved base executable
+                    executable_dir = real_executable_dir = dirname(base_executable)
             break
     else:
         # We didn't find a 'home' key in pyvenv.cfg (no break), reset venv_prefix.

I frequently encounter this problem because of how I've set up my multiple local builds of Python (used to test version compatibility for packages, and also to investigate differences between Python versions). Because of the wrong home path, the standard library can't be found at all, with more dramatic consequences:

$ py3.13 -m venv --without-pip test && test/bin/python
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007464fa539740 (most recent call first):
  <no Python frame>

This seems to be a regression introduced in 3.11 (my py3.12 works because it sees the standard library of my system Python).

Can you run ldd py3.13? If libpython is linked, please also let me know if the linker is loading it from the correct place.

$ ldd `which py3.13`
	linux-vdso.so.1 (0x00007ffe127f4000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007c9ffec28000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007c9ffe200000)
	/lib64/ld-linux-x86-64.so.2 (0x00007c9ffed2b000)

Not there. I just used all default compilation options for these builds.

But it looks the same for the ones where it works.

Right, if you did not build with --enable-shared, it will static link libpython. Are you moving or copying the python binary to a different location? Or rather, how exactly are you installing Python?

Python is installed by using configure to set a prefix and using make altinstall, then manually symlinking to the altinstalled binary from /usr/local/bin. At least, that's what my notes say to do; I'll try to make time this week to redo it from scratch and confirm.

I ran into this issue in python-build-standalone, astral-sh/python-build-standalone#380

Is there still interested in a change to getpath.py to address this? I'd be happy to create a PR for discussion.

@jjhelmus

@jjhelmus sure, let's see what a PR looks like.

Is there still interested in a change to getpath.py to address this? I'd be happy to create a PR for discussion.

It's unclear to me what would be changed, but feel free to open a PR. Just beware that touching getpath.py is always very delicate, and there's a big possibility that whatever you change will slightly change its behavior in ways other use-cases may not be expecting, and break them. This is why I am always so cautious when touching getpath 😅

Would it be possible to add all those untested use cases / branches to the unit test coverage before changing getpath?

I believe a lot of them are, IIRC the getpath tests are pretty extensive. If you change something, most likely some test will fail, and we'll have to carefully consider the behavior change.

I've come back to this issue after some time and put up a proposed fix with test cases in #156483.

Currently, getpath.py searches for landmark files relative to the home value in pyvenv.cfg. When that value refers to an symlink, discovery can fall back to the prefix value set during the build. If the install does not match this location, either from relocating after and install or installing into a different location, then sys.base_prefix, etc are set incorrectly. This is the case that occured n python-build-standalone as the distribution are almost never installed into the configured prefix, /install.

Additionally, if an landmark file exist near the symlink, perhaps from another valid python installation, these can result in the symlinked directory being identified as the base installation.

Both of these can cases be reproduced with Docker containers:

FROM python:3.14-slim

# Relocate Python
RUN cp -a /usr/local /opt/python \
    && rm -rf /usr/local/lib/python3.14

# Create a symlink
RUN mkdir -p /opt/public/bin \
    && ln -s /opt/python/bin/python3 /opt/public/bin/python

# Creating venv
RUN /opt/public/bin/python -m venv --without-pip /opt/venv \
    && cat /opt/venv/pyvenv.cfg

CMD ["/opt/venv/bin/python", "-c", "import sys; print(sys.base_prefix)"]

Result:

❯ docker build -f Dockerfile.relocate -t relocate_example .
...
❯ docker run --rm relocate_example
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/opt/venv/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/local/lib/python3.14'
  sys.path[0] = (not set)
  sys._base_executable = '/opt/python/bin/python3.14'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/opt/venv/bin/python'
  sys.prefix = '/opt/venv'
  sys.exec_prefix = '/opt/venv'
  sys.path = [
    '/usr/local/lib/python314.zip',
    '/usr/local/lib/python3.14',
    '/usr/local/lib/python3.14/lib-dynload',
  ]
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x0000ffffa492d4c0 [python] (most recent call first):
  <no Python frame>

And the misleading landmark file:

FROM python:3.14-slim

# Symlink the interpreter
RUN mkdir -p /opt/public/bin \
    && ln -s /usr/local/bin/python3 /opt/public/bin/python

# Create misleading standard-library landmarks near the symlink.
RUN mkdir -p /opt/public/lib/python3.14/lib-dynload \
    && touch /opt/public/lib/python3.14/os.py

# Creating venv
RUN /opt/public/bin/python -m venv --without-pip /opt/venv \
    && cat /opt/venv/pyvenv.cfg

CMD ["/opt/venv/bin/python", "-c", "import sys; print(sys.base_prefix)"]

Result:

❯ docker build -f Dockerfile.landmark -t landmark_example .
...
❯ docker run --rm landmark_example
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/opt/venv/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/opt/public/lib/python3.14'
  sys.path[0] = (not set)
  sys._base_executable = '/usr/local/bin/python3.14'
  sys.base_prefix = '/opt/public'
  sys.base_exec_prefix = '/opt/public'
  sys.platlibdir = 'lib'
  sys.executable = '/opt/venv/bin/python'
  sys.prefix = '/opt/venv'
  sys.exec_prefix = '/opt/venv'
  sys.path = [
    '/opt/public/lib/python314.zip',
    '/opt/public/lib/python3.14',
    '/opt/public/lib/python3.14/lib-dynload',
  ]
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x0000ffffadbd84c0 [python] (most recent call first):
  <no Python frame>

Both of these layouts work as expected with the C-based path-discovery in Python 3.10:

Relocated Python

FROM python:3.10-slim

# Relocate Python
RUN cp -a /usr/local /opt/python \
    && rm -rf /usr/local/lib/python3.10

# Create a symlink
RUN mkdir -p /opt/public/bin \
    && ln -s /opt/python/bin/python3 /opt/public/bin/python

# Creating venv
RUN /opt/public/bin/python -m venv --without-pip /opt/venv \
    && cat /opt/venv/pyvenv.cfg

CMD ["/opt/venv/bin/python", "-c", "import sys; print(sys.base_prefix)"]

Results:

❯ docker build -f Dockerfile.relocate310 -t relocate_example310 .
...
❯ docker run --rm relocate_example310
/opt/python

Landmark

FROM python:3.10-slim

# Symlink the interpreter
RUN mkdir -p /opt/public/bin \
    && ln -s /usr/local/bin/python3 /opt/public/bin/python

# Create misleading standard-library landmarks near the symlink.
RUN mkdir -p /opt/public/lib/python3.10/lib-dynload \
    && touch /opt/public/lib/python3.10/os.py

# Creating venv
RUN /opt/public/bin/python -m venv --without-pip /opt/venv \
    && cat /opt/venv/pyvenv.cfg

CMD ["/opt/venv/bin/python", "-c", "import sys; print(sys.base_prefix)"]

❯ docker build -f Dockerfile.landmark310 -t landmark_example310 .
...
❯ docker run --rm landmark_example310
/usr/local

gh-106045: Fix venv path discovery for symlinked executables (#156483)

[3.15] gh-106045: Fix venv path discovery for symlinked executables (G…

[3.14] gh-106045: Fix venv path discovery for symlinked executables (G…

pythongh-106045: Keep searching from venv home when the resolved base…

gh-106045: Keep searching from venv home when the resolved base execu…

[3.14] gh-106045: Keep searching from venv home when the resolved bas…

[3.15] gh-106045: Keep searching from venv home when the resolved bas…

Navigation Menu

Repository navigation

venv base path does not resolve symlinks using realpath() #106045

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

venv base path does not resolve symlinks using realpath() #106045

Description

Linked PRs

Activity

mayeut commented on Sep 28, 2024

pelson commented on Jan 8, 2025

pelson commented on Jan 8, 2025

FFY00 commented on Jan 8, 2025

mayeut commented on Jan 11, 2025

mayeut commented on Jan 16, 2025

zahlman commented on Mar 28, 2025

FFY00 commented on Mar 28, 2025

zahlman commented on Mar 29, 2025

FFY00 commented on Mar 30, 2025

zahlman commented on Mar 31, 2025

jjhelmus commented on Oct 31, 2025

brettcannon commented on Oct 31, 2025

FFY00 commented on Dec 10, 2025

reneleonhardt commented on Dec 10, 2025

FFY00 commented on Dec 10, 2025

jjhelmus commented on Aug 27, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions