sys._base_executable resolves symlinks too eagerly, resulting in undesirable venv from venv behaviour

[pelson]

Bug report

Bug description:

In #106045, the use case of putting a symlink in /usr/local/bin to a Python binary from another prefix was highlighted. For homebrew, this use case is used with the idea that the symlink is the public interface/location, and the prefix where it is actually installed is an implementation detail (and can change over time) (more details in astral-sh/uv#1640).

#106045 seems to have now been resolved by #127974 (I wrote a test for that). The originating issue proposed a solution which would fix the problem with the broken virtual environment in that case, but would actually break the aforementioned homebrew example by eagerly resolving the symlinked executable and using the resolved path to determine the home value in pyvenv.cfg - thereby exposing the "internal" prefix into pyvenv.cfg.

The problem I am reporting here is that

• because sys._base_executable eagerly resolves symlinks in getpath.py
• and venv, virtualenv and uv all use sys._base_executable to determine the home location
• then when you take a virtual environment from another virtual environment in the aforementioned setup, the second virtual environment's home location will be resolved, whereas the first will not be. Thereby bleeding the implementation detail once more.

In uv, the behaviour was solidified in astral-sh/uv#8433 to avoid exposing the internal prefix in the venv. Note that virtualenv is (accidentally?) now exposing the internal prefix even on the first virtual environment pypa/virtualenv#2770.

To reproduce this:

$ mkdir -p /tmp/public/bin/
$ ln -s /usr/local/bin/python3.14 /tmp/public/bin/

$ /tmp/public/bin/python3.14 -m venv /tmp/venv1
$ /tmp/venv1/bin/python -m venv /tmp/venv2

$ cat /tmp/venv1/pyvenv.cfg 
home = /tmp/public/bin
include-system-site-packages = false
version = 3.14.0
executable = /usr/local/bin/python3.14
command = /tmp/public/bin/python3.14 -m venv /tmp/venv1

$ cat /tmp/venv2/pyvenv.cfg 
home = /usr/local/bin
include-system-site-packages = false
version = 3.14.0
executable = /usr/local/bin/python3.14
command = /tmp/venv1/bin/python -m venv /tmp/venv2

And observe that the second venv's home is not /tmp/public/bin but the "internal detail" one.

A quick test for getpath, and a complete test for venv are included below (both failing):

diff --git a/Lib/test/test_getpath.py b/Lib/test/test_getpath.py
index f86df9d0d03..1c2e2a0b3fc 100644
--- a/Lib/test/test_getpath.py
+++ b/Lib/test/test_getpath.py
@@ -864,6 +864,55 @@ def test_PYTHONHOME_in_venv(self):
         actual = getpath(ns, expected)
         self.assertEqual(expected, actual)
 
+    def test_venv_w_symlinked_base_executable(self):
+        """
+        If we symlink the base executable, and point to it via home in pyvenv.cfg,
+        we should have it as sys.executable (and sys.prefix should be the resolved location)
+        """
+        ns = MockPosixNamespace(
+            argv0="/venv/bin/python3",
+            PREFIX="/some/_internal/prefix",
+        )
+        # Setup venv
+        ns.add_known_xfile("/venv/bin/python3")
+        ns.add_known_xfile("/usr/local/bin/python3")
+        ns.add_known_xfile("/some/_internal/prefix/bin/python3")
+
+        ns.add_known_file("/venv/pyvenv.cfg", [
+            # The published based executable location is /usr/local/bin - we don't want to
+            # expose /some/internal/directory (this location can change under our feet)
+            r"home = /usr/local/bin"
+        ])
+        ns.add_known_link("/venv/bin/python3", "/usr/local/bin/python3")
+        ns.add_known_link("/usr/local/bin/python3", "/some/_internal/prefix/bin/python3")
+
+        ns.add_known_file("/some/_internal/prefix/lib/python9.8/os.py")
+        ns.add_known_dir("/some/_internal/prefix/lib/python9.8/lib-dynload")
+
+        # Put a file completely outside of /usr/local to validate that the issue
+        # in https://github.com/python/cpython/issues/106045 is resolved.
+        ns.add_known_dir("/usr/lib/python9.8/lib-dynload")
+
+        expected = dict(
+            executable="/venv/bin/python3",
+            prefix="/venv",
+            exec_prefix="/venv",
+            base_prefix="/some/_internal/prefix",
+            base_exec_prefix="/some/_internal/prefix",
+            # It is important to maintain the link to the original executable, as this
+            # is used when creating a new virtual environment (which should also have home
+            # set to /usr/local/bin to avoid bleeding the internal path to the venv)
+            base_executable="/usr/bin/python3",
+            module_search_paths_set=1,
+            module_search_paths=[
+                "/some/_internal/prefix/lib/python98.zip",
+                "/some/_internal/prefix/lib/python9.8",
+                "/some/_internal/prefix/lib/python9.8/lib-dynload",
+            ],
+        )
+        actual = getpath(ns, expected)
+        self.assertEqual(expected, actual)
+
 # ******************************************************************************
 
 DEFAULT_NAMESPACE = dict(
diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py
index 0b09010c69d..bd1b19a9c15 100644
--- a/Lib/test/test_venv.py
+++ b/Lib/test/test_venv.py
@@ -756,6 +756,53 @@ def test_zippath_from_non_installed_posix(self):
         out, err = check_output(cmd)
         self.assertTrue(zip_landmark.encode() in out)
 
+    @unittest.skipIf(os.name == 'nt', 'not relevant on Windows')
+    @requireVenvCreate
+    def test_venv_from_venv_with_symlink(self):
+        """
+        Test that we can create a venv from a venv using a base Python that is
+        a symlink, without exposing the location of the symlink in pyvenv.cfg.
+        """
+        rmtree(self.env_dir)
+        public_prefix = os.path.realpath(tempfile.mkdtemp())
+        self.addCleanup(rmtree, public_prefix)
+        public_bin_dir = os.path.join(public_prefix, 'bin')
+        os.mkdir(public_bin_dir)
+        public_exe = os.path.join(public_bin_dir, self.exe)
+        os.symlink(sys.executable, public_exe)
+        cmd = [public_exe,
+               "-m",
+               "venv",
+               "--without-pip",
+               "--without-scm-ignore-files",
+               self.env_dir]
+
+        subprocess.check_call(cmd)
+
+        # Verify that we don't expose the internal prefix to the first venv config:
+        contents = (pathlib.Path(self.env_dir) / 'pyvenv.cfg').read_text()
+        self.assertIn(f'home = {public_bin_dir}\n', contents)
+
+        # Now use the venv to make another, and assert that the internal env is
+        # also not exposed there.
+        second_venv = os.path.realpath(tempfile.mkdtemp())
+        self.addCleanup(rmtree, second_venv)
+
+        cmd = [os.path.join(self.env_dir, 'bin', 'python3'),
+               "-m",
+               "venv",
+               "--without-pip",
+               "--without-scm-ignore-files",
+               second_venv]
+
+        subprocess.check_call(cmd)
+
+        contents = (pathlib.Path(second_venv) / 'pyvenv.cfg').read_text()
+        self.assertIn(f'home = {public_bin_dir}\n', contents)
+
     @requireVenvCreate
     def test_activate_shell_script_has_no_dos_newlines(self):
         """

cc @FFY00 following on from our conversation in #127974 (comment) (I would love to get a commit for the tests, if not the fix 😜).

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

[pelson]

There is also some useful context on cases that are covered by uv in astral-sh/uv#9723. The approach taken there allows for the additional possibility of having a symlink to Python which is not named python, and still being able to create a venv from it. The issue I've raised here is only intended to get the second and third cases right - IMO the first case should be handled by a change to the pyvenv.cfg.

[FFY00]

As of GH-127972, does this even really matter? Most users will be using Python installations with a shared libpython, so we can know the base prefix from there.

Realistically, the home key in pyvenv.cfg only matters on statically-linked interpreters and when not symlinking the interpreter in the virtual environment. Otherwise, we should be able to find the prefix.

Apart from the home key pointing to an internal prefix, are you experiencing any actual issue when running the interpreter?

All these complications exist in modern system that adopt a non-standard installation model. It seems reasonable to me that for these complicated scenarios to be supported to ask that we are running in an installation with a shared libpython. That should be working right now in main. We could possibly also stop relying on the home in pyvenv.cfg in statically linked installations that use symlinks from virtual environments by also searching for the prefix at the real executable location.

[zanieb]

See also

• Creating venvs via standard library's venv.create(..., with_pip=True) fails for binaries linked against libpython dynamically astral-sh/python-build-standalone#381
• Virtualenvs created from links are broken astral-sh/python-build-standalone#380

cc @charliermarsh who is definitely our authority on this topic.

[charliermarsh]

In uv, for the relocatable builds, we now resolve symlinks iteratively until we find a path that "would" be a valid home, by replicating some of the logic used in CPython startup. This tends to "do the right thing" in practice.

(Edit: missed that this was stated above in #128670 (comment), but happy to answer questions about that approach.)