[BUG] 71.0.0 fails with backports.tarfile on Python 3.8 when other backports imported

[amotl]

setuptools version

setuptools==71.0.0

Python version

Python 3.8

OS

Linux

Additional environment information

No response

Description

CI jobs started tripping when using setuptools 71, released two hours ago.

• Setup: Fix package installation woes with setuptools/backports.tarfile, and PyInstaller crate/cratedb-toolkit#201

Expected behavior

CI builds work as expected.

How to Reproduce

No reproducer yet, but a stacktrace on GHA.

Output

ImportError: cannot import name 'tarfile' from 'backports' 
  (/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/backports/__init__.py)

-- https://github.com/crate/cratedb-toolkit/actions/runs/9982521975/job/27589490156?pr=196#step:6:229

[leonardwellthy]

A slightly longer stack trace from a python3.10 install:

  File "/usr/local/lib/python3.10/dist-packages/simple_history/__init__.py", line 1, in <module>
    from pkg_resources import DistributionNotFound, get_distribution
  File "/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py", line 93, in <module>
    from jaraco.text import (
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_vendor/jaraco/text/__init__.py", line 12, in <module>
    from jaraco.context import ExceptionTrap
  File "/usr/local/lib/python3.10/dist-packages/setuptools/_vendor/jaraco/context.py", line 17, in <module>
    from backports import tarfile
ImportError: cannot import name 'tarfile' from 'backports' (unknown location)

[jaraco]

It works for me. It sounds like the issue may be isolated to PyInstaller.

It's interesting that backports.tarfile is the affected package. It's the only namespace package that's not PEP 420, but instead relies on pkgutil.

[jaraco]

When I say it works for me, here's Dockerfile that works:

from ubuntu:focal
run apt update
run apt upgrade -y
run apt install -y python3-pip python3-venv
run python3 -m venv .venv
run .venv/bin/pip install -U setuptools
cmd .venv/bin/python -c "import pkg_resources"

We'll need to figure out what factors are necessary to trigger the failure.

[larsevj]

I am also encountering this issue. Can reproduce it with the following packages installed:

Package                          Version
-------------------------------- -------
backports-datetime-fromisoformat 2.0.1
pip                              24.1.2
setuptools                       71.0.1

and then running:

>>> from backports.datetime_fromisoformat import MonkeyPatch
>>> import pkg_resources
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/temp/debug_setuptools_issue/lib64/python3.11/site-packages/pkg_resources/__init__.py", line 93, in <module>
    from jaraco.text import (
  File "/data/temp/debug_setuptools_issue/lib64/python3.11/site-packages/setuptools/_vendor/jaraco/text/__init__.py", line 12, in <module>
    from jaraco.context import ExceptionTrap
  File "/data/temp/debug_setuptools_issue/lib64/python3.11/site-packages/setuptools/_vendor/jaraco/context.py", line 17, in <module>
    from backports import tarfile
ImportError: cannot import name 'tarfile' from 'backports' (/data/temp/debug_setuptools_issue/lib64/python3.11/site-packages/backports/__init__.py)

[AbdealiLoKo]

Not sure what is happening, but with both 71.0.0 and 71.0.1 - my pytest flow stopped working

We use pyfilesystem2 - which is using pkg_resources internally and has the same issue:

pytest is failing here:

$ venv38/bin/pytest -v
====================================== test session starts ======================================
platform linux -- Python 3.8.19, pytest-7.4.4, pluggy-1.5.0 -- venv38/bin/python
cachedir: .pytest_cache
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
configfile: setup.cfg
plugins: env-1.1.3, mock-3.14.0, error-for-skips-2.0.2, fail-slow-0.6.0, unordered-0.6.1, split-0.9.0, flask-1.3.0, timeout-2.3.1, benchmark-4.0.0, cov-5.0.0
collected 0 items / 1 error

==================== ERRORS =========================
_________________________________ ERROR collecting test session _________________________________
../.conda/envs/py38/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
venv38/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
    exec(co, module.__dict__)
apps/myapp/myapp_test/conftest.py:13: in <module>
    from myapp.db.models import SQLAlchemy
apps/myapp/myapp/__init__.py:11: in <module>
    from myapp.config import setup_logging, setup_settings
apps/myapp/myapp/config/__init__.py:13: in <module>
    import fs as fs_library
venv38/lib/python3.8/site-packages/fs/__init__.py:4: in <module>
    __import__("pkg_resources").declare_namespace(__name__)  # type: ignore
venv38/lib/python3.8/site-packages/pkg_resources/__init__.py:93: in <module>
    from jaraco.text import (
venv38/lib/python3.8/site-packages/setuptools/_vendor/jaraco/text/__init__.py:12: in <module>
    from jaraco.context import ExceptionTrap
venv38/lib/python3.8/site-packages/setuptools/_vendor/jaraco/context.py:17: in <module>
    from backports import tarfile
E   ImportError: cannot import name 'tarfile' from 'backports' (venv38/lib/python3.8/site-packages/backports/__init__.py)

But note that - When I don't use pytest - it works fine.
When I do venv38/bin/python -c "import fs" and venv38/bin/python -c "import pkg_resources" - there is no error
So, I am unclear how to make a Minimal Reproducible Example ...

Workarounds:

1. Downgrade setuptools to <70
2. Install it myself with: venv38/bin/pip install backports.tarfile

[jaraco]

Thanks larsevj. That detail is quite helpful. It implicates a problem when there are other backports.* packages present. I thought maybe the issue was that package isn't using pkgutil-style namespaces, but it is.

I'm now thinking the issue might be that pkgutil-based namespace packages (such as 'backports') don't support loading from more than one location.

[jaraco]

It's not as simple as the two namespaces not being honored:

 draft 🐚 mkdir path1 path2
 draft 🐚 py -3.12 -m pip install -q -t path1 backports.tarfile
 draft 🐚 py -3.12 -m pip install -q -t path2 backports-datetime-fromisoformat
 draft 🐚 env PYTHONPATH=path1:path2 py -3.12 -c "import backports.datetime_fromisoformat; import backports.tarfile" && echo worked
worked

[xolox]

We just got bitten by a variant of this issue. We don't use PyInstaller, however in our installation directory for Python requirements there existed an "empty" backports package, with only an empty __init__.py file). This was a leftover from an old install of the backports namespace package that should have been cleaned up but wasn't (caused by internal tooling on our side, so mea culpa). This empty backports package was shadowing the backports package bundled (vendored?) with pkg_resources and that caused the issue for us, so when a new build included the latest setuptools this "latent bug" was suddenly exposed. In our case resolution was as simple as deleting the empty backports namespace package (which should have been done a long time ago).

[jaraco]

Aha, so the issue appears to be when backports is added to sys.path. This example reproduces the failure:

 draft 🐚 env PYTHONPATH=path2 py -3.12 -c "import backports.datetime_fromisoformat; import sys; sys.path.append('path1'); import backports.tarfile" && echo worked
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'backports.tarfile'

[jaraco]

Thanks xolox for the additional detail. I do agree a latent backports module could be problemmatic, but I think there's a broader cause. Here's what I think is happening:

• Something in the environment imports some backports subpackage already on sys.path but before setuptools has added its vendored packages to sys.path.
• Then Setuptools is imported and adds its vendored packages to sys.path.
• Because the discovery for backports was already done, it's __path__ not recalculated and any vendored backports are never found.

I believe that PEP 420 namespace packages do not have this problem, because they allow for dynamic recalculation of the __path__ if sys.path changes. I'm going to investigate pkgutil-style namespace packages some more to see if I can confirm my suspicion.

[jaraco]

I read through the extend_path source and confirmed that the backports.__path__ will only be calculated once when the first backports package is found.

I did find that I can work around the issue by clearing out the backports package after adjusting sys.path:

 draft 🐚 env PYTHONPATH=path2 py -3.12 -c "import backports.datetime_fromisoformat; import sys; sys.path.append('path1'); del sys.modules['backports']; import backports.tarfile" && echo worked
worked

So maybe that's what setuptools should do to work around the issue (until some day backports can migrate to PEP 420).

[jaraco]

I'm thinking something like:

diff --git a/setuptools/__init__.py b/setuptools/__init__.py
index 8b0e494f0..fbc41f205 100644
--- a/setuptools/__init__.py
+++ b/setuptools/__init__.py
@@ -7,6 +7,8 @@ import sys
 from typing import TYPE_CHECKING
 
 sys.path.extend(((vendor_path := os.path.join(os.path.dirname(os.path.dirname(__file__)), 'setuptools', '_vendor')) not in sys.path) * [vendor_path])  # fmt: skip
+# workaround for #4476
+sys.modules.pop('backports', None)  # noqa: E402
 
 import _distutils_hack.override  # noqa: F401
 import distutils.core

Would do the trick. Unfortunately, that introduces 10 new nitpicky E402 lint errors:

 setuptools main 🐚 ruff check --output-format concise
setuptools/__init__.py:13:1: E402 Module level import not at top of file
setuptools/__init__.py:14:1: E402 Module level import not at top of file
setuptools/__init__.py:15:1: E402 Module level import not at top of file
setuptools/__init__.py:17:1: E402 Module level import not at top of file
setuptools/__init__.py:18:1: E402 Module level import not at top of file
setuptools/__init__.py:19:1: E402 Module level import not at top of file
setuptools/__init__.py:20:1: E402 Module level import not at top of file
setuptools/__init__.py:21:1: E402 Module level import not at top of file
setuptools/__init__.py:22:1: E402 Module level import not at top of file
setuptools/__init__.py:23:1: E402 Module level import not at top of file
Found 10 errors.

[jaraco]

Is someone available to test if the PR in #4486 fixes the issue? Maybe pip install git+https://github.com/pypa/setuptools@refs/pull/4486/head.

[edgarrmondragon]

@jaraco I can confirm that #4486 fixes the problem for me :)

[larsevj]

#4486 seems to be working for me as well.

[Tolulade-A]

I'm getting this error on aws apprunner ImportError: cannot import name 'tarfile' from 'backports' (/usr/local/lib64/python3.8/site-packages/backports/__init__.py)

[jaraco]

@Tolulade-A I'm confident that the Setuptools 71.0.3 release fixes the issue. Can you confirm that you're on setuptools!=71.0.0,!=71.0.1,71.0.2?

[Tolulade-A]

I'm on setuptools>=67 and downgraded to setuptools>=50.0.0

@Tolulade-A I'm confident that the Setuptools 71.0.3 release fixes the issue. Can you confirm that you're on setuptools!=71.0.0,!=71.0.1,71.0.2?

[yasamanparhizkar]

Workarounds:

1. Downgrade setuptools to <70
2. Install it myself with: venv38/bin/pip install backports.tarfile

I faced a similar bug with python 3.10. It was fixed by downgrading to <70

pip install --force-reinstall -v "setuptools<70"

[cacti77]

On Azure DevOps I'm seeing this with setuptools 71.1.0 (i.e., > 71.0.3) when trying to create a source distribution in a Docker container running Python 3.10.10:

Traceback (most recent call last):
  File "/__w/1/s/my-package/setup.py", line 30, in <module>
    import setuptools
  File "/home/jovyan/.local/lib/python3.10/site-packages/setuptools/__init__.py", line 21, in <module>
    from .dist import Distribution
  File "/home/jovyan/.local/lib/python3.10/site-packages/setuptools/dist.py", line 29, in <module>
    from . import _entry_points
  File "/home/jovyan/.local/lib/python3.10/site-packages/setuptools/_entry_points.py", line 6, in <module>
    from jaraco.text import yield_lines
  File "/home/jovyan/.local/lib/python3.10/site-packages/setuptools/_vendor/jaraco/text/__init__.py", line 12, in <module>
    from jaraco.context import ExceptionTrap
  File "/home/jovyan/.local/lib/python3.10/site-packages/setuptools/_vendor/jaraco/context.py", line 17, in <module>
    from backports import tarfile
ImportError: cannot import name 'tarfile' from 'backports' (/opt/conda/lib/python3.10/site-packages/backports/__init__.py)

UPDATE: setuptools 71.0.4 also failed with the same error, but 70.3.0 worked.

[jaraco]

UPDATE: setuptools 71.0.4 also failed with the same error, but 70.3.0 worked.

If you're still having the error on setuptools 71.0.4 or later, it's a different root cause. Please open a new issue and importantly describe a reproducer (or at least more detail about your environment such that we might be able to reproduce it). Crucial will be to describe what other packages you have installed in the environment where the failure occurs.

[austin3dickey]

Thanks, @jaraco. I was also running into the issue on 71.0.4, so I just opened #4508.

[cacti77]

UPDATE: setuptools 71.0.4 also failed with the same error, but 70.3.0 worked.

If you're still having the error on setuptools 71.0.4 or later, it's a different root cause. Please open a new issue and importantly describe a reproducer (or at least more detail about your environment such that we might be able to reproduce it). Crucial will be to describe what other packages you have installed in the environment where the failure occurs.

Done. See #4509

[wdcs-rutvikvasoya]

I am also encountering this issue. Can reproduce it with the following packages installed:

Package                          Version
-------------------------------- -------
backports-datetime-fromisoformat 2.0.1
pip                              24.1.2
setuptools                       71.0.1

and then running:

>>> from backports.datetime_fromisoformat import MonkeyPatch
>>> import pkg_resources
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/temp/debug_setuptools_issue/lib64/python3.11/site-packages/pkg_resources/__init__.py", line 93, in <module>
    from jaraco.text import (
  File "/data/temp/debug_setuptools_issue/lib64/python3.11/site-packages/setuptools/_vendor/jaraco/text/__init__.py", line 12, in <module>
    from jaraco.context import ExceptionTrap
  File "/data/temp/debug_setuptools_issue/lib64/python3.11/site-packages/setuptools/_vendor/jaraco/context.py", line 17, in <module>
    from backports import tarfile
ImportError: cannot import name 'tarfile' from 'backports' (/data/temp/debug_setuptools_issue/lib64/python3.11/site-packages/backports/__init__.py)

this is work for me thank you

[nayanjha16]

Is someone available to test if the PR in #4486 fixes the issue? Maybe pip install git+https://github.com/pypa/setuptools@refs/pull/4486/head.

I tried it, however the issue still persists for me.

Versions are listed as under:

Python 3.8.19
mlflow 2.14.3

Issue is as follows:

{
	"name": "ImportError",
	"message": "cannot import name 'tarfile' from 'backports' (c:\\Users\\NayanAnand\\anaconda3\\lib\\site-packages\\backports\\__init__.py)",
	"stack": "---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
~\\AppData\\Local\\Temp\\ipykernel_25044\\2545141633.py in <module>
      3 from sklearn.ensemble import RandomForestRegressor
      4 
----> 5 from mlflow import MlflowClient

c:\\Users\\NayanAnand\\anaconda3\\lib\\site-packages\\mlflow\\__init__.py in <module>
     32 
     33 __version__ = VERSION
---> 34 from mlflow import (
     35     artifacts,  # noqa: F401
     36     client,  # noqa: F401

c:\\Users\\NayanAnand\\anaconda3\\lib\\site-packages\\mlflow\\models\\__init__.py in <module>
     42 )
     43 from mlflow.models.flavor_backend import FlavorBackend
---> 44 from mlflow.models.model import Model, get_model_info, set_model
     45 from mlflow.models.model_config import ModelConfig
     46 from mlflow.models.python_api import build_docker

c:\\Users\\NayanAnand\\anaconda3\\lib\\site-packages\\mlflow\\models\\model.py in <module>
     26 from mlflow.utils.databricks_utils import get_databricks_runtime_version, is_in_databricks_runtime
     27 from mlflow.utils.docstring_utils import LOG_MODEL_PARAM_DOCS, format_docstring
---> 28 from mlflow.utils.environment import (
     29     _CONDA_ENV_FILE_NAME,
     30     _PYTHON_ENV_FILE_NAME,

c:\\Users\\NayanAnand\\anaconda3\\lib\\site-packages\\mlflow\\utils\\environment.py in <module>
     22 from mlflow.utils.os import is_windows
     23 from mlflow.utils.process import _exec_cmd
---> 24 from mlflow.utils.requirements_utils import (
     25     _infer_requirements,
     26     _parse_requirements,

c:\\Users\\NayanAnand\\anaconda3\\lib\\site-packages\\mlflow\\utils\\requirements_utils.py in <module>
     18 
     19 import importlib_metadata
---> 20 import pkg_resources  # noqa: TID251
     21 from packaging.requirements import Requirement
     22 from packaging.version import InvalidVersion, Version

c:\\Users\\NayanAnand\\anaconda3\\lib\\site-packages\\pkg_resources\\__init__.py in <module>
     93 
     94 import packaging.specifiers
---> 95 from jaraco.text import (
     96     yield_lines,
     97     drop_comment,

c:\\Users\\NayanAnand\\anaconda3\\lib\\site-packages\\setuptools\\_vendor\\jaraco\\text\\__init__.py in <module>
     10 
     11 from jaraco.functools import compose, method_cache
---> 12 from jaraco.context import ExceptionTrap
     13 
     14 

c:\\Users\\NayanAnand\\anaconda3\\lib\\site-packages\\setuptools\\_vendor\\jaraco\\context.py in <module>
     15 
     16 if sys.version_info < (3, 12):
---> 17     from backports import tarfile
     18 else:
     19     import tarfile

ImportError: cannot import name 'tarfile' from 'backports' (c:\\Users\\NayanAnand\\anaconda3\\lib\\site-packages\\backports\\__init__.py)"
}

[jaraco]

If the issue isn't fixed by 71.0.4, then there's a more narrow environmental concern to be addressed. I notice you're using conda. Have you seen #4508? There was a bug in the backports-feedstock of anaconda that's since been fixed.