`reprotest-wheels` stalls installing `python-dateutil` #350

cfm · 2022-06-24T01:56:36Z

Specifically, installing python-dateutil for securedrop-client in:

https://github.com/freedomofpress/securedrop-debian-packaging/blob/b13e30d06d01a43b4909434f1cb38017b0dec951/scripts/build-sync-wheels#L75-L90

Evident in CI (e.g.) and reproducible by invoking manually:

$ pip3 download --no-binary :all: --require-hashes --dest /tmp/tmp8zgdeb0_ --requirement /tmp/securedrop-clientlh0mh054/requirements/requirements.txt
[...]
Collecting python-dateutil==2.7.5
  Using cached python-dateutil-2.7.5.tar.gz (316 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... |

Originally posted by @cfm in #349 (comment)

The text was updated successfully, but these errors were encountered:

legoktm · 2022-06-29T22:29:25Z

The key part of the log seems to be:

  Installing build dependencies ... -� �\� �|� �/� �-� �\� �|� �/� �-� �\� �done
  Getting requirements to build wheel ... -� �done
  Installing backend dependencies ... -� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �-� �\� �|� �/� �done
  Preparing metadata (pyproject.toml) ... -� �error
  ERROR: Command errored out with exit status 1:
   command: /home/ci/project/.venv/bin/python3 /home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpa46sr7yq
       cwd: /tmp/pip-download-uomjt3pc/python-dateutil_f166952022704c91aa4cd2f016a88be2
  Complete output (32 lines):
  Traceback (most recent call last):
    File "/home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 154, in prepare_metadata_for_build_wheel
      backend = _build_backend()
    File "/home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
      obj = import_module(mod_path)
    File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
    File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
    File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 728, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/tmp/pip-build-env-52am7k_j/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 18, in <module>
      from setuptools.dist import Distribution
    File "/tmp/pip-build-env-52am7k_j/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 35, in <module>
      from ._importlib import metadata
    File "/tmp/pip-build-env-52am7k_j/overlay/lib/python3.7/site-packages/setuptools/_importlib.py", line 39, in <module>
      disable_importlib_metadata_finder(metadata)
    File "/tmp/pip-build-env-52am7k_j/overlay/lib/python3.7/site-packages/setuptools/_importlib.py", line 30, in disable_importlib_metadata_finder
      for ob in sys.meta_path
    File "/tmp/pip-build-env-52am7k_j/overlay/lib/python3.7/site-packages/setuptools/_importlib.py", line 31, in <listcomp>
      if isinstance(ob, importlib_metadata.MetadataPathFinder)
  AttributeError: module 'importlib_metadata' has no attribute 'MetadataPathFinder'
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/0e/01/68747933e8d12263d41ce08119620d9a7e5eb72c876a3442257f74490da0/python-dateutil-2.7.5.tar.gz#sha256=88f9287c0174266bb0d8cedd395cfba9c58e87e5ad86b2ce58859bc11be3cf02 (from https://pypi.org/simple/python-dateutil/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*). Command errored out with exit status 1: /home/ci/project/.venv/bin/python3 /home/ci/project/.venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpa46sr7yq Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement python-dateutil==2.7.5 (from versions: 1.4, 1.4.1, 1.5, 2.1, 2.2, 2.3, 2.4.0, 2.4.1, 2.4.1.post1, 2.4.2, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.7.5, 2.8.0, 2.8.1, 2.8.2)
ERROR: No matching distribution found for python-dateutil==2.7.5

gonzalo-bulnes · 2022-06-30T01:34:15Z

What does this mean? (Using a diff just to highlight the parts that surprise me):

  ERROR: Could not find a version that satisfies the requirement python-dateutil==2.7.5 (from versions: 1.4, 1.4.1, 1.5, 2.1, 2.2, 2.3, 2.4.0, 2.4.1, 2.4.1.post1, 2.4.2, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.7.5, 2.8.0, 2.8.1, 2.8.2)
+ ERROR: ------------------------------------------------------------------------>2.7.5 (from versions: -keep scrolling-------------------------------------------------------------------------------------------------------------------------------->2.7.5<--------------------)

legoktm · 2022-06-30T01:45:24Z

Nothing significant, it's just how pip expresses the error. Because 2.7.5 couldn't be installed, obviously none of the other versions on pypi work since we're pinned to 2.7.5.

I did a bit of searching, I think bumping us up to 2.8.0 might fix it (https://dateutil.readthedocs.io/en/latest/changelog.html#id6 - addition of pyproject.toml) but also didn't finish looking into what that would take.

gonzalo-bulnes · 2022-06-30T02:00:22Z

I tried reproducing this locally and couldn't... 💭 not sure why.

cd ~/src/securedrop-client
git checkout main
cp ~/src/securedrop-client /tmp/securedrop-clientlh0mh054

cd ~/src/securedrop-debian-packaging
git checkout release-securedrop-workstation-config # known to be failing in CI
pip3 download --no-binary :all: --require-hashes --dest /tmp/tmp8zgdeb0_ --requirement /tmp/securedrop-clientlh0mh054/requirements/requirements.txt
# exits successfully within a few minutes

Closes freedomofpress/securedrop-builder#350, in which reprotest-wheels is failing on securedrop-client's requirement of python-dateutil 2.7.5.

cfm · 2022-07-05T17:25:58Z

To unblock #355, #356, and #357 (with more to come), I started to update securedrop-client's requirements for python-dateutil 2.8.0. However, quick testing reveals that neither v2.7.5 and v2.8.0 installs under Python 3.7 on Debian buster, while both install fine under Python 3.9 on Debian bullseye. Sure enough, reprotest-wheels's prebaked environment dates from last July on Debian buster. So updating that image for Debian bullseye has got to be our first step, which I'll start now.

…lseye:2022_05_10) Now that we're assuming Python 3.9 under Debian bullseye, that's the environment we should use for testing wheel-level reproducibility. Closes #350.

cfm · 2022-07-07T18:26:31Z

In preparation for next week's retrospective on this problem in the Workstation release process, @creviera has asked me to look more deeply into why the python-dateutil==2.7.5 dependency chain began failing for us under buster but not under bullseye. I'll close this ticket again next week with my findings.

legoktm · 2022-07-07T19:30:49Z

My theory was that something in pip/setuptools/wheel/etc. was automatically upgraded and prevented the old version of the package from installing. But that doesn't really address why it started working on bullseye. I would also like to get rid of the "builder" image, it doesn't actually add any value here...

cfm · 2022-07-12T01:01:40Z

I've been able to construct a minimal reproduction of this failure, revealing that:

The relevant difference in our testing wasn't Debian buster versus bullseye but Python 3.7 versus 3.9.
This is pip install --no-binary fails on Python 3.7 python/importlib_metadata#392, a side effect of setuptools_scm 7.0.0 fails with Python 3.7 pypa/setuptools-scm#718, an instance of the bootstrapping problem described in Bootstrapping problem (how to bootstrap all-source environments) pypa/packaging-problems#342.
Specifically, reprotest-wheels began to fail under Python 3.7 with setuptools_scm==7.0.1, which added an explicit dependency for
```
importlib-metadata;python_version < '3.8'
```

We would have needed to wait for an upstream fix for python/importlib_metadata#392 to restore Python 3.7 compatibility, but we were able to sidestep that requirement by switching reprotest-wheels to newly-preferred Python 3.9.

legoktm · 2022-07-13T19:00:58Z

@cfm ooh, nice discoveries. Was the dependency not properly pinned?

cfm · 2022-07-13T19:25:04Z

Good question. It's not obvious to me what to pin. I've just updated the reproduction to pin all of setuptools, setuptools_scm, and importlib-metadata, and the build-time circularity described in pypa/packaging-problems#342 (comment) remains under Python 3.7.

cfm · 2022-07-13T20:42:01Z

After today's retrospective, I'm working on a simplified reproduction, demonstrating the interactions of just setuptools_scm and importlib-metadata under Python 3.7....

cfm · 2022-07-14T01:18:02Z

Let's do some Socratic troubleshooting based on this even-narrower reproduction....

Why does python-dateutil==2.7.5 fail to install under Python 3.7? Because it fails to build from source, as of 22 June 2022.

Why does it fail to build from source? Because its backend dependency setuptools_scm fails to build from source. (pypa/setuptools-scm#722)

Why does setuptools_scm fail to build from source? Because, under Python < 3.8, its backend dependency importlib-metadata fails to build from source. (python/importlib_metadata#392)

Why does importlib-metadata fail to build from source? Because it depends on setuptools_scm....

When was this change introduced in setuptools_scm? Version 7.0.1, 21 June 2022.

Can't we pin to setuptools_scm<7.0.1 Sure. But this will still fail:

pip install --no-binary :all: "setuptools_scm<7.0.1" python-dateutil==2.7.5

$ pip install --no-binary :all: "setuptools_scm<7.0.1" python-dateutil==2.7.5
[...]
Collecting python-dateutil==2.7.5
  Downloading python-dateutil-2.7.5.tar.gz (316 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.0/316.0 KB 10.5 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: still running...
  Installing backend dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 1
  ╰─> [122 lines of output]
      Collecting setuptools_scm
        Downloading setuptools_scm-7.0.5.tar.gz (70 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 70.7/70.7 KB 3.9 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting tomli>=1.0.0
        Using cached tomli-2.0.1.tar.gz (15 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting importlib-metadata
        Downloading importlib_metadata-4.12.0.tar.gz (48 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.2/48.2 KB 6.8 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'error'
        error: subprocess-exited-with-error
      
        × pip subprocess to install build dependencies did not run successfully.
        │ exit code: 2
        ╰─> [81 lines of output]
            Collecting setuptools>=56
              Using cached setuptools-63.1.0.tar.gz (2.6 MB)
              Getting requirements to build wheel: started
              Getting requirements to build wheel: finished with status 'done'
              Installing backend dependencies: started
              Installing backend dependencies: finished with status 'done'
              Preparing metadata (pyproject.toml): started
              Preparing metadata (pyproject.toml): finished with status 'done'
            Collecting setuptools_scm[toml]>=3.4.1
              Using cached setuptools_scm-7.0.5.tar.gz (70 kB)
              Installing build dependencies: started
              Installing build dependencies: finished with status 'done'
              Getting requirements to build wheel: started
              Getting requirements to build wheel: finished with status 'done'
              Installing backend dependencies: started
              Installing backend dependencies: finished with status 'done'
              Preparing metadata (pyproject.toml): started
              Preparing metadata (pyproject.toml): finished with status 'done'
            Collecting typing-extensions
              Using cached typing_extensions-4.3.0.tar.gz (47 kB)
              Installing build dependencies: started
              Installing build dependencies: finished with status 'done'
              Getting requirements to build wheel: started
              Getting requirements to build wheel: finished with status 'done'
              Preparing metadata (pyproject.toml): started
              Preparing metadata (pyproject.toml): finished with status 'done'
            Collecting importlib-metadata
              Using cached importlib_metadata-4.12.0.tar.gz (48 kB)
            ERROR: Exception:
            Traceback (most recent call last):
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
                status = run_func(*args)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/cli/req_command.py", line 205, in wrapper
                return func(self, options, args)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/commands/install.py", line 340, in run
                reqs, check_supported_wheels=not options.target_dir
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
                collected.requirements, max_rounds=try_to_avoid_resolution_too_deep
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
                state = resolution.resolve(requirements, max_rounds=max_rounds)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
                failure_causes = self._attempt_to_pin_criterion(name)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
                criteria = self._get_updated_criteria(candidate)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria
                self._add_to_criteria(criteria, requirement, parent=candidate)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
                if not criterion.candidates:
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
                return bool(self._sequence)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
                return any(self)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
                return (c for c in iterator if id(c) not in self._incompatible_ids)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
                candidate = func()
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/resolution/resolvelib/factory.py", line 220, in _make_candidate_from_link
                version=version,
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 294, in __init__
                version=version,
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
                self.dist = self._prepare()
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
                dist = self._prepare_distribution()
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 299, in _prepare_distribution
                return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/operations/prepare.py", line 487, in prepare_linked_requirement
                return self._prepare_linked_requirement(req, parallel_builds)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/operations/prepare.py", line 560, in _prepare_linked_requirement
                self.build_isolation,
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/operations/prepare.py", line 57, in _get_prepared_distribution
                with req_tracker.track(req):
              File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
                return next(self.gen)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/req/req_tracker.py", line 122, in track
                self.add(req)
              File "/tmp/pip-standalone-pip-eie7jl27/__env_pip__.zip/pip/_internal/req/req_tracker.py", line 92, in add
                raise LookupError(message)
            LookupError: https://files.pythonhosted.org/packages/1a/16/441080c907df829016729e71d8bdd42d99b9bdde48b01492ed08912c0aa9/importlib_metadata-4.12.0.tar.gz#sha256=637245b8bab2b6502fcbc752cc4b7a6f6243bb02b31c5c26156ad103d3d45670 (from https://pypi.org/simple/importlib-metadata/) (requires-python:>=3.7) is already being built: importlib-metadata from https://files.pythonhosted.org/packages/1a/16/441080c907df829016729e71d8bdd42d99b9bdde48b01492ed08912c0aa9/importlib_metadata-4.12.0.tar.gz#sha256=637245b8bab2b6502fcbc752cc4b7a6f6243bb02b31c5c26156ad103d3d45670 (from setuptools_scm)

Wait, why is importlib-metadata trying to install setuptools_scm==7.0.5 when it already has setuptools_scm<7.0.1? Because pip is installing importlib-metadata's build-system.requires dependencies under "build isolation", ignoring the setuptools_scm we've already installed.

But aren't we specifying setuptools_scm<7.0.1 in the same requirements specification as python-dateutil==7.0.1? Yup.

What if we do pip install -c constraints.txt [...] with setuptools_scm<7.0.1 in constraints.txt? Nope, because "generated pip commands do not respect constraints files in outer install command".

So what about PIP_CONSTRAINT=constraints.txt pip install [...]? YES!

What does this mean for us? If we want fully deterministic dependencies—including, for reproducible wheels, fully deterministic build dependencies—then we must pin these either:

in a fully pip-compile --allow-unsafe'd requirements.txt, enforced in CI via PIP_CONSTRAINT=requirements.txt pip install -r requirements.txt; or
in a selective constraints.txt or build-requirements.txt, enforced in CI in the same way.

We started doing this in securedrop in freedomofpress/securedrop#4686. I'll open a follow-up ticket to consider options (1) and (2) for securedrop-client, since—even if #358 let us sidestep this presenting problem under under Python 3.7—we've been bitten by this constraint-enforcement problem in the past and surely will be bitten by it again.

cfm added the bug Something isn't working label Jun 24, 2022

gonzalo-bulnes mentioned this issue Jun 30, 2022

add changelog entry for securedrop-workstation-config 0.2.0 #353

Merged

cfm self-assigned this Jul 5, 2022

cfm mentioned this issue Jul 5, 2022

ci(reprotest-wheels): switch to Debian bullseye #358

Merged

2 tasks

legoktm closed this as completed in #358 Jul 5, 2022

cfm reopened this Jul 7, 2022

cfm closed this as completed Jul 12, 2022

sssoleileraaa mentioned this issue Jul 13, 2022

Document process around building and testing new containers before pushing them to quay.io #363

Closed

cfm mentioned this issue Jul 14, 2022

constrain build requirements for reproducible wheels freedomofpress/securedrop-client#1538

Open

legoktm mentioned this issue Sep 2, 2022

Configure required CI checks in this repo #374

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`reprotest-wheels` stalls installing `python-dateutil` #350

`reprotest-wheels` stalls installing `python-dateutil` #350

cfm commented Jun 24, 2022 •

edited

Loading

legoktm commented Jun 29, 2022

gonzalo-bulnes commented Jun 30, 2022 •

edited

Loading

legoktm commented Jun 30, 2022

gonzalo-bulnes commented Jun 30, 2022 •

edited

Loading

cfm commented Jul 5, 2022

cfm commented Jul 7, 2022

legoktm commented Jul 7, 2022

cfm commented Jul 12, 2022

legoktm commented Jul 13, 2022

cfm commented Jul 13, 2022

cfm commented Jul 13, 2022

cfm commented Jul 14, 2022

reprotest-wheels stalls installing python-dateutil #350

reprotest-wheels stalls installing python-dateutil #350

Comments

cfm commented Jun 24, 2022 • edited Loading

legoktm commented Jun 29, 2022

gonzalo-bulnes commented Jun 30, 2022 • edited Loading

legoktm commented Jun 30, 2022

gonzalo-bulnes commented Jun 30, 2022 • edited Loading

cfm commented Jul 5, 2022

cfm commented Jul 7, 2022

legoktm commented Jul 7, 2022

cfm commented Jul 12, 2022

legoktm commented Jul 13, 2022

cfm commented Jul 13, 2022

cfm commented Jul 13, 2022

cfm commented Jul 14, 2022

`reprotest-wheels` stalls installing `python-dateutil` #350

`reprotest-wheels` stalls installing `python-dateutil` #350

cfm commented Jun 24, 2022 •

edited

Loading

gonzalo-bulnes commented Jun 30, 2022 •

edited

Loading

gonzalo-bulnes commented Jun 30, 2022 •

edited

Loading