Bug in CircleCI tests for python 3.6: caching issue

[baileythegreen]

Problem Overview

The CircleCI tests for PR #338, adding ANIb(lastall) are now failing on python 3.6 (which is new), with this output:

#!/bin/bash -eo pipefail
pip install --upgrade pip setuptools virtualenv
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -r requirements-pip.txt
pip install -r requirements-pyqt-pip.txt

/home/circleci/.pyenv/versions/3.10.1/bin/python3.10: /lib/x86_64-linux-gnu/libm.so.6: version 'GLIBC_2.29' not found (required by /home/circleci/.pyenv/versions/3.10.1/lib/libpython3.10.so.1.0)
/home/circleci/.pyenv/versions/3.10.1/bin/python3.10: /lib/x86_64-linux-gnu/libpthread.so.0: version 'GLIBC_2.30' not found (required by /home/circleci/.pyenv/versions/3.10.1/lib/libpython3.10.so.1.0)
/home/circleci/.pyenv/versions/3.10.1/bin/python3.10: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.28' not found (required by /home/circleci/.pyenv/versions/3.10.1/lib/libpython3.10.so.1.0)

Exited with code exit status 1
CircleCI received exit code 1

which looks like the addition of python 3.10 is affecting the cache.

Looking at the cache that is loaded:

Unarchiving cache...
Skipping writing "home/circleci/.pyenv/bin/pyenv" - symlink ../libexec/pyenv /home/circleci/.pyenv/bin/pyenv: file exists

update apt-get
53s

it becomes clear that this is the case. The same is seen in the test runs for other versions of python. I would guess that this might also represent a race condition—whichever version is loaded first is the one that is cached, but this is currently just a hunch.

Lines 84–86 of .circleci/config.yml are where this is saved; is there a reason for this (perhaps the package takes a long time to load/install)?

      - save_cache:
          paths:
            - /home/circleci/.pyenv

Question

@widdowquinn Should we be using a different version of pyenv for each python installation, or do we only want to change this for the point of failure?

[widdowquinn]

Please take a look at the current config file in master - especially the {{ .Environment.CIRCLE_JOB }} variable. This allows generation of a different cache for each job (which is labelled by Python version). It might help solve this issue.

[widdowquinn]

The issue is not due to addition of 3.10 to the testing regime, nor is it a race condition. See, for example, these tests from two weeks ago which include 3.10. The addition of 3.10 did not affect the caches or cause tests to fail.

This error shows up in the output of the failing weekly test from last night, and also a rerun attempt from this morning:

No cache is found for key: pyani-dependencies-pip-test-3_6-master-da3jx1LEx0gkAIgefJ5iMRLVM7hdjqi9JjBD3OvOMPU=-L0fQL8O1iX_M7RxM4JztjAKBr1MQZkPTPjPEEDWFUsg=-9d74ucTAWWIVfvhVDcd7BefcSv0vkZ6HoBAR7SgHFGU=-d9izQU5kl66+eu+mVmhbBpQrXnf_WmOX7GuPDo_aEHs=-7aGwkbVHgb1mq7X3jvzfi0HG+BpEQnskhs+SPAyb7vk=-46ue63j8fWJKEaZqizie8cVoT6U3WSakueY_GUGmoIA=-m+9vCcft76CH2_rNJE7yS7OyRr9a8NXsXSG865RBfS0=
Found a cache from build 3125 at pyani-dependencies-pip-

which shows that the caching information for test-3_6 (encoded in {{ .Environment.CIRCLE_JOB }}) was not present, so CircleCI picked up a different cache from build 3125 - possibly the one from the failing tests immediately preceding it, which also reused an earlier non-specific cache - possibly the source of the incompatibility @baileythegreen saw in the anib123 test and describes above.

I have removed the fallback option in CircleCI (which allows the pickup of non-specific caches that match the wildcard) with commit 6656761. This prevents reuse of any other than the exactly matching cache ID. The Python 3.6 tests now pass - as do the other test runs.

The documentation for CircleCI dependency caching can be found here.

[widdowquinn]

To answer this question:

Lines 84–86 of .circleci/config.yml are where this is saved; is there a reason for this (perhaps the package takes a long time to load/install)?

      - save_cache:
          paths:
            - /home/circleci/.pyenv

I think there is a misunderstanding about the meaning of that field. The path field in save_cache declares which paths (directories or files) are stored in the cache - further details can be found here. Most, if not all, of the Python dependency files we install are under that path and - yes - caching the Python dependencies does save time in the tests.

[widdowquinn]

@widdowquinn Should we be using a different version of pyenv for each python installation, or do we only want to change this for the point of failure?

Hopefully the comments above are clear enough, but to answer directly: no.

CircelCI uses pyenv to manage Python versions. By running a different Python version in each test job, we are already using a different instance of pyenv on a different image, for each Python version.