How to cache Poetry virtual environments in GitLab CI/CD?

[nlykkei]

I want to cache Poetry virtual environments between builds of my Python application. I have configured the pipeline so that the virtual environment is recreated every time pyproject.toml changes (poetry.lock is not comitted):

default:
  cache:
    - key:
        files:
        - lego/pyproject.toml
      paths:
        - lego/.venv
        - artifacts/build/layer/build/.venv
    - paths:
        - artifacts/pypoetry

The virtual environment is created as follows:

poetry config virtualenvs.in-project true
poetry install

This works locally, but it fails in GitLab CI/CD on the second pipeline run:

$ LOG_LEVEL=ERROR poetry run pytest tests -n auto -v --cov=./src --junitxml=lego-report.xml

ImportError while loading conftest '/builds/5565557/lddpro-bff/lego/tests/conftest.py'.
tests/conftest.py:22: in <module>
   from lego.common.aws import S3, EventBridge, SSM, Secrets
E ModuleNotFoundError: No module named 'lego'

The reason is that pytest cannot locate the root package (lego). However, I have verified that the root package is located in the virtual environment (.venv):

$ cat .venv/lib/python*/site-packages/lego.pth
/builds/5565557/lddpro-bff/lego/src

Here lego.pth includes the current job ID in the path.

How can I properly cache Poetry virtual environments between CI/CD pipeline runs?

[Kashemir001]

If the absolute path to the .venv directory changes, it kinda breaks, throwing ModuleNotFoundError even if poetry install command says all the dependecies are OK. You seem to be using Shell executor for Gitlab runner (there is a mutating job ID in your absolute path), that's why it breaks for you on subsequent runs.
With Shell runner being used, I think poetry global cache for dependencies should always kick in for your jobs, so poetry does not need to spend time downloading dependencies each run, only to install them from local files, which I guess should be quick. Does poetry install still takes a lot of time in you CI jobs?
If using Docker executor, the path to the .venv should be somewhat consistent, ie. /builds/groupname/projectname/ (so unless your project path changes), and then .venv cache works - I'm in this thread because I thought about caching it as well :) Although it feels flimsy and prone to errors so far. If you look at the Gitlab CI caching examples, they keep cache for package managers and not project installed dependencies. So probably just having package manager cache would be a better idea?

[ferreira68]

I don't know about GitLab, but the following works well for me in GitHub:

      - name: Install Poetry
        uses: abatilo/actions-poetry@v2
        with:
          poetry-version: ${{ matrix.poetry-version }}

      - name: Setup local virtual environment
        run: |
          poetry config virtualenvs.create true --local
          poetry config virtualenvs.in-project false --local

      - name: Setup Poetry Caching
        uses: actions/cache@v3
        with:
          path: ~/.cache/pypoetry/virtualenvs
          key: >
            ${{ steps.os_info.outputs.type }}-Python-${{ matrix.python-version }}-Poetry-${{ matrix.poetry-version }}-${{ hashFiles('**/poetry.lock') }}
          restore-keys: >
            ${{ steps.os_info.outputs.type }}-Python-${{ matrix.python-version }}-Poetry-${{ matrix.poetry-version }}-

The steps.os_info.outputs.type comes from a previous step, but otherwise everything else is pretty much boilerplate. I have discovered that the poetry config virtualenvs.in-project false --local generally works more reliably than including the install in the project root.

I hope this helps.