Supporting different package source urls for development and build

[abn]

In other words, the source is used even though the packages are locked for another source? That sound like a bug. I have to check if I can reproduce that later. Or do you use any Poetry plugins that alter the behavior?

I have the specific need to use PyPI in development and a private repository in the Jenkins pipeline. If poetry will not offer me this necessity, then I will need to reevaluate the package manager our team uses for our projects.

Originally posted by @aabmets in #9042 (comment)

[abn]

@aabmets if there is a scenario that should be better supported by poetry (eg: installation inside air-gaped or restricted network environments), we should probably raise a new issue for that. (See also: #5317, #1632)

The issue, as @radoering correctly points out, is that in your sequence of actions you effectively disregard the lockfile by installing a dependency from a source that differs from the one that the project was locked against. Having the checksums should allow us to verify we have the right files even if they are sourced form a different server.

In a lot of cases it makes sense to be able to "swap out" the source without having to regenerate the lock file. However, there are tradeoffs in your security posture there.

I suspect what we need is the ability to define proxies or mirrors for use in the runtime environment. It is also feasible that we provide the ability for install to ignore lock file inconsistencies explicitly (but I am not sure if that is any different from simply adding an extra step of poetry lock --no-update after poetry source add. We should discuss this in a dedicated issue.

Here is an example how you could retain your workflow with Poetry >= 1.8.0. This is effectively working around #8737. But obviously this means your lock file will be tainted.

podman run --rm -i --entrypoint bash docker.io/python:3.12 <<EOF
set -xe
python -m pip install --disable-pip-version-check -q poetry

poetry config virtualenvs.in-project true
poetry config virtualenvs.options.no-pip true
poetry config virtualenvs.options.no-setuptools true

poetry new demo
pushd demo
poetry add cowsay

rm -rf .venv

poetry source add --priority=primary pypi-test https://test.pypi.org/simple/

# this will fail because of drift
poetry install || :

poetry lock --no-update
poetry install

# you should see the packages installed
poetry run pip list
EOF

console.log

+ python -m pip install --disable-pip-version-check -q poetry
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
+ poetry config virtualenvs.in-project true
+ poetry config virtualenvs.options.no-pip true
+ poetry config virtualenvs.options.no-setuptools true
+ poetry new demo
Created package demo in demo
+ pushd demo
/demo /
+ poetry add cowsay
Creating virtualenv demo in /demo/.venv
Using version ^6.1 for cowsay

Updating dependencies
Resolving dependencies...

Package operations: 1 install, 0 updates, 0 removals

  - Installing cowsay (6.1)

Writing lock file
+ rm -rf .venv
+ poetry source add --priority=primary pypi-test https://test.pypi.org/simple/
Adding source with name pypi-test.
+ poetry install
Creating virtualenv demo in /demo/.venv
Installing dependencies from lock file

pyproject.toml changed significantly since poetry.lock was last generated. Run `poetry lock [--no-update]` to fix the lock file.
+ :
+ poetry lock --no-update
Resolving dependencies...

Writing lock file
+ poetry install
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  - Installing cowsay (6.1)

Installing the current project: demo (0.1.0)
+ poetry run pip list
Package Version Editable project location
------- ------- -------------------------
cowsay  6.1
demo    0.1.0   /demo

[abn]

@aabmets can you confirm if this scenario replicates your use case functionally?

[aabmets]

@abn > @aabmets can you confirm if this scenario replicates your use case functionally?
I guess it does, yes. The "air-gap" in my specific use case is more like a "SonarType Nexus Intermediary for Automatic CVE Detection".
The private repository just proxies the contents of PyPI and checks for CVE-s.