Multiple sources for dependencies

[jogielis]

Multiple sources for the same dependency

Is there a way to specify multiple sources for a project's dependencies?

Example:
During development I specify my dependencies the 'normal' way and just collect them through pypi ('online').
But during deployment I would also like to have a way to create an environment for a project locally / offline. The dependencies will be collected beforehand by using the pdm-download or pdm-wheel plugins. And all of the dependencies will be available locally on disk, shared between multiple projects. So I would like to be able to install them from that 'local pool'.
A way to achieve this is to make two dependency groups, both defining the same dependencies but with different sources (?). But this is not really desirable since I don't want to define the dependencies twice, possible getting specific versions wrong etc.

Any other way to achieve this? Specify a source with a fallback source maybe?

[pawamoy]

Have you considered specifying another, local index for installation of deps during deployment?

• https://pdm-project.org/latest/usage/config/#configure-the-package-indexes
• PDM_PYPI_URL in https://pdm-project.org/latest/reference/configuration/#available-configurations

[jogielis]

Looks like a good solution, thanks, would be even better if I didn't need to setup a local pypi server for it though. But got it working pretty quick using a pypiserver docker image.

But.. I don't seem to get it configured to use both though:

Adding a source

[[tool.pdm.source]]
name = "private"
url = "http://localhost:6066/simple"
verify_ssl = false

Works when my docker container is running (local pypi server), but when stopped, I would expect it to fallback to pypi.org, but that doesn't seem to happen?

Using pypi.extra.url

$ pdm config pypi.extra.url "http://localhost:6066/simple"

Remove the tool.pdm.source entry, and setting configuration above seems to have no effect?

[jogielis]

I also tried to define two sources, like described here: #2406

[tool.pdm.resolution]
respect-source-order = true

[[tool.pdm.source]]
name = "pypi"
url = "http://localhost:6066/simple"
verify_ssl = false

[[tool.pdm.source]]
name = "fallback"
url = "https://pypi.org/simple"
verify_ssl = true

but it will not continue to the second source after the first one fails:

$ pdm add grpcio 
Adding packages to default dependencies: grpcio
See /tmp/pdm-lock-qazx_gxe.log for detailed debug log.
[ConnectionError]: HTTPConnectionPool(host='localhost', port=6066): Max retries exceeded with url: /simple/grpcio/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3ff71defa0>: Failed to establish a new connection: [Errno 111] Connection refused'))
WARNING: Add '-v' to see the detailed traceback

[jogielis]

Was looking into why this wasn't working, and found it was easily fixed by catching requests.exceptions.ConnectionError, which was already proposed by PR frostming/unearth#54 but denied (see discussion #1945).

Following this discussion (#1945), adapting the global PDM configuration is probably the best way forward.