profile/debug poetry install

[till]

Hi!

I am using poetry (v1.7.0) to maintain tool versions (ansible and openstack client) in a project.

The project is a workspace and setup inside a container (to ensure all versions stay as they should). Aside from installing the tools, I don't use python — in a nutshell poetry install and then poetry run ansible-playbook etc..

My install command:

poetry install \
    --no-ansi -vvv \
    --no-interaction \
    --only main --no-root \

My pyproject.toml:

[tool.poetry]
name = "bootstrap"
version = "0.1.0"
description = ""
authors = []
license = ""

[tool.poetry.dependencies]
python = "~3.11"
ansible = "~5.6.0"
python-openstackclient = "~5.8.0"
PyYaml = "!=5.4.*,!=6.0.0" # These versions of don't work with Cython 3, see https://github.com/yaml/pyyaml/issues/724

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

The log of the install looks similar to this:

#34 2.991 Installing dependencies from lock file
#34 3.177 
#34 3.177 Finding the necessary packages for the current system
#34 3.318 
#34 3.318 Package operations: 55 installs, 0 updates, 0 removals
#34 3.318 
#34 3.319   • Installing certifi (2024.2.2)
#34 3.321   • Installing charset-normalizer (3.3.2)
#34 3.324   • Installing idna (3.6)
#34 3.328   • Installing pbr (6.0.0)
#34 3.333   • Installing pycparser (2.21)
#34 3.334   • Installing urllib3 (2.2.1)
#34 3.338   • Installing wrapt (1.16.0)
#34 5.916   • Installing attrs (23.2.0)
#34 5.920   • Installing cffi (1.16.0)
#34 5.927   • Installing debtcollector (2.5.0)
#34 5.927   • Installing decorator (5.1.1)
#34 5.936   • Installing iso8601 (2.1.0)
#34 5.939   • Installing netaddr (1.2.1)
#34 5.942   • Installing netifaces (0.11.0)
#34 5.949   • Installing oslo-i18n (6.2.0)
#34 5.954   • Installing pyparsing (3.1.1)
#34 5.954   • Installing jsonpointer (2.4)
#34 5.968   • Installing packaging (23.2)
#34 5.981   • Installing os-service-types (1.7.0)
#34 6.520 Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
#34 6.534 Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
#34 6.764   • Installing pyperclip (1.8.2)
#34 7.083   • Installing pytz (2024.1)
#34 7.148   • Installing pyyaml (6.0.1)
#34 7.697   • Installing requests (2.31.0)
...

Now, when I update one of the components in my project and run poetry install, it often takes a long time (at least 2 minutes, sometimes 5 to 10). I've read a lot of the issues regarding performance and figured I'd ask here before I create yet another one.

Long story short — is there documentation to describe best practices as to how to add dependencies?

• I've looked at narrowing my constraints, but it didn't help much, etc..
• I've updated Python, OS (in the container), Poetry, ...
• I've also tried to increase the connection pool (poetry config installer.max-workers 100), but it doesn't seem like that is picked up.
• I was also wondering if there is a setting to profile an install run, just to see what takes the longest to be aware of it, or maybe mirror it, etc..

Anyway, appreciate what this project is doing. At least for myself who's not a Python developer, this seemed more in line with the experience in other languages. So kudos for that.

Thank you!

[radoering]

Now, when I update one of the components in my project

How do you update the component? Do you edit pyproject.toml manually? Do you use a poetry command?

[till]

poetry update

But I'd like to focus on a a full install — e.g. when I rebuild the container. Or imagine you clone the project to a new machine and then do poetry install.

[abn]

@till can provide the Containerfile or Dockerfile you use to build the container?

Here are a few pointers since you mentioned this is within a container.

1. Try disabling v6 networking and or checking DNS etc are working within the container happily.
2. Make sure you have enough cores (rather CPU threads) in the container to serve your installer workers.
3. Ensure your cache (~/.cache/pypoetry/) is preserved across runs (or in your case, builds).

[abn]

installer.max-workers 100

Do you really have 100 CPU threads to give your Poetry process?

I'd recommend you reduce that to whatever CPU numbers the docker host has access to or minus one that. Beyond that it is diminishing returns.

Also can't really comment on why your tools do not run without logs. You might want to check if the packages installed correctly first.

You can also add tighter constraints for openstack in your own project to improve locking performance.

[till]

@abn What kind of logs would you like to see?

I did the poetry install in the Dockerfile, so during build and the image builds. When I docker run it, the tools were not there. I removed the --mount=type=cache,target=/home/user/.cache/pypoetry,uid=${USER_UID},gid=${USER_GID} from the Dockerfile, build again and the tools are installed.

When I check Docker desktop, I see it can use 8 CPU cores (it's an Intel Macbook Air) — not sure how many threads that equals. But I'll try to re-run this with max-workers set to eight then?

Generally, is there a way to profile the run to see what actually takes the longest (download, resolver, etc.)?

And you you have any suggestions how to lock down the dependencies from the root project?

[abn]

The easiest one to start with is poetry install -vvv. It is pretty verbose.

Try setting https://python-poetry.org/docs/configuration/#virtualenvsin-project. I think your virtual env is being created inside the cache mount and not preserved into your image layers. This will avoid that by placing the venv alongside the project source.

As for max workers, you can let poetry manage it or set it to 8.

Also, fyi you can configure these via env vars as well, might be cleaner. :)

[abn]

As for constraining dependencies for the lock file, try creating an optional group for transitive dependency constraints.

https://python-poetry.org/docs/managing-dependencies#optional-groups

[till]

Thanks again, @abn! I'll try to get back to this on Monday and will update.