Python interpreter download is a bottleneck

[konstin]

When performing an uncached install, the majority of the time is spent on waiting for python to download and unpack. This download is a bottleneck, because nothing else happens in parallel. We should parallelize the python download to run in parallel with package downloads and unpacks, while stalling package builds. We have to add the interpreter info for each interpreter we can download to uv (or make it available online) so we start evaluating wheel tags without having a physical interpreter.

Reproducer:

pyproject.toml
uv.lock

FROM ubuntu:latest
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates
RUN curl -LsSf https://astral.sh/uv/install.sh > /tmp/uv-installer.sh && sh /tmp/uv-installer.sh && rm /tmp/uv-installer.sh
ENV PATH="/root/.cargo/bin/:$PATH"
ADD pyproject.toml pyproject.toml
ADD uv.lock uv.lock
RUN mkdir -p github_wikidata_bot
RUN touch github_wikidata_bot/__init__.py && echo "cache bump"
RUN uv python install 3.12
RUN uv sync

uv python install 3.12 takes 2.5s on my machine, and uv sync takes 1.5s, while only uv sync takes 4s. On a shared server i tried, i get 6.5s for python and 3.5s for the sync. While both use network, disk and io, they are still much faster running in parallel, just like parallel download and unpack for wheels is much faster than sequential wheel install.

[zanieb]

Oo that sounds cool.

[zanieb]

Though wouldn't people have the uv python install cached in their Docker image and make this optimization irrelevant?

[konstin]

The docker is more for illustration purposes, i'm thinking about any uncached deployment enviroment where you also want to install python, but also the first run on a new machine experience of uv.

[konstin]

For docker specifically, i'd recommend, in order:

• Use an image we provide that has python installed
• Run uv python install 3.x in a separate cached step like the apt-get invocations
• Install python through your image's mechanism and have it in a cached layer (make sure build stage and run stage match exactly on their python)

[samypr100]

Use an image we provide that has python installed

🤞 #6053