Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for named and explicit indexes #7481

Merged
merged 2 commits into from
Oct 15, 2024
Merged

Conversation

charliermarsh
Copy link
Member

@charliermarsh charliermarsh commented Sep 18, 2024

Summary

This PR adds a first-class API for defining registry indexes, beyond our existing --index-url and --extra-index-url setup.

Specifically, you now define indexes like so in a uv.toml or pyproject.toml file:

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"

You can also provide indexes via --index and UV_INDEX, and override the default index with --default-index and UV_DEFAULT_INDEX.

Index priority

Indexes are prioritized in the order in which they're defined, such that the first-defined index has highest priority.

Indexes are also inherited from parent configuration (e.g., the user-level uv.toml), but are placed after any indexes in the current project, matching our semantics for other array-based configuration values.

You can mix --index and --default-index with the legacy --index-url and --extra-index-url settings; the latter two are merely treated as unnamed [[tool.uv.index]] entries.

Index pinning

If an index includes a name (which is optional), it can then be referenced via tool.uv.sources:

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"

[tool.uv.sources]
torch = { index = "pytorch" }

If an index is marked as explicit = true, it can only be used via such references, and will never be searched implicitly:

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"
explicit = true

[tool.uv.sources]
torch = { index = "pytorch" }

Indexes defined outside of the current project (e.g., in the user-level uv.toml) can not be explicitly selected.

(As of now, we only support using a single index for a given tool.uv.sources definition.)

Default index

By default, we include PyPI as the default index. This remains true even if the user defines a [[tool.uv.index]] -- PyPI is still used as a fallback. You can mark an index as default = true to (1) disable the use of PyPI, and (2) bump it to the bottom of the prioritized list, such that it's used only if a package does not exist on a prior index:

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"
default = true

Name reuse

If a name is reused, the higher-priority index with that name is used, while the lower-priority indexes are ignored entirely.

For example, given:

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu121"

[[tool.uv.index]]
name = "pytorch"
url = "https://test.pypi.org/simple"

The https://test.pypi.org/simple index would be ignored entirely, since it's lower-priority than https://download.pytorch.org/whl/cu121 but shares the same name.

Closes #171.

Future work

  • Users should be able to provide authentication for named indexes via environment variables.
  • uv add should automatically write --index entries to the pyproject.toml file.
  • Users should be able to provide multiple indexes for a given package, stratified by platform:
[tool.uv.sources]
torch = [
  { index = "cpu", markers = "sys_platform == 'darwin'" },
  { index = "gpu", markers = "sys_platform != 'darwin'" },
]
  • Users should be able to specify a proxy URL for a given index, to avoid writing user-specific URLs to a lockfile:
[[tool.uv.index]]
name = "test"
url = "https://private.org/simple"
proxy = "http://<omitted>/pypi/simple"

@charliermarsh charliermarsh added the enhancement New feature or improvement to existing functionality label Sep 18, 2024
@charliermarsh
Copy link
Member Author

This still needs docs (beyond the inline documentation), but I'd like to align on the semantics described in the PR summary first.

// /// structured as a flat list of distributions (e.g., `--find-links`). In both cases, indexes
// /// can point to either local or remote resources.
// #[serde(default)]
// pub r#type: IndexKind,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left this in for now but can remove before merging. Eventually I want this to support --find-links.

@charliermarsh charliermarsh force-pushed the charlie/index-api branch 4 times, most recently from 98c909f to e6f9d9a Compare September 18, 2024 02:14
@zanieb
Copy link
Member

zanieb commented Sep 18, 2024

Any indexes defined via [[tool.uv.index]] take priority over any indexes defined via --index-url or --extra-index-url.

(This is problematic, since as of now, there's no way to provide a [[tool.uv.index]] on the command line, so you can never override in-file configuration with command-line configuration.)

Yeah this seems wrong. Why is it this way?

Do we need an explicit explicit tag or can we just use the presence of a name? I guess the name is useful for providing credentials externally so... I guess the explicit tag is important.

Do we set the explicit tag during a uv add --index-url <url> <pkg> operation?

How does index pinning work for transitive dependencies?

How can we teach the PyPI fallback behavior? Are there any bad user experiences where we could suggest adding default = true to their index definition?

@charliermarsh
Copy link
Member Author

charliermarsh commented Sep 18, 2024

Yeah this seems wrong. Why is it this way?

Well... we could just invert the priority, if that's what you mean (such that the new index stuff comes last, and the legacy arguments come first). Then the CLI arguments would work as expected. As-is, though, we don't have a way to differentiate between indexes passed on the command line (with --index-url or --extra-index-url) and indexes defined in a file via index-url and extra-index-url... So, like, we can't have index-url from the uv.toml be lower priority than tool.uv.index, but --index-url be higher priority. Alternatively, we could add a new command-line argument (--index?) for these tool.uv.index sources, which would also solve the problem?

Do we need an explicit explicit tag or can we just use the presence of a name? I guess the name is useful for providing credentials externally so... I guess the explicit tag is important.

Yeah roughly this.

Do we set the explicit tag during a uv add --index-url operation?

We don't as of this PR... We can though. I think we probably should? I guess by default we add the index, and make it explicit for that package? The unfortunate thing is that we need a name for the index in that case (as we discussed on Discord). Alternatively, we could just add a tool.uv.index, and not make it explicit.

How does index pinning work for transitive dependencies?

It doesn't have any effect on transitive dependencies. I don't know that it can or should, honestly, because transitive dependencies can be required from multiple different first-party dependencies that could come from different indexes. I think this would be extremely hard to implement correctly and could lead to confusing behavior.

@inflation
Copy link

I prefer to make explicit=true the default behavior, since the first foot gun in the current situation after adding extra-index-url, are packages in the pytorch channel taking precedence over pypi.

@smphhh
Copy link

smphhh commented Sep 18, 2024

How does index pinning work for transitive dependencies?

It doesn't have any effect on transitive dependencies. I don't know that it can or should, honestly, because transitive dependencies can be required from multiple different first-party dependencies that could come from different indexes. I think this would be extremely hard to implement correctly and could lead to confusing behavior.

Did you consider allowing pinning packages to indexes using glob-patterns as mentioned here? AFAIK that is the only solution for pinning transitive dependencies that is straightforward enough but still useful for a set of real use-cases.

@charliermarsh
Copy link
Member Author

Did you consider allowing pinning packages to indexes using glob-patterns as mentioned #171 (comment)? AFAIK that is the only solution for pinning transitive dependencies that is straightforward enough but still useful for a set of real use-cases.

There's nothing stopping us from supporting that in addition to the schema described above. It strikes me as somewhat backwards but I understand why it's useful. My only concern is that we're complicating the schema and creating two ways to assign a package to an index.

@corleyma
Copy link

@charliermarsh I know it's painful/stupid/messy, but the regex assignment is still really useful for handling transitive dependencies in corporate environments. I don't think there are easy alternatives given the default assumption that python wants to make about indices being equivalent.

@zanieb
Copy link
Member

zanieb commented Sep 20, 2024

I wonder if we should have a separate schema for pinning transitive packages to a defined index? It could allow globs as well. I'm not sure what all the trade-offs are.

@charliermarsh
Copy link
Member Author

@zanieb -- That's a good call. Instead of putting this on the index schema, we could have a separate table for assigning packages to named indexes.

@charliermarsh
Copy link
Member Author

I'll probably be pushing to this branch a lot over the next few days, so you may want to unsubscribe if the notifications are annoying!

@charliermarsh charliermarsh force-pushed the charlie/index-api branch 7 times, most recently from 3208ab0 to 518f3b3 Compare September 27, 2024 01:39
@szobov
Copy link

szobov commented Oct 7, 2024

Hey @charliermarsh
Thanks a lot for the work you're doing to improve python's ecosystem.
I'm very much looking forward for this feature since I'm currently blocked by it.

I wanted to ask what are the steps to get it merged and released and if I can somehow assist you?

I tested it locally by installing uv from the branch and it worked smoothly 🎉

details on the setup
```
$ uv version
uv 0.4.18 (f9ee2a9e2 2024-10-03)

$ cat pyproject.toml
[project]
name = "inference"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "numpy>=2.1.2",
    "torch==2.4.0+cu118",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.uv.sources]
torch = { index = "pytorch" }

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu118"
explicit = true

$ cat uv.lock
....
[[package]]
name = "torch"
version = "2.4.0+cu118"
source = { registry = "https://download.pytorch.org/whl/cu118" }
dependencies = [
    { name = "filelock" },
    { name = "fsspec" },
    { name = "jinja2" },
    { name = "networkx" },
    { name = "nvidia-cublas-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "nvidia-cuda-cupti-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "nvidia-cuda-nvrtc-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "nvidia-cuda-runtime-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "nvidia-cudnn-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "nvidia-cufft-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "nvidia-curand-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "nvidia-cusolver-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "nvidia-cusparse-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "nvidia-nccl-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "nvidia-nvtx-cu11", marker = "platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "setuptools" },
    { name = "sympy" },
    { name = "triton", marker = "python_full_version < '3.13' and platform_machine == 'x86_64' and platform_system == 'Linux'" },
    { name = "typing-extensions" },
]
wheels = [
    { url = "https://download.pytorch.org/whl/cu118/torch-2.4.0%2Bcu118-cp312-cp312-linux_x86_64.whl", hash = "sha256:d38bd98e5faf9565f04d2b59a481cf576cdf4078444cdf24868fbf5ad685dc4d" },
    { url = "https://download.pytorch.org/whl/cu118/torch-2.4.0%2Bcu118-cp312-cp312-win_amd64.whl", hash = "sha256:bd16a12a003fe58276d7b7394a3760d576eb8dbfb4bacea025eb52a1eaf5172b" },
]
```

Thanks in advance 🙏

@charliermarsh
Copy link
Member Author

I’m planning to release it early next week. Sorry for the delay — I’m on vacation today, then at EuroRust later in the week.

@pythonbyte
Copy link

Hey @charliermarsh

Amazing work, would be amazing if in this PR we could have the
uv add should automatically write --index entries to the pyproject.toml file.

That would help my team and me not to add a library to write/dump the data.toml file 🙏🏽

@odie5533
Copy link
Contributor

Hey @charliermarsh

Amazing work, would be amazing if in this PR we could have the uv add should automatically write --index entries to the pyproject.toml file.

That would help my team and me not to add a library to write/dump the data.toml file 🙏🏽

See PRs #7746 and #7747

@pythonbyte
Copy link

Hey @charliermarsh
Amazing work, would be amazing if in this PR we could have the uv add should automatically write --index entries to the pyproject.toml file.
That would help my team and me not to add a library to write/dump the data.toml file 🙏🏽

See PRs #7746 and #7747

That's awesome, thank you so much, really excited about these features 😄

@charliermarsh charliermarsh force-pushed the charlie/index-api branch 2 times, most recently from f1be5d9 to 4b0e753 Compare October 15, 2024 21:40
@charliermarsh charliermarsh merged commit 5b39177 into main Oct 15, 2024
61 checks passed
@charliermarsh charliermarsh deleted the charlie/index-api branch October 15, 2024 22:24
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Oct 18, 2024
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.4.22` -> `0.4.24` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.4.24`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0424)

[Compare Source](astral-sh/uv@0.4.23...0.4.24)

##### Bug fixes

-   Fix Python executable name in Windows free-threaded Python distributions ([#&#8203;8310](astral-sh/uv#8310))
-   Redact index credentials from lockfile sources ([#&#8203;8307](astral-sh/uv#8307))
-   Respect `UV_INDEX_` rather than `UV_HTTP_BASIC_` as documented ([#&#8203;8306](astral-sh/uv#8306))
-   Improve sources deserialization errors ([#&#8203;8308](astral-sh/uv#8308))

##### Documentation

-   Correct pytorch-to-torch reference in docs ([#&#8203;8291](astral-sh/uv#8291))

### [`v0.4.23`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0423)

[Compare Source](astral-sh/uv@0.4.22...0.4.23)

This release introduces a revamped system for defining package indexes, as an alternative to the existing pip-style
`--index-url` and `--extra-index-url` configuration options.

You can now define named indexes in your `pyproject.toml` file using the `[[tool.uv.index]]` table:

```toml
[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cpu"
```

Packages can be pinned to a specific index via `tool.uv.sources`, to ensure that a given package is installed from the
correct index. For example, to ensure that `torch` is *always* installed from the `pytorch` index:

```toml
[tool.uv.sources]
torch = { index = "pytorch" }

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cpu"
```

Indexes can also be marked as `explicit = true` to prevent packages from being installed from that index
unless explicitly pinned. For example, to ensure that `torch` is installed from the `pytorch` index, but all other
packages are installed from the default index:

```toml
[tool.uv.sources]
torch = { index = "pytorch" }

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
```

To define an additional index outside a `pyproject.toml` file, use the `--index` command-line argument
(or the `UV_INDEX` environment variable); to replace the default index (PyPI), use the `--default-index` command-line
argument (or `UV_DEFAULT_INDEX`).

These changes are entirely backwards-compatible with the deprecated `--index-url` and `--extra-index-url` options,
which continue to work as before.

See the [Index](https://docs.astral.sh/uv/configuration/indexes/) documentation for more.

##### Enhancements

-   Add index URLs when provided via `uv add --index` or `--default-index` ([#&#8203;7746](astral-sh/uv#7746))
-   Add support for named and explicit indexes ([#&#8203;7481](astral-sh/uv#7481))
-   Add templates for popular build backends ([#&#8203;7857](astral-sh/uv#7857))
-   Allow multiple pinned indexes in `tool.uv.sources` ([#&#8203;7769](astral-sh/uv#7769))
-   Allow users to incorporate Git tags into dynamic cache keys ([#&#8203;8259](astral-sh/uv#8259))
-   Pin named indexes in `uv add` ([#&#8203;7747](astral-sh/uv#7747))
-   Respect named `--index` and `--default-index` values in `tool.uv.sources` ([#&#8203;7910](astral-sh/uv#7910))
-   Update to latest PubGrub version ([#&#8203;8245](astral-sh/uv#8245))
-   Enable environment variable authentication for named indexes ([#&#8203;7741](astral-sh/uv#7741))
-   Avoid showing lower-bound warning outside of explicit lock and sync ([#&#8203;8234](astral-sh/uv#8234))
-   Improve logging during lock errors ([#&#8203;8258](astral-sh/uv#8258))
-   Improve styling of `requires-python` warnings ([#&#8203;8240](astral-sh/uv#8240))
-   Show hint in resolution failure on `Forbidden` (`403`) or `Unauthorized` (`401`) ([#&#8203;8264](astral-sh/uv#8264))
-   Update to latest `cargo-dist` version (includes new installer features) ([#&#8203;8270](astral-sh/uv#8270))
-   Warn when patch version in `requires-python` is implicitly `0` ([#&#8203;7959](astral-sh/uv#7959))
-   Add more context on client errors during range requests ([#&#8203;8285](astral-sh/uv#8285))

##### Bug fixes

-   Avoid writing duplicate index URLs with `--emit-index-url` ([#&#8203;8226](astral-sh/uv#8226))
-   Fix error leading to out-of-bound panic in `uv-pep508` ([#&#8203;8282](astral-sh/uv#8282))
-   Fix managed distributions of free-threaded Python on Windows ([#&#8203;8268](astral-sh/uv#8268))
-   Fix selection of free-threaded interpreters during default Python discovery ([#&#8203;8239](astral-sh/uv#8239))
-   Ignore sources in build requirements for non-source trees ([#&#8203;8235](astral-sh/uv#8235))
-   Invalid cache when adding lower bound to lockfile ([#&#8203;8230](astral-sh/uv#8230))
-   Respect index priority when storing credentials ([#&#8203;8256](astral-sh/uv#8256))
-   Respect relative paths in `uv build` sources ([#&#8203;8237](astral-sh/uv#8237))
-   Narrow what the pip3.<minor> logic drops from entry points. ([#&#8203;8273](astral-sh/uv#8273))

##### Documentation

-   Add some additional notes to `--index-url` docs ([#&#8203;8267](astral-sh/uv#8267))
-   Add upgrade note to README ([#&#8203;7937](astral-sh/uv#7937))
-   Remove note that "only a single source may be defined for each dependency" ([#&#8203;8243](astral-sh/uv#8243))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
@danieleades
Copy link
Contributor

does this support, or is support planned for, using API keys for authentication against private registries (ie artifactory) rather than using username/password?

@zanieb
Copy link
Member

zanieb commented Oct 22, 2024

You can use an API key in the PASSWORD variable — it works for anything that supports HTTP Basic Authentication.

@danieleades
Copy link
Contributor

You can use an API key in the PASSWORD variable — it works for anything that supports HTTP Basic Authentication.

thanks for the support!

i'm trying this now and seeing issues. For context, i have a project which has one dependency from a private artifactory registry.

with or without credentials exported to the environment i see the following output:

uv sync
  × No solution found when resolving dependencies:
  ╰─▶ Because my-private-package was not found in the package registry and your project depends on my-private-package>=2.0.0,<2.1.dev0, we can conclude that your project's requirements are unsatisfiable.

      hint: my-private-package was requested with a pre-release marker (e.g., sb-qag-sphinx>=2.0.0,<2.1.dev0), but pre-releases weren't enabled (try: `--prerelease=allow`)

the dependencies are specified as follows:

dependencies = [
    "my-private-package~=2.0.0",
    # ...
]

[tool.uv.sources]
my-private-package = {index = "mycompany-py-release"}

[[tool.uv.index]]
name = "artifactory-pypi"
url = "https://artifactory.mycompany.com/artifactory/api/pypi/pypi/simple"
default = true

[[tool.uv.index]]
# Requires:
name = "mycompany-py-release"
url = "https://artifactory.mycompany.com/artifactory/api/pypi/my-private-package/simple"
explicit = true

with

export UV_INDEX_MYCOMPANY_PY_RELEASE_USERNAME={username}
export UV_INDEX_MYCOMPANY_PY_RELEASE_PASSWORD={artifactory token}

@zanieb
Copy link
Member

zanieb commented Oct 22, 2024

You should open a new issue for this and we can discuss there

@dariocurr
Copy link

  • Users should be able to specify a proxy URL for a given index, to avoid writing user-specific URLs to a lockfile:
[[tool.uv.index]]
name = "test"
url = "https://private.org/simple"
proxy = "http://<omitted>/pypi/simple"

Any news about this feature? This is critical for some users

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or improvement to existing functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for pinning a package to a specific index