update-python-resources should support non-PyPI indexes

[ctaintor]

Verification

• This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature

I'd like to have --update-python-resources support --extra-pacakges where the package is a URL, not a name. A call would look like:

brew update-python-resources Formula/a.rb --include-packages "https://example.com/...../0.2.1.tar.gz#SHA256"

What is the motivation for the feature?

my main motivation is to make minimal changes to allow --update-python-resource work when some of the main package dependencies are not in PyPi. In my case, I have a main package (let's call it package A) which depends on a private package B and that package B depends on another private package C. A, B and C all also depend on packages in PyPi.

How will the feature be relevant to at least 90% of Homebrew users?

It's unlikely that this would be useful to 90% of Homebrew users but it will be relevant to users who are using private packages.

What alternatives to the feature have been considered?

An alternative is to adjust --update-python-resources support non-PyPi sources for the pypi_info call – by e.g. taking a parameter or using PIP_INDEX_URL. While this sounds easy, the core problem with supporting this is that package sources are not required to support the JSON API, so one of the most popular private package sources (Artifactory) does not support the JSON api. This means an alternative is to:

• modify --update-python-resources to accept a simple api endpoint as a parameter (or similar)
• modify --update-python-resources to be ale to use the JSON API or the simple api

the work to support --extra-packages which are URLs simple seems a little easier and potentially more flexible.

[ctaintor]

I wrote a sort of brute-force patch of this just to see what was necessary and if this made any sense. Seems to work, however this prevents running it a 2nd time (unless you delete the "other" resources).

[carlocab]

I wrote a sort of brute-force patch of this just to see what was necessary and if this made any sense. Seems to work, however this prevents running it a 2nd time (unless you delete the "other" resources).

Given this, it seems more generally useful if:

• your non-PyPI resources declared livecheck blocks
• bump-formula-pr knew to update resources with livecheck blocks

The latter has been on our nice-to-have-list for a while now. There is some support for the former, but I'm not sure it's complete.

[MikeMcQuaid]

• bump-formula-pr knew to update resources with livecheck blocks

Agree that I think this is what should be built here.

[ctaintor]

in my exact case, simply supporting non-JSON-api repos would solve the problem. should I look at doing this instead? It just means pulling in another dep (REXML?) and then having two implementations of pypi_info i believe

[carlocab]

in my exact case, simply supporting non-JSON-api repos would solve the problem.

I'm not sure I understand this. Could you elaborate?

It just means pulling in another dep (REXML?) and then having two implementations of pypi_info i believe

I don't think we should expand the scope of update-python-resources. bump-formula-pr should handle this correctly instead.

[Bo98]

Given this, it seems more generally useful if:

• your non-PyPI resources declared livecheck blocks
• bump-formula-pr knew to update resources with livecheck blocks

This part is a dupe of #17401

[carlocab]

Closing as a duplicate of #17401.

[ctaintor]

@carlocab – let me explain. In this example, I have a CLI that is published privately as a Python package. The CLI depends on another private package and that private package depends on another. Let's call the CLI my-cli and it depends on direct-dep and that one depends on indirect-dep. my-cli, direct-dep and indirect-dep are all available from a private python repository hosted by Artifactory and all of them also rely on public packages. Someone would install my-cli by simply running:

 PIP_INDEX_URL=https://<host>/artifactory/api/pypi/v-pypi-production/simple pip install my-cli

the repository is what Artifactory calls a "virtual repository". It just means that it presents itself as a single repository and will resolve libraries first from the included local repositories (where my-cli, direct-dep and indirect-dep are published) before resolving from the remote pypi repository. So – when pip installs my-cli, it will resolve dependencies, finding direct-dep, resolving that and finding indirect-dep etc etc. It will of course also find and resolve public packages.

The problem with update-python-resources today is that it only relies on pypi to resolve what content should go into the resources blocks. If it instead used my PIP_INDEX_URL above, then everything would work. However, it's not that simple. The JSON API to pypi supports is relatively new and it is not required that repositories implement it. This means that Artifactory does not support it and it's also probably unlikely that it would ever support it unless it became mandatory. Artifactory (and I assume many other, similar tools) only supports the simple api, which is an HTML API.

If update-python-resources used the simple api then things would "just work" by me changing the URL to be my artifactory URL. Instead, supporting other repository hosts requires updating update-python-resources to either support only the simple api (so, remove the JSON api usage to make it universal) or modify it to support both json or simple.

[carlocab]

Thanks for the explanation!

In that case, this should be covered by the more general solution proposed at #17401 (but you may need to add livecheck blocks to your non-PyPI resources).

[ctaintor]

well, not really because a private libraries dependencies could change, such that resource blocks could be added or subtracted. Just to say that you'd want to update the entire dep tree and resolve all private package deps.

I'm not sure I see the value of livecheck on Python resources – mostly because it would be more correct to run update-python-resources in all cases? Or is the livecheck there just to suggest that update-python-resources should be run?

[carlocab]

Got it; thanks for explaining.

This seems like a relatively niche use-case, so I don't think any maintainers are likely to work on this. Still, we'd be willing to review PRs, but I suspect the barrier for merging them will be a bit high (given that it's a feature that isn't relevant to very many users).