convert/python: parse requirements.txt, skip if package exist on APKINDEX, rework error printing #704
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As we discussed at #621, this PR is initial attempt to support parsing
requirements.txt
that given by argument as a either a FILE or URL.Motivation
One of the limitations of the
convert python
subcommand is that we can not provide arbitraryrequirements.txt
. So this makes melange tightly coupled topypi
to search and get dependencies of given package.By this limitation, it makes impossilbe to generate melange manifests if package is not hosted on
pypi
.This commands also does not respect already-created packages that hosted on APKINDEX:
https://packages.wolfi.dev
. Since it do recursive calls to find all of the dependencies, it makes the process even longer and eventually we have duplicated (already-crafted) packages.Implementation
To tackle with this problems to enrich UX even further;
--use-existing
! It basically gathers all the Python packages from Wolfi APK to not re-generate again to speed up the processconvert python
now supports FILE and URL:go run . convert python requirements.txt --use-existing
go run . convert python https://raw.githubusercontent.com/kubeflow/pipelines/master/backend/src/apiserver/visualization/requirements.txt --use-existing
- took25m
on my machine - follow up proposal:convert {apkbuild, gem, python}
: make concurrent generation pipeline #7052
(from4
) to match with the current manifests