-
Notifications
You must be signed in to change notification settings - Fork 180
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parse and import both rpm and deb packages metadata #9101
base: master
Are you sure you want to change the base?
Parse and import both rpm and deb packages metadata #9101
Commits on Aug 2, 2024
-
Create lzreposync subdirectory
lzreposync will be a spacewalk-repo-sync replacement written in Python. It uses a src layout and a pyproject.toml. The target Python version is 3.11, compatibility with older Python versions is explicitly not a goal.
Configuration menu - View commit details
-
Copy full SHA for 860a02f - Browse repository at this point
Copy the full SHA 860a02fView commit details
Commits on Aug 4, 2024
-
Configuration menu - View commit details
-
Copy full SHA for d9b3efe - Browse repository at this point
Copy the full SHA d9b3efeView commit details -
Added the remote_path column that will hold the remote path/ url of a given package. This information will help locate the package later-on on the remote repository and download it.
Configuration menu - View commit details
-
Copy full SHA for e594cfa - Browse repository at this point
Copy the full SHA e594cfaView commit details -
Add expand_full_filelist parameter
A boolean argument that checks whether we should call the header.hdr.fullFilelist() We added this argument to disable the header.hdr.fullFilelist() function only for the lzreposync service.
Configuration menu - View commit details
-
Copy full SHA for 199381e - Browse repository at this point
Copy the full SHA 199381eView commit details -
The inspect.getargspec() method is deprecated in Python 3 It can be replaced by inspect.getfullargspec()
Configuration menu - View commit details
-
Copy full SHA for 5acf7ab - Browse repository at this point
Copy the full SHA 5acf7abView commit details -
Add import_signatures parameter
The import_signatures is a boolean argument that specifies whether we should execute the _import_signatures() method. We added this parameter to disable the _import_signatures() method for the lzreposync service.
Configuration menu - View commit details
-
Copy full SHA for 86df68e - Browse repository at this point
Copy the full SHA 86df68eView commit details -
Implement Primary.xml file parser
Parsing the rpm's Primary.xml packages metadata file using pulldom xml parser as a memory efficient parsing library. Note that some attributes in the returned parsed object are faked, and maybe filled in elsewhere. The faking of some of the data is done because some attributes are required by the importer service.
Configuration menu - View commit details
-
Copy full SHA for f772e64 - Browse repository at this point
Copy the full SHA f772e64View commit details -
Implement filelists.xml file parser
Parsing the rpm's filelists.xml metadata file using pulldom xml parser as a memory efficient parsing library. The parser parses the given filelists.xml file (normally in gz format), and cache the filelist information of each package in a separate file in the cache directory, using the package's hash as the filename, with no file extension.
Configuration menu - View commit details
-
Copy full SHA for 3b2d236 - Browse repository at this point
Copy the full SHA 3b2d236View commit details -
Implement full rpm metadata parsing
Using both primary_parser and filelists_parser, return the full packages' metadata, pacakge by package, using lazing parsing. Note that there some attributes that are faked, because we can't fetch them now, and they're required by the package importer later-on. However, we can fake them more efficiently, using less memory.
Configuration menu - View commit details
-
Copy full SHA for 1649f34 - Browse repository at this point
Copy the full SHA 1649f34View commit details -
Parse and import rpm patches/updates
Parsed the update-info.xml file and imported the parsed patches/updates to the database. We used pretty much the same code from the old Reposync class.
Configuration menu - View commit details
-
Copy full SHA for a6d462c - Browse repository at this point
Copy the full SHA a6d462cView commit details -
Import parsed rpm & deb packages to db
Import the parsed rpm and debian packages to the database in batche, and associate each pacakge with the corresponding channel
Configuration menu - View commit details
-
Copy full SHA for 0df0030 - Browse repository at this point
Copy the full SHA 0df0030View commit details -
Implement the deb Packages md file
Parsed the debian Packages metadata file in a lazy way and yield the metadata of each package separately.
Configuration menu - View commit details
-
Copy full SHA for fda9e51 - Browse repository at this point
Copy the full SHA fda9e51View commit details -
Implement the Translation file parser
Parsed the debian's Translation file that contains the full description of packages, grouped by description-md5, and cache the parsed descriptions in a cache directory.
Configuration menu - View commit details
-
Copy full SHA for eee052b - Browse repository at this point
Copy the full SHA eee052bView commit details -
Implement full deb metadata parsing
Using both packages_parser and translation_parser, return the full packages' metadata, pacakge by package, using lazing parsing Also set the debian repository's information in a DebRepo class
Configuration menu - View commit details
-
Copy full SHA for 5a51d0d - Browse repository at this point
Copy the full SHA 5a51d0dView commit details -
Fetch repository information from the db
Given the channel label, fetch important repository's information form the database, and store it in a temporary object RepoDTO
Configuration menu - View commit details
-
Copy full SHA for 5e26b7a - Browse repository at this point
Copy the full SHA 5e26b7aView commit details -
Complete lzreposync service entry point
Added the necessary command line arguments. Identify the target repositories, prepare the datastructures, and execute the lazy synchronization of repositories/packages.
Configuration menu - View commit details
-
Copy full SHA for 8ef299a - Browse repository at this point
Copy the full SHA 8ef299aView commit details -
Added a new dependency python-gnupg used to verify repo signature.
Configuration menu - View commit details
-
Copy full SHA for 5435907 - Browse repository at this point
Copy the full SHA 5435907View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3567817 - Browse repository at this point
Copy the full SHA 3567817View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8bad179 - Browse repository at this point
Copy the full SHA 8bad179View commit details -
Configuration menu - View commit details
-
Copy full SHA for 61b0a74 - Browse repository at this point
Copy the full SHA 61b0a74View commit details -
Ignored two linting complains about rasing exceptions floowing the approach in the old reposync. We can enhance the code instead of doing this though.
Configuration menu - View commit details
-
Copy full SHA for 898d571 - Browse repository at this point
Copy the full SHA 898d571View commit details
Commits on Aug 15, 2024
-
Complete code for lzreposync version 0.1
This commit completes almost all the logic and use cases of the new lazy reposync. **Note** that this commit will be restructured and possibly divided into smaller and more convenient commits. This commit is for review purposes.
Configuration menu - View commit details
-
Copy full SHA for 4c7db58 - Browse repository at this point
Copy the full SHA 4c7db58View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f8a070 - Browse repository at this point
Copy the full SHA 4f8a070View commit details -
Fix error: too many clients already
Seemingly this error happened because we reached the maximum number of unclosed db connections. And thought that this might be due to the fact that the close() method in the Database class was not implemented, and the rhnSQL.closeDB() was not closing any connection. However, we're still hesitating about whether this is the root cause of the problem, because the old(current) reposync is was using it without any error.
Configuration menu - View commit details
-
Copy full SHA for f329e51 - Browse repository at this point
Copy the full SHA f329e51View commit details
Commits on Aug 19, 2024
-
This is the latest and almost the final version of the lzreposync service. (gpg sig check not complete) It contains pretty much all the necessary tests, including the ones for updates/patches import. Some of the remaining 'todos' are either for code enhancements or some unclear concepts that will be discussed with the team. Of course, this commit will be split into smaller ones later after rebase.
Configuration menu - View commit details
-
Copy full SHA for 913f21c - Browse repository at this point
Copy the full SHA 913f21cView commit details
Commits on Aug 26, 2024
-
Optimize code and do some cleanup
- Removed some todos. - Changed some sql queries with equivalent ones using JOIN...ON. - Some other minor cleanup
Configuration menu - View commit details
-
Copy full SHA for 8a49313 - Browse repository at this point
Copy the full SHA 8a49313View commit details
Commits on Aug 29, 2024
-
Optimized some code by changing classes and methods in some logics with free functions. Consolidated the debian repo parsing.
Configuration menu - View commit details
-
Copy full SHA for 6ccb3bf - Browse repository at this point
Copy the full SHA 6ccb3bfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2157d56 - Browse repository at this point
Copy the full SHA 2157d56View commit details
Commits on Sep 2, 2024
-
Configuration menu - View commit details
-
Copy full SHA for dceccec - Browse repository at this point
Copy the full SHA dceccecView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2a79e72 - Browse repository at this point
Copy the full SHA 2a79e72View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2f4c998 - Browse repository at this point
Copy the full SHA 2f4c998View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9a95e5a - Browse repository at this point
Copy the full SHA 9a95e5aView commit details -
Configuration menu - View commit details
-
Copy full SHA for fed31fe - Browse repository at this point
Copy the full SHA fed31feView commit details
Commits on Sep 9, 2024
-
Complete gpg signature check for rpm
Completed the gpg signature check for rpm repositories, mainly for the repomd.xml file. This is done by downloading the signature file from the remote rpm repo, and executing 'gpg verify' to verify the repomd.xml file against its signature using the already added gpg keys on the filesystem. So, if you haven't already added the required gpg keyring on your system, you'll not be able to verify the repo. You should ideally run this version directly on the uyuni- server, because the gpg keyring will probably be present there.
Configuration menu - View commit details
-
Copy full SHA for a033e4d - Browse repository at this point
Copy the full SHA a033e4dView commit details -
Configuration menu - View commit details
-
Copy full SHA for b890b1c - Browse repository at this point
Copy the full SHA b890b1cView commit details -
Refactor: Allow more input variants in makedirs()
makedirs() in uyuni.common.fileutils now accepts relative paths that consist of only a directory name or paths with trailing slashes.
Configuration menu - View commit details
-
Copy full SHA for 7d0c57c - Browse repository at this point
Copy the full SHA 7d0c57cView commit details -
Merge pull request #1 from agraul/refactor-makedirs
Refactor: Allow more input variants in makedirs()
Configuration menu - View commit details
-
Copy full SHA for 7aa602c - Browse repository at this point
Copy the full SHA 7aa602cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4042fc4 - Browse repository at this point
Copy the full SHA 4042fc4View commit details
Commits on Sep 11, 2024
-
Complete gpg signature check for debian
Completed the gpg signature check for debian repositories. If you haven't already added the required gpg keyring on your system, you'll not be able to verify the repo, and you'll normally get a GeneralRepoException. You should ideally run this version directly on the uyuni- server, because the gpg keyring will probably be present there.
Configuration menu - View commit details
-
Copy full SHA for a7270cd - Browse repository at this point
Copy the full SHA a7270cdView commit details