Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: "language" parser for rpm SPEC files #2916

Open
terriko opened this issue Apr 19, 2023 · 7 comments · May be fixed by #2964 or #3870
Open

feat: "language" parser for rpm SPEC files #2916

terriko opened this issue Apr 19, 2023 · 7 comments · May be fixed by #2964 or #3870
Assignees
@bcieszko
Labels
enhancement New feature or request hackathon Issues for folk participating in the Open Ecosystems hackathon

Comments

@terriko
Copy link
Contributor

terriko commented Apr 19, 2023 •
edited
Loading

RPMs have a SPEC file which contains a bunch of information that looks awfully like the {vendor, product, version} tuplet we use for NVD lookups:

https://rpm-packaging-guide.github.io/#what-is-a-spec-file

I think it should be possible to read these files similar to the way we read package lists.

For NVD lookup, we would want to intentionally build in some special cases for things like python files which are all packaged with the prefix python3- and so on rather than just assuming the name is correct, and maybe trying to do some nice parsing of the url included to see if that can be used to better guess the vendor.

The release-monitoring.org website may help you with names across distros. e.g. https://release-monitoring.org/project/3779/

We might also want to use some data from redhat to see if we can fine tune those mappings: https://access.redhat.com/security/data

Anyone working on this:

This issue is reserved for a participant in the Open Source Hackaton 2023. Please leave it for hackathon participants through the end of April. If it hasn't been claimed by May 5 it will be open to any contributor who wants to work on it.
@terriko terriko added enhancement New feature or request hackathon Issues for folk participating in the Open Ecosystems hackathon labels Apr 19, 2023
@bcieszko
Copy link
Contributor

CVEBnTool_Team3 working on this

@bcieszko
Copy link
Contributor

The rpm are built using SPEC file, but the final result is in the Lead Section and Header Section

more details here:
http://ftp.rpm.org/max-rpm/s1-rpm-file-format-rpm-file-format.html

here is sample dump from fedora emacs rpm:

rpm -qi emacs-28.2-3.fc36.x86_64.rpm
warning: emacs-28.2-3.fc36.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 38ab71f4: NOKEY
Name        : emacs
Epoch       : 1
Version     : 28.2
Release     : 3.fc36
Architecture: x86_64
Install Date: (not installed)
Group       : Unspecified
Size        : 158377917
License     : GPLv3+ and CC0
Signature   : RSA/SHA256, Sat Jan 28 16:31:02 2023, Key ID 999f7cbf38ab71f4
Source RPM  : emacs-28.2-3.fc36.src.rpm
Build Date  : Sat Jan 28 10:16:47 2023
Build Host  : buildvm-x86-25.iad2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.gnu.org/software/emacs/
Bug URL     : https://bugz.fedoraproject.org/emacs
Summary     : GNU Emacs text editor
Description :
Emacs is a powerful, customizable, self-documenting, modeless text
editor. Emacs contains special code editing features, a scripting
language (elisp), and the capability to read mail, news, and more
without leaving the editor.

This package provides an emacs binary with support for X windows.

Unfortunately as we can see the Vendor is set to Fedora Project

I've checked the OpenSuse rpm for comparision:

rpm -qi emacs-28.2-4.1.x86_64.rpm
warning: emacs-28.2-4.1.x86_64.rpm: Header V3 RSA/SHA512 Signature, key ID 29b700a4: NOKEY
Name        : emacs
Version     : 28.2
Release     : 4.1
Architecture: x86_64
Install Date: (not installed)
Group       : Productivity/Text/Editors
Size        : 77707579
License     : GPL-3.0-or-later
Signature   : RSA/SHA512, Mon Mar 20 10:37:28 2023, Key ID 35a2f86e29b700a4
Source RPM  : emacs-28.2-4.1.src.rpm
Build Date  : Mon Mar 20 10:24:46 2023
Build Host  : cloud118
Relocations : (not relocatable)
Packager    : https://bugs.opensuse.org
Vendor      : openSUSE
URL         : http://www.gnu.org/software/emacs/
Summary     : GNU Emacs Base Package
Description :
Basic package for the GNU Emacs editor.  For a documentation see https://www.emacsdocs.org/.
This package requires emacs-x11 and/or emacs-nox to have the GNU Emacs editor its self.

And the Vendor is set to the distro.

So we could get from the rpm file for the name and version, but not the vendor, I could try to extract it from the Summary, but i think it will be error-prone

@bcieszko bcieszko mentioned this issue Apr 28, 2023
Closed
@bcieszko
Copy link
Contributor

Got the nice output

cve-bin-tool --offline -r none emacs-28.2-3.fc36.x86_64.rpm
[14:43:38] INFO     cve_bin_tool - CVE Binary Tool v3.2.1dev0                                                                                                       cli.py:483
           INFO     cve_bin_tool - This product uses the NVD API but is not endorsed or certified by the NVD.                                                       cli.py:484
           WARNING  cve_bin_tool - Not verifying CVE DB cache                                                                                                       cli.py:661
           INFO     cve_bin_tool.CVEDB - There are 217388 CVE entries in the database                                                                             cvedb.py:216
           INFO     cve_bin_tool.CVEDB - There are 201036 CVE entries from NVD in the database                                                                    cvedb.py:218
           INFO     cve_bin_tool.CVEDB - There are 8868 CVE entries from OSV in the database                                                                      cvedb.py:218
           INFO     cve_bin_tool.CVEDB - There are 7275 CVE entries from GAD in the database                                                                      cvedb.py:218
           INFO     cve_bin_tool.CVEDB - There are 209 CVE entries from REDHAT in the database                                                                    cvedb.py:218
           INFO     cve_bin_tool.CVEDB - There are 217388 CVE entries in the database                                                                             cvedb.py:216
           INFO     cve_bin_tool.CVEDB - There are 201036 CVE entries from NVD in the database                                                                    cvedb.py:218
           INFO     cve_bin_tool.CVEDB - There are 8868 CVE entries from OSV in the database                                                                      cvedb.py:218
           INFO     cve_bin_tool.CVEDB - There are 7275 CVE entries from GAD in the database                                                                      cvedb.py:218
           INFO     cve_bin_tool.CVEDB - There are 209 CVE entries from REDHAT in the database                                                                    cvedb.py:218
           INFO     cve_bin_tool - CVE database contains CVEs from National Vulnerability Database (NVD), Open Source Vulnerability Database (OSV), Gitlab Advisory cli.py:683
                    Database (GAD) and RedHat                                                                                                                                 
           INFO     cve_bin_tool - CVE database last updated on 26 April 2023 at 10:58:52                                                                           cli.py:686
           INFO     cve_bin_tool - Number of checkers: 0                                                                                                            cli.py:860
           INFO     cve_bin_tool.VersionScanner - Checkers:                                                                                             version_scanner.py:108
           INFO     cve_bin_tool - Number of language checkers: 11                                                                                                  cli.py:865
           INFO     cve_bin_tool.VersionScanner - Language Checkers: Go, Java, Javascript, Perl, Php, Python, R, Rpm, Ruby, Rust, Swift                 version_scanner.py:131
[14:43:40] INFO     cve_bin_tool.CVEScanner - 5 CVE(s) in gnu.emacs v28.2                                                                                   cve_scanner.py:268
[14:47:34] INFO     cve_bin_tool - Overall CVE summary:                                                                                                             cli.py:900
           INFO     cve_bin_tool - There are 1 products with known CVEs detected                                                                                    cli.py:901
           INFO     cve_bin_tool - Known CVEs in ('gnu.emacs', '28.2'):                                                                                             cli.py:912
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                          CVE BINARY TOOL version: 3.2.1dev0                                          ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

 • Report Generated: 2023-04-28  14:47:34                                                                               
 • Time of last update of CVE Data: 2023-04-26  10:58:52                                                                
╭─────────────╮
│ CVE SUMMARY │
╰─────────────╯
┏━━━━━━━━━━┳━━━━━━━┓
┃ Severity ┃ Count ┃
┡━━━━━━━━━━╇━━━━━━━┩
│ CRITICAL │ 0     │
│ HIGH     │ 5     │
│ MEDIUM   │ 0     │
│ LOW      │ 0     │
│ UNKNOWN  │ 0     │
└──────────┴───────┘
╭─────────────────╮
│  NewFound CVEs  │
╰─────────────────╯
┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Vendor ┃ Product ┃ Version ┃ CVE Number     ┃ Source ┃ Severity ┃ Score (CVSS Version) ┃
┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ gnu    │ emacs   │ 28.2    │ CVE-2007-6109  │ NVD    │ HIGH     │ 10 (v2)              │
│ gnu    │ emacs   │ 28.2    │ CVE-2022-48338 │ NVD    │ HIGH     │ 7.3 (v3)             │
│ gnu    │ emacs   │ 28.2    │ CVE-2022-48339 │ NVD    │ HIGH     │ 7.8 (v3)             │
│ gnu    │ emacs   │ 28.2    │ CVE-2023-27985 │ NVD    │ HIGH     │ 7.8 (v3)             │
│ gnu    │ emacs   │ 28.2    │ CVE-2023-27986 │ NVD    │ HIGH     │ 7.8 (v3)             │
└────────┴─────────┴─────────┴────────────────┴────────┴──────────┴──────────────────────┘
┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Vendor ┃ Product ┃ Version ┃ Root ┃ Filename                     ┃
┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ gnu    │ emacs   │ 28.2    │      │ emacs-28.2-3.fc36.x86_64.rpm │
└────────┴─────────┴─────────┴──────┴──────────────────────────────┘
╭───────────────────────────────────────────────╮
│  Products with No Identified Vulnerabilities  │
╰───────────────────────────────────────────────╯
┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ Vendor ┃ Product ┃ Version ┃
┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
└────────┴─────────┴─────────┘

all checkers disabled, only languages enabled, and emacs not even yet in checkers :)

need to cleanup code and peer review, and will post pr later

bcieszko added a commit to bcieszko/cve-bin-tool that referenced this issue Apr 28, 2023
@bcieszko
feat: "language" parser for rpm files
b37d64c 
* fixes intel#2916

Signed-off-by: Bartlomiej Cieszkowski <bartlomiej.cieszkowski@intel.com>
Signed-off-by: Przemyslaw Romaniak <przemyslaw.romaniak@intel.com>
@bcieszko bcieszko linked a pull request Apr 28, 2023 that will close this issue
Open
bcieszko added a commit to bcieszko/cve-bin-tool that referenced this issue May 1, 2023
@bcieszko
feat: "language" parser for rpm files
061a8bb 
* fixes intel#2916

Signed-off-by: Bartlomiej Cieszkowski <bartlomiej.cieszkowski@intel.com>
Signed-off-by: Przemyslaw Romaniak <przemyslaw.romaniak@intel.com>
@joydeep049 joydeep049 mentioned this issue Feb 9, 2024
Closed
@tahifahimi
Copy link
Contributor

tahifahimi commented Feb 14, 2024 •
edited
Loading

Since this issue is still open, I checked the PR related to this issue. This pull request added a parser for RPM files based on the RPM file format. The test for file "test/condensed-downloads/dovecot-2.3.14-1.fc34.i686.rpm" in line 137 of "test/test_helper_script.py" file fails.
The output of this file is as follows:
Screenshot from 2024-02-14 16-36-58

And this test fails because out.split("VERSION_PATTERNS")[1] contains "\n". This test is added in pull request #1690 which add multiline string finder.
I think the implementation for RPM parser is correct and the test needs minor enhancement.
@terriko what do you think?

@terriko
Copy link
Contributor Author

terriko commented Feb 15, 2024

@tahifahimi That sounds reasonable. This one just didn't get finished during the hackathon and everyone moved on to other projects. If you want to propose a fix for the test, we can integrate that and get this merged.

@tahifahimi
Copy link
Contributor

tahifahimi commented Feb 16, 2024 •
edited
Loading

The purpose of this test was to have a better (more beautiful) output, but the function that is supposed to ensure the VERSION_PATTERN does not contain any "\n" or "\r" is not implemented. So as you can check the output_single function in the "cve_bin_tool/helper_script.py" file, at line 385, where it prints a "\n" at the end of "VERSION_PATTERN".
I guess that we do not need this test right now (test from line 114, in file "/test/test_helper_script.py"), since the output is good enough right now.
@terriko, I will apply the changes and send a PR request.

This was referenced Feb 20, 2024
Closed
Draft
@tahifahimi
Copy link
Contributor

As discussed before in #3845, merging this PR may fail due to two tests related to multiline capacities. “test_scan_files_multipline” in file "/test/test_helper_script.py" tests the system to check the multiline capability for two files “dovecot-2.3.14-1.fc34.i686.rpm” and “gnome-shell-41.2-1.fc35.x86_64.rpm”

In “extract_and_parse_file”, dovecot file is detected as an executable file, so the test can be fixed through minor changes in #3870. What I realized is that This file should not be detected as an executable file. I think it is related to a bug in the implementation of "Versionscanner.is_executable()".

For the second file, the output version pattern is incorrect and contains “\n.” It shows that the multiline-pattern is not supported for non executable files. (I am not pretty sure how we can remedy this)
I would appreciate any feedback @terriko

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bcieszko bcieszko

Labels
enhancement New feature or request hackathon Issues for folk participating in the Open Ecosystems hackathon
Projects
None yet
Milestone
No milestone
3 participants
@terriko @tahifahimi @bcieszko