feat: add hyperscan support

[ffontaine]

hyperscan will run simultaneously all version checkers on a file which reduce processing time.

pyperscan package is used instead of the most well-known hyperscan package as pyperscan allows to add a tag for each pattern. This feature will allow to retrieve easily the checker associated to the matched pattern.

Fix #2485

Signed-off-by: Fabrice Fontaine fabrice.fontaine@orange.com

[terriko]

This is so cool, but it looks like it's not getting the same scan results in 28 tests. Here's a snippet:

FAILED test/test_scanner.py::TestScanner::test_version_in_package[http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/r/-rh-nodejs14-nodejs-14.16.0-1.el7.x86_64.rpm-node.js-14.16.0-other_products585] - AssertionError: assert '14.16.0' in {'1.0.2k', '1.2.11', '10.12.0'}
FAILED test/test_scanner.py::TestScanner::test_version_in_package[https://kojipkgs.fedoraproject.org/packages/openssh/6.8p1/1.1.fc23/x86_64/-openssh-clients-6.8p1-1.1.fc23.x86_64.rpm-openssh-6.8p1-other_products608] - AssertionError: assert '6.8p1' in {'3.1'}
FAILED test/test_scanner.py::TestScanner::test_version_in_package[https://www.rpmfind.net/linux/fedora/linux/development/rawhide/Everything/aarch64/os/Packages/t/-thunderbird-102.5.1-1.fc38.aarch64.rpm-thunderbird-102.5.1-other_products787] - AssertionError: sqlite not found in thunderbird-102.5.1-1.fc38.aarch64.rpm. Remove sqlite from other_products.
assert 'sqlite' in {'libjpeg', 'libjpeg-turbo', 'lz4', 'rust', 'thunderbird'}
FAILED test/test_scanner.py::TestScanner::test_version_in_package[https://downloads.openwrt.org/releases/packages-19.07/x86_64/packages/-tor_0.4.5.10-1_x86_64.ipk-tor-0.4.5.10-other_products799] - AssertionError: assert '0.4.5.10' in {'0.1.2.17', '1.1.1n'}
FAILED test/test_scanner.py::TestScanner::test_version_in_package[http://rpmfind.net/linux/epel/9/Everything/aarch64/Packages/z/-zeek-core-4.2.0-1.el9.aarch64.rpm-zeek-4.2.0-other_products883] - AssertionError: sqlite not found in zeek-core-4.2.0-1.el9.aarch64.rpm. Remove sqlite from other_products.
assert 'sqlite' in {'zeek'}
==== 28 failed, 1513 passed, 43 skipped, 65 warnings in 1656.27s (0:27:36) =====

It also looks like it's not installing on windows because the wheel won't build:

  ERROR: Failed building wheel for pyperscan
ERROR: Could not build wheels for pyperscan, which is required to install pyproject.toml-based projects
Successfully built cve-bin-tool gsutil
Failed to build pyperscan

I didn't dig through the actual error messages deeply but we could be missing a component for the build or something.

[ffontaine]

Indeed, I'll fix those build failures in separate PR (for example, Tor signature can raise false positives)

[ffontaine]

#2681, #2682, #2684, #2688, #2689, #2690 and #2691 should be merged then I'll rebase this PR to have a clean hyperscan addition

[terriko]

The relevant PRs should be merged now, whenever you've got time to get back to this one.

[ffontaine]

I updated the PR but the tests don't start

[ffontaine]

Still working on this one, I assume that we should wait EOL of python 3.7 before merging

[terriko]

We could also make it an optional addition, similar to what we do with PDF export. I don't know off the top of my head if an option can require a different version of python the same way it requires extra packages, but we could just document the heck out of it until 3.7 support gets dropped officially.

[terriko]

@ffontaine I'm closing this as part of my "clean up stale pull requests before the new year" chores this week, but do feel free to re-open if you want to keep working on it. We've gotten rid of python 3.7 which was the original reason this went stale, but it's probably got other issues in the conflicts by now.