describe how component de-duplication works

[jkowalleck]

goal: discuss implementation details, processes and alternatives for #304

💡 please read the problem's description first: https://github.com/CycloneDX/cyclonedx-node-npm/milestone/2
🔍 please read the general thoughts and docs regarding NodeJS here: https://github.com/CycloneDX/cyclonedx-node-npm/blob/main/docs/result.md#de-duplication

[jkowalleck]

Proposal: de-duplicate on a minimal set of component properties

To be taken into account:

• Same name
• Same (nullable) group
• Same (nullable) version
• same non-empty download-source
• same non-empty set of hashes
• property cdx:npm:package:bundled must be unset or false

This way all non-bundled component "@acme/foo@1.33.7" that was downloaded from "https://my.org/acme/foo-1.33.7" with a hash "sha512-f7fbba6e0636f890e56fbbf3283e524c6fa3204ae298382d624741d0dc6638326e282c41be5e4254d8820772c5518a2c5a8c0c7f7eda19594a7eb539453e1ed7"
will be treated as the same - and should be de-duplicated.

[rkg-mm]

So, i read through your analysis and some further sources. Unfortunately I am not too experienced in how npm internals work. And I see your point. I experimented with one of our projects and figured some modules are deduped, some are not. I don't 100% understand why npm would not dedupe some packages here, while it does others.

Is it a bug in npm? Cause I tried to check some of these chains and compare them. I don't see the difference. Neither did I see bundled packages in those paths I checked, which would be the only reasonable explanation I could think of.

Nonetheless, whats the normal use case of this? I got a package.json, run npm install, build and then want a BOM file. Technically, what you do seems correct to me, representing what nodejs does internally. But logically, the main use cases of a BOM is to identify security or license issues, right? And for both use cases a logical representation makes more sense, than a technical one. Different licenses in same package&version on different installation paths makes no sense. Neither does it for checking vulnerability databases. And thats what most of us (all?) need this BOM file for.

in my opinion, the most reasonable and still simple solution would be:
add an option like "--deduplicateByPURL". Do a simple deduplication based on PURL with some warning in the docs, which explains the tradeoff. User can decide then, what he prefers and what he needs it for. I would totally go with this option as it serves my needs. not using the option would have 0 further benefit for me (and I would be interested if anybody sees an actual use case where the technical representation is beneficial)

edit: and thanks a lot for your work here!

[jkowalleck]

module-resolution-paths are a design-decision in node, that NPM and others try to flatten and predict.

Practical use-case: Remember when log4shell was around and people needed to judge the level of effect on their project? In the Node-world a case similar to that cases, the technical view comes in handy, because you not only know that you have the vulnerable component installed, but you could see which other component was actually using this vulnerable component -- since in node dependency-graphs and module-resolution-paths are not the same.
(PS: When the flatten-components=true option is used, then this critical information is lost anyways, so then it would be okay to de-duplicate components -- which led to the current implementations of #309)

if your use case of a SBOM is the license type and vuln-check, then the tools you use for this could narrow the focus/view of the original data and do the dedupe on their needs/criteria accordingly.
For example a license-agregation-tool could dedupe on (group, name, version, licenses),
and dependency-check could dedupe on (group, name, version, checksums, purl)

[rkg-mm]

For your mentioned log4shell use case there is the dependency-graph information in cyclonedx. Each deduplicated library surely must be listed in that graph then, all pointing to the same component entry. This should be sufficient to identify also the "which component was actually using it" case, unless I am missing something?

[rkg-mm]

@nscuro @stevespringett what do you think about this?

[jkowalleck]

re: #307 (reply in thread)

unless I am missing something?

yes you are.
in node the mdules are not flat. and they do not follow some arbirary dependency-graph when it comes to loading.
nodejs does have a module-resolution-tree, which is not the dependency graph. See an example here: https://github.com/CycloneDX/cyclonedx-node-npm/blob/main/docs/result.md#the-corresponding-module-resolution-graph or read the full description here https://github.com/CycloneDX/cyclonedx-node-npm/blob/main/docs/nodejs_internals.md

[jkowalleck]

Proposal: de-duplicate in a near-full set of component properties

To be taken into account: every property, but

• bom-ref value
• set of dependencies
• some properties (which ones?)
• set of sub-components

Having a wide set of markers makes it harder to de-duplicate by accident - less false-positives.