Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(server): support passing fetchLicense #1286

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
+40 −26
Draft

feat(server): support passing fetchLicense #1286

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f6a83ab
feat(server): add fetchLicense arg
setchy Jul 31, 2024
5d6b390
feat(server): add fetchLicense arg
setchy Aug 1, 2024
074b4d3
feat(server): add fetchLicense arg
setchy Aug 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/SERVER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ Arguments can be passed either via the query string or as a JSON body. The follo
| only | Include components only containing this word in purl. Useful to generate BOM with first party components alone. Multiple values allowed. [array] |
| autoCompositions | Automatically set compositions when the BOM was filtered. [boolean] [default: true] |
| gitBranch | Git branch used when cloning the repository. If not specified will use the default branch assigned to the repository. |
| fetchLicense | Automatically query public registries such as maven, npm, or nuget to resolve the package licenses. This is a time-consuming operation and is disabled by default. |

## Ways to use server mode

Expand Down
8 changes: 4 additions & 4 deletions index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ import {
CARGO_CMD,
CLJ_CMD,
DEBUG_MODE,
FETCH_LICENSE,
LEIN_CMD,
MAX_BUFFER,
PREFER_MAVEN_DEPS_TREE,
Expand Down Expand Up @@ -135,6 +134,7 @@ import {
parseSwiftResolved,
parseYarnLock,
readZipEntry,
shouldFetchLicense,
splitOutputByGradleProjects,
} from "./utils.js";
let url = import.meta.url;
Expand Down Expand Up @@ -3100,7 +3100,7 @@ export async function createPythonBom(path, options) {
if (tempDir?.startsWith(tmpdir()) && rmSync) {
rmSync(tempDir, { recursive: true, force: true });
}
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
pkgList = await getPyMetadata(pkgList, false);
}
return buildBomNSData(options, pkgList, "pypi", {
Expand Down Expand Up @@ -4280,7 +4280,7 @@ export async function createSwiftBom(path, options) {
}
}
}
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
pkgList = await getSwiftPackageMetadata(pkgList);
}
return buildBomNSData(options, pkgList, "swift", {
Expand Down Expand Up @@ -5209,7 +5209,7 @@ export async function createCsharpBom(path, options) {
dependsOn: Array.from(parentDependsOn),
});
}
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
const retMap = await getNugetMetadata(pkgList, dependencies);
if (retMap.dependencies?.length) {
dependencies = mergeDependencies(
Expand Down
4 changes: 4 additions & 0 deletions server.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ const parseQueryString = (q, body, options = {}) => {
"includeFormulation",
"includeCrypto",
"standard",
"fetchLicense",
];

for (const param of queryParams) {
Expand All @@ -115,6 +116,9 @@ const parseQueryString = (q, body, options = {}) => {
if (options.profile) {
applyProfileOptions(options);
}
if (options.fetchLicense) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since node is a single process event driven model, one request will affect multiple responses. We need to refactor the codebase and make the fetch license operate with an options argument instead of only relying on env variables

process.env.FETCH_LICENSE = options.fetchLicense;
}
return options;
};

Expand Down
53 changes: 31 additions & 22 deletions utils.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,9 +128,12 @@ export const PREFER_MAVEN_DEPS_TREE =
["true", "1"].includes(process.env.PREFER_MAVEN_DEPS_TREE);

// Whether license information should be fetched
export const FETCH_LICENSE =
process.env.FETCH_LICENSE &&
["true", "1"].includes(process.env.FETCH_LICENSE);
export function shouldFetchLicense() {
return (
process.env.FETCH_LICENSE &&
["true", "1"].includes(process.env.FETCH_LICENSE)
);
}

// Whether search.maven.org will be used to identify jars without maven metadata; default, if unset shall be 'true'
export const SEARCH_MAVEN_ORG =
Expand Down Expand Up @@ -765,12 +768,18 @@ export async function getNpmMetadata(pkgList) {
let body = {};
if (metadata_cache[key]) {
body = metadata_cache[key];
if (DEBUG_MODE) {
console.log(`npm metadata: using ${key} metadata from local cache`);
}
} else {
const res = await cdxgenAgent.get(NPM_URL + key, {
responseType: "json",
});
body = res.body;
metadata_cache[key] = body;
if (DEBUG_MODE) {
console.log(`npm metadata: looking up ${key} metadata on npm`);
}
}
p.description =
body.versions?.[p.version]?.description || body.description;
Expand Down Expand Up @@ -872,7 +881,7 @@ export async function parsePkgJson(pkgJsonFile, simple = false) {
// continue regardless of error
}
}
if (!simple && FETCH_LICENSE && pkgList && pkgList.length) {
if (!simple && shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parsePkgJson`,
Expand Down Expand Up @@ -1215,7 +1224,7 @@ export async function parsePkgLock(pkgLockFile, options = {}) {
options,
));

if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parsePkgLock`,
Expand Down Expand Up @@ -1532,7 +1541,7 @@ export async function parseYarnLock(yarnLockFile) {
}
});
}
if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parseYarnLock`,
Expand Down Expand Up @@ -1609,7 +1618,7 @@ export async function parseNodeShrinkwrap(swFile) {
}
}
}
if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parseNodeShrinkwrap`,
Expand Down Expand Up @@ -2043,7 +2052,7 @@ export async function parsePnpmLock(pnpmLock, parentComponent = null) {
});
}
}
if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parsePnpmLock`,
Expand Down Expand Up @@ -2102,7 +2111,7 @@ export async function parseBowerJson(bowerJsonFile) {
// continue regardless of error
}
}
if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parseBowerJson`,
Expand Down Expand Up @@ -2187,7 +2196,7 @@ export async function parseMinJs(minJsFile) {
// continue regardless of error
}
}
if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parseMinJs`,
Expand Down Expand Up @@ -3176,7 +3185,7 @@ export async function getMvnMetadata(pkgList, jarNSMapping = {}) {
if (!pkgList || !pkgList.length) {
return pkgList;
}
if (DEBUG_MODE && FETCH_LICENSE) {
if (DEBUG_MODE && shouldFetchLicense()) {
console.log(`About to query maven for ${pkgList.length} packages`);
}
for (const p of pkgList) {
Expand Down Expand Up @@ -3212,7 +3221,7 @@ export async function getMvnMetadata(pkgList, jarNSMapping = {}) {
}
const group = p.group || "";
// If the package already has key metadata skip querying maven
if (group && p.name && p.version && !FETCH_LICENSE) {
if (group && p.name && p.version && !shouldFetchLicense()) {
cdepList.push(p);
continue;
}
Expand Down Expand Up @@ -3454,7 +3463,7 @@ export function guessPypiMatchingVersion(versionsList, versionSpecifiers) {
* @param {Boolean} fetchDepsInfo Fetch dependencies info from pypi
*/
export async function getPyMetadata(pkgList, fetchDepsInfo) {
if (!FETCH_LICENSE && !fetchDepsInfo) {
if (!shouldFetchLicense() && !fetchDepsInfo) {
return pkgList;
}
const PYPI_URL = process.env.PYPI_URL || "https://pypi.org/pypi/";
Expand Down Expand Up @@ -4368,7 +4377,7 @@ export async function getGoPkgLicense(repoMetadata) {
export async function getGoPkgComponent(group, name, version, hash) {
let pkg = {};
let license = undefined;
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
if (DEBUG_MODE) {
console.log(
`About to fetch go package license information for ${group}:${name}`,
Expand Down Expand Up @@ -4697,7 +4706,7 @@ export async function parseGosumData(gosumData) {
const version = tmpA[1].replace("/go.mod", "");
const hash = tmpA[tmpA.length - 1].replace("h1:", "sha256-");
let license = undefined;
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
if (DEBUG_MODE) {
console.log(
`About to fetch go package license information for ${name}`,
Expand Down Expand Up @@ -4749,7 +4758,7 @@ export async function parseGopkgData(gopkgData) {
case "name":
pkg.group = "";
pkg.name = value;
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
pkg.license = await getGoPkgLicense({
group: pkg.group,
name: pkg.name,
Expand Down Expand Up @@ -4959,7 +4968,7 @@ export async function parseGemspecData(gemspecData) {
}
});
pkgList = [pkg];
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
return await getRubyGemsMetadata(pkgList);
}
return pkgList;
Expand Down Expand Up @@ -5211,7 +5220,7 @@ export async function parseGemfileLockData(gemLockData, lockFile) {
dependsOn: Array.from(dependenciesMap[k]),
});
}
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
pkgList = await getRubyGemsMetadata(pkgList);
return { pkgList, dependenciesList };
}
Expand Down Expand Up @@ -5569,7 +5578,7 @@ export async function parseCargoTomlData(
if (pkg) {
addPackageToList(pkgList, pkg, { packageMode, simple });
}
if (!simple && FETCH_LICENSE) {
if (!simple && shouldFetchLicense()) {
return await getCratesMetadata(pkgList);
}
return pkgList;
Expand Down Expand Up @@ -5709,7 +5718,7 @@ export async function parseCargoData(
if (pkg) {
addPackageToList(pkgList, pkg, { simple });
}
if (FETCH_LICENSE && !simple) {
if (!simple && shouldFetchLicense()) {
return await getCratesMetadata(pkgList);
}
return pkgList;
Expand Down Expand Up @@ -5875,7 +5884,7 @@ export async function parseCargoAuditableData(cargoData) {
});
}
});
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
return await getCratesMetadata(pkgList);
}
return pkgList;
Expand Down Expand Up @@ -5914,7 +5923,7 @@ export async function parsePubLockData(pubLockData) {
}
}
});
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
return await getDartMetadata(pkgList);
}
return pkgList;
Expand Down
Loading