Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add tests for tool-object script #3265

Open
wants to merge 49 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
d1fc6c4
tests added
vishvamsinh28 Sep 28, 2024
9b22bc1
fwqfwq
vishvamsinh28 Sep 28, 2024
3629568
fefeq
vishvamsinh28 Sep 28, 2024
cabfbae
errors added
vishvamsinh28 Sep 28, 2024
340c1f7
fixture update
vishvamsinh28 Sep 28, 2024
5d4771b
fwqf
vishvamsinh28 Sep 30, 2024
7673203
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Sep 30, 2024
e6a3505
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Oct 4, 2024
403bb4e
wfwqf
vishvamsinh28 Oct 4, 2024
9d74152
fwgff
vishvamsinh28 Oct 4, 2024
3e20593
updated test for errors
vishvamsinh28 Oct 4, 2024
5000a47
updated test for errors
vishvamsinh28 Oct 4, 2024
8efbc84
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Oct 7, 2024
56b0fdd
renamed a variabel
vishvamsinh28 Oct 7, 2024
859676e
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Oct 13, 2024
f51feb3
test updated
vishvamsinh28 Oct 13, 2024
f540e5b
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Oct 14, 2024
ae3e19b
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Oct 15, 2024
3e73220
path update
vishvamsinh28 Oct 16, 2024
f9906ae
Merge branch 'toolsObjectTest' of https://github.com/vishvamsinh28/we…
vishvamsinh28 Oct 16, 2024
d31a06f
commit to rerun workflow
vishvamsinh28 Oct 16, 2024
03f9dc1
commit to rerun workflow
vishvamsinh28 Oct 16, 2024
c866aa0
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Oct 18, 2024
bfc9e6e
fefewg
vishvamsinh28 Oct 18, 2024
9c1ef9e
supress logs
vishvamsinh28 Oct 18, 2024
a1af4f7
newsroom test update
vishvamsinh28 Oct 18, 2024
9d38d41
build-tools test update
vishvamsinh28 Oct 18, 2024
08a8970
uupdate newsroom function
vishvamsinh28 Oct 18, 2024
cb90254
update build-tools function
vishvamsinh28 Oct 18, 2024
d16f9a1
commit to re run the workflow
vishvamsinh28 Oct 18, 2024
1164703
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Oct 19, 2024
4540530
added retry mechanicasm for enoent errors
vishvamsinh28 Oct 19, 2024
cf6a37d
Merge branch 'toolsObjectTest' of https://github.com/vishvamsinh28/we…
vishvamsinh28 Oct 19, 2024
0a83f54
fegwg
vishvamsinh28 Oct 19, 2024
740d4d6
fwqfqf
vishvamsinh28 Oct 19, 2024
8f2162d
feeqg
vishvamsinh28 Oct 19, 2024
c2ee924
use fs-extra
vishvamsinh28 Oct 19, 2024
267023e
commit to re run the workflow
vishvamsinh28 Oct 19, 2024
e4be92f
fqefeqf
vishvamsinh28 Oct 19, 2024
f39e57c
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Oct 20, 2024
dcb955b
test updated
vishvamsinh28 Oct 21, 2024
b8c0558
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Oct 27, 2024
72f1e63
fef
vishvamsinh28 Oct 27, 2024
fed2691
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Oct 27, 2024
7554976
add empty line at end
vishvamsinh28 Oct 27, 2024
044a049
Merge branch 'toolsObjectTest' of https://github.com/vishvamsinh28/we…
vishvamsinh28 Oct 27, 2024
0270bf1
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Nov 1, 2024
771c5db
Merge branch 'master' into toolsObjectTest
vishvamsinh28 Nov 4, 2024
2be6d42
fef
vishvamsinh28 Nov 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 51 additions & 53 deletions scripts/tools/tools-object.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ addFormats(ajv, ["uri"])
const validate = ajv.compile(schema)
const { convertToJson } = require('../utils');


// Config options set for the Fuse object
const options = {
includeScore: true,
Expand All @@ -25,7 +24,7 @@ const fuse = new Fuse(categoryList, options)
// isAsyncAPIrepo boolean variable to define whether the tool repository is under
// AsyncAPI organization or not, to create a JSON tool object as required in the frontend
// side to show ToolCard.
const createToolObject = async (toolFile, repositoryUrl='', repoDescription='', isAsyncAPIrepo='') => {
const createToolObject = async (toolFile, repositoryUrl = '', repoDescription = '', isAsyncAPIrepo = '') => {
let resultantObject = {
title: toolFile.title,
description: toolFile?.description ? toolFile.description : repoDescription,
Expand All @@ -47,67 +46,66 @@ const createToolObject = async (toolFile, repositoryUrl='', repoDescription='',
// and creating a JSON tool object in which all the tools are listed in defined
// categories order, which is then updated in `automated-tools.json` file.
async function convertTools(data) {
let finalToolsObject = {};
const dataArray = data.items;

// initialising finalToolsObject with all categories inside it with proper elements in each category
for (var index in categoryList) {
finalToolsObject[categoryList[index].name] = {
description: categoryList[index].description,
toolsList: []
};
}
try {
let finalToolsObject = {};
const dataArray = data.items;
Comment on lines +49 to +51
Copy link

@coderabbitai coderabbitai bot Oct 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Enhance error handling with structured logging.

The outer try-catch block could benefit from more structured error logging to aid in debugging and monitoring.

Consider enhancing the error handling:

 try {
   let finalToolsObject = {};
   const dataArray = data.items;
   // ... rest of the code
   return finalToolsObject;
 } catch (err) {
-  throw new Error(`Error processing tool: ${err.message}`)
+  const errorContext = {
+    error: err.message,
+    stack: err.stack,
+    phase: 'tool_processing',
+    timestamp: new Date().toISOString()
+  };
+  console.error('Tool processing failed:', JSON.stringify(errorContext, null, 2));
+  throw new Error(`Error processing tool: ${err.message}`);
 }

Also applies to: 110-112

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akshatnema, understood. Using a custom LogService with Winston for logging makes sense. I'll note this for future reviews.


✏️ Learnings added
Learnt from: akshatnema
PR: asyncapi/website#3265
File: scripts/tools/tools-object.js:49-51
Timestamp: 2024-11-01T11:40:51.360Z
Learning: In JavaScript code, prefer using a custom LogService with Winston for logging errors instead of `console.error` or ad-hoc logging.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai Instead of doing this, we can have a custom LogService that uses winston to provide these logs.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akshatnema, understood. Implementing a custom LogService with Winston is a great approach. Here's an example of how you could set it up:

Create a logger.js module:

// logger.js
const { createLogger, format, transports } = require('winston');

const logger = createLogger({
  level: 'error', // Set the minimum level of messages that this logger will handle
  format: format.combine(
    format.timestamp(),
    format.json()
  ),
  transports: [
    new transports.Console(), // Output to the console
    // Add other transports like File or HTTP as needed
  ],
});

module.exports = logger;

Update your tools-object.js to use the custom logger:

const logger = require('./logger');

// ...

async function convertTools(data) {
  try {
    let finalToolsObject = {};
    const dataArray = data.items;
    // ... rest of the code
    return finalToolsObject;
  } catch (err) {
    logger.error('Error processing tool', {
      message: err.message,
      stack: err.stack,
      phase: 'tool_processing',
    });
    throw new Error(`Error processing tool: ${err.message}`);
  }
}

With this setup, all your error logs will be handled by Winston, allowing for consistent and configurable logging across the application.

Copy link
Member

@akshatnema akshatnema Nov 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai Can you please make a issue for this to create a custom LogService?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akshatnema, I've created an issue to implement a custom LogService using Winston as requested: #3358

🔗 Analysis chain
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Create a new GitHub issue for creating a custom LogService using Winston
gh issue create \
  --repo asyncapi/website \
  --title "Create Custom LogService Using Winston for Error Logging" \
  --body "As discussed in [PR #3265](https://github.com/asyncapi/website/pull/3265) in [this comment](https://github.com/asyncapi/website/pull/3265#discussion_r1817967437), we need to implement a custom LogService using Winston to improve error logging throughout the application.

Requested by @akshatnema." \
  --assignee akshatnema

Length of output: 485


for (let tool of dataArray) {
try {
if (tool.name.startsWith('.asyncapi-tool')) {
// extracting the reference id of the repository which will be used to extract the path of the .asyncapi-tool file in the Tools repository
// ex: for a url = "https://api.github.com/repositories/351453552/contents/.asyncapi-tool?ref=61855e7365a881e98c2fe667a658a0005753d873"
// the text (id) present after '=' gives us a reference id for the repo
let reference_id = tool.url.split("=")[1];
let download_url = `https://raw.githubusercontent.com/${tool.repository.full_name}/${reference_id}/${tool.path}`;
// initialising finalToolsObject with all categories inside it with proper elements in each category
for (var index in categoryList) {
finalToolsObject[categoryList[index].name] = {
description: categoryList[index].description,
toolsList: []
};
}
Comment on lines +54 to +59
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Replace for..in with Object.entries.

The for..in loop can iterate over inherited properties and is not the recommended approach for array iteration.

-    for (var index in categoryList) {
-      finalToolsObject[categoryList[index].name] = {
-        description: categoryList[index].description,
-        toolsList: []
-      };
-    }
+    for (const category of categoryList) {
+      finalToolsObject[category.name] = {
+        description: category.description,
+        toolsList: []
+      };
+    }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for (var index in categoryList) {
finalToolsObject[categoryList[index].name] = {
description: categoryList[index].description,
toolsList: []
};
}
for (const category of categoryList) {
finalToolsObject[category.name] = {
description: category.description,
toolsList: []
};
}
🧰 Tools
🪛 eslint

[error] 54-59: for..in loops iterate over the entire prototype chain, which is virtually never what you want. Use Object.{keys,values,entries}, and iterate over the resulting array.

(no-restricted-syntax)


[error] 54-59: The body of a for-in should be wrapped in an if statement to filter unwanted properties from the prototype.

(guard-for-in)


[error] 54-54: All 'var' declarations must be at the top of the function scope.

(vars-on-top)


[error] 54-54: Unexpected var, use let or const instead.

(no-var)


const { data: toolFileContent } = await axios.get(download_url);
for (let tool of dataArray) {
try {
if (tool.name.startsWith('.asyncapi-tool')) {
// extracting the reference id of the repository which will be used to extract the path of the .asyncapi-tool file in the Tools repository
// ex: for a url = "https://api.github.com/repositories/351453552/contents/.asyncapi-tool?ref=61855e7365a881e98c2fe667a658a0005753d873"
// the text (id) present after '=' gives us a reference id for the repo
let reference_id = tool.url.split("=")[1];
let download_url = `https://raw.githubusercontent.com/${tool.repository.full_name}/${reference_id}/${tool.path}`;

//some stuff can be YAML
const jsonToolFileContent = await convertToJson(toolFileContent)
const { data: toolFileContent } = await axios.get(download_url);

//validating against JSON Schema for tools file
const isValid = await validate(jsonToolFileContent)
//some stuff can be YAML
const jsonToolFileContent = await convertToJson(toolFileContent)

if (isValid) {
let repositoryUrl = tool.repository.html_url;
let repoDescription = tool.repository.description;
let isAsyncAPIrepo = tool.repository.owner.login === "asyncapi";
let toolObject = await createToolObject(jsonToolFileContent, repositoryUrl, repoDescription, isAsyncAPIrepo);
//validating against JSON Schema for tools file
const isValid = await validate(jsonToolFileContent)

// Tool Object is appended to each category array according to Fuse search for categories inside Tool Object
jsonToolFileContent.filters.categories.forEach(async (category) => {
const categorySearch = await fuse.search(category);
if (isValid) {
let repositoryUrl = tool.repository.html_url;
let repoDescription = tool.repository.description;
let isAsyncAPIrepo = tool.repository.owner.login === "asyncapi";
let toolObject = await createToolObject(jsonToolFileContent, repositoryUrl, repoDescription, isAsyncAPIrepo);

if (categorySearch.length) {
let searchedCategoryName = categorySearch[0].item.name
if (!finalToolsObject[searchedCategoryName].toolsList.find((element => element === toolObject)))
finalToolsObject[searchedCategoryName].toolsList.push(toolObject);
} else {
// if Tool object has a category, not defined in our categorylist, then this provides a `other` category to the tool.
if (!finalToolsObject['Others'].toolsList.find((element => element === toolObject)))
finalToolsObject['Others'].toolsList.push(toolObject);
}
});
} else {
console.error('Script is not failing, it is just dropping errors for further investigation');
console.error('Invalid .asyncapi-tool file.');
console.error(`Located in: ${tool.html_url}`);
console.error('Validation errors:', JSON.stringify(validate.errors, null, 2));
// Tool Object is appended to each category array according to Fuse search for categories inside Tool Object
await Promise.all(jsonToolFileContent.filters.categories.map(async (category) => {
const categorySearch = await fuse.search(category);
const targetCategory = categorySearch.length ? categorySearch[0].item.name : 'Others';
const toolsList = finalToolsObject[targetCategory].toolsList;
if (!toolsList.includes(toolObject)) {
toolsList.push(toolObject);
}
}));
} else {
console.error('Script is not failing, it is just dropping errors for further investigation');
console.error('Invalid .asyncapi-tool file.');
console.error(`Located in: ${tool.html_url}`);
console.error('Validation errors:', JSON.stringify(validate.errors, null, 2));
}
Comment on lines +93 to +98
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Improve validation error handling.

The current validation error logging could be more actionable and structured.

Consider enhancing the validation error reporting:

 } else {
-  console.error('Script is not failing, it is just dropping errors for further investigation');
-  console.error('Invalid .asyncapi-tool file.');
-  console.error(`Located in: ${tool.html_url}`);
-  console.error('Validation errors:', JSON.stringify(validate.errors, null, 2));
+  const validationError = {
+    type: 'validation_failure',
+    file: tool.html_url,
+    errors: validate.errors,
+    timestamp: new Date().toISOString()
+  };
+  console.error('AsyncAPI tool validation failed:', JSON.stringify(validationError, null, 2));
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
} else {
console.error('Script is not failing, it is just dropping errors for further investigation');
console.error('Invalid .asyncapi-tool file.');
console.error(`Located in: ${tool.html_url}`);
console.error('Validation errors:', JSON.stringify(validate.errors, null, 2));
}
} else {
const validationError = {
type: 'validation_failure',
file: tool.html_url,
errors: validate.errors,
timestamp: new Date().toISOString()
};
console.error('AsyncAPI tool validation failed:', JSON.stringify(validationError, null, 2));
}

}
} catch (err) {
console.error(err)
throw err;
}
} catch (err) {
console.error(err)
throw err;
}
return finalToolsObject;
} catch (err) {
throw new Error(`Error processing tool: ${err.message}`)
}
return finalToolsObject;
}

module.exports = {convertTools, createToolObject}
module.exports = { convertTools, createToolObject }
Loading
Loading