Fix Leaderboard Sub Query Node

[marc-aurele-besner]

User description

Fix Leaderboard Sub Query Node

The new flow dump in leaderboard_entries a new entry for each related event, with the entry type, owner, value, block number and timestamp to be incremented.

Then every 50 blocks, we call the task manager to increment the actual leaderboard, and rank the entries

The task pickup all the leaderboard_entries for the block interval called for, sort them by type, aggregate the value per owner, then increment each of the leaderboard's entry (or create them) and then re-rank all leaderboard

---

PR Type

enhancement, configuration changes

---

Description

• Implemented a new block handler for sorting and ranking the leaderboard every 50 blocks.
• Refactored event handlers to utilize a unified function for creating and saving leaderboard entries.
• Added utility functions and constants to support leaderboard entry management.
• Enabled and configured the leaderboard sort and rank task in the taskboard.
• Added metadata for new leaderboard-related tables in the database.

---

Changes walkthrough 📝

	Relevant files
Enhancement	8 files project.ts Add block handler for leaderboard sorting and ranking        indexers/gemini-3h/leaderboard/project.ts Added a new block handler for sorting and ranking the leaderboard. Imported BLOCK_SORT_AND_RANK_INTERVAL constant. +8/-1      db.ts Implement function to create and save leaderboard entries indexers/gemini-3h/leaderboard/src/mappings/db.ts Implemented function to create and save leaderboard entries. Utilized randomUUID for entry ID generation. +26/-0    helper.ts Add function for sorting and ranking leaderboard                  indexers/gemini-3h/leaderboard/src/mappings/helper.ts Added function to sort and rank leaderboard. Configured HTTP request for taskboard interaction. +36/-0    mappingHandlers.ts Refactor event handlers and add leaderboard sorting            indexers/gemini-3h/leaderboard/src/mappings/mappingHandlers.ts Refactored event handlers to use createAndSaveLeaderboardEntry. Removed redundant logger statements. Added handler for sorting and ranking leaderboard. +216/-786 utils.ts Add utility function for JSON stringification                        indexers/gemini-3h/leaderboard/src/mappings/utils.ts Added utility function to stringify values, handling BigInt. +4/-0      index.js Add leaderboard sort and rank task                                              indexers/taskboard/tasks/index.js Added leaderboard sort and rank task to task list. Set task concurrency to 1. +6/-1      leaderboardSortAndRank.js Implement leaderboard sorting and ranking logic                    indexers/taskboard/tasks/leaderboardSortAndRank.js Implemented leaderboard sorting and ranking logic. Handled database interaction for leaderboard entries. +135/-0  db.js Add utility functions and queries for leaderboard management indexers/taskboard/utils/db.js Added functions for converting entry types to table names and strings to UUIDs. Defined queries for leaderboard entry management. +58/-1
Configuration changes	7 files contants.ts Define constants for leaderboard entry types and interval indexers/gemini-3h/leaderboard/src/mappings/contants.ts Defined constants for leaderboard entry types. Added BLOCK_SORT_AND_RANK_INTERVAL constant. +29/-0    index.js Enable leaderboard task and add entry type constants          indexers/taskboard/constants/index.js Enabled leaderboard sort and rank task. Added constants for leaderboard entry types. +30/-1    leaderboard_account_extrinsic_failed_total_counts.yaml Add metadata for account extrinsic failed total counts table indexers/db/metadata/databases/gemini_3h/tables/leaderboard_account_extrinsic_failed_total_counts.yaml Added table metadata for account extrinsic failed total counts. +19/-0    leaderboard_account_extrinsic_success_total_counts.yaml Add metadata for account extrinsic success total counts table indexers/db/metadata/databases/gemini_3h/tables/leaderboard_account_extrinsic_success_total_counts.yaml Added table metadata for account extrinsic success total counts. +19/-0    leaderboard_account_extrinsic_total_counts.yaml Add metadata for account extrinsic total counts table        indexers/db/metadata/databases/gemini_3h/tables/leaderboard_account_extrinsic_total_counts.yaml Added table metadata for account extrinsic total counts. +19/-0    leaderboard_account_remark_counts.yaml Add metadata for account remark counts table                          indexers/db/metadata/databases/gemini_3h/tables/leaderboard_account_remark_counts.yaml Added table metadata for account remark counts. +19/-0    leaderboard_account_transaction_fee_paid_total_values.yaml Add metadata for account transaction fee paid total values table indexers/db/metadata/databases/gemini_3h/tables/leaderboard_account_transaction_fee_paid_total_values.yaml Added table metadata for account transaction fee paid total values. +19/-0

---

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

[github-actions]

PR Reviewer Guide 🔍

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Key issues to review Error Handling The error handling in the sortAndRankLeaderboard function could be improved by providing more specific error messages related to the context of the failure, especially in the database operations. Hardcoded Values Consider replacing the hardcoded hostname and port in the HTTP request configuration with environment variables to enhance configurability and security. Type Conversion Direct type conversion using BigInt(value) could lead to runtime errors if value is not a valid number. It's safer to validate the input before conversion. Timestamp Handling The use of event.block.timestamp ?? new Date(0) as a fallback for timestamps might lead to misleading data. Consider a more appropriate default value or handling mechanism.

[github-actions]

PR Code Suggestions ✨

Category	Suggestion	Score
Reliability	Wrap multiple queries in a database transaction to ensure data integrity Consider using a transaction for multiple database queries to ensure data integrity and rollback capabilities in case of errors. indexers/taskboard/tasks/leaderboardSortAndRank.js [96-112] -await pool.query(createOrUpdateQuery, values); -await pool.query(rankingQuery); +await pool.query('BEGIN'); +try { + await pool.query(createOrUpdateQuery, values); + await pool.query(rankingQuery); + await pool.query('COMMIT'); +} catch (error) { + await pool.query('ROLLBACK'); + throw error; +} Suggestion importance[1-10]: 10 Why: Using transactions for multiple database operations is critical for ensuring data integrity and consistency, especially in the event of an error.	10
Possible bug	Add error handling for BigInt conversion to prevent runtime errors Consider using a more robust error handling mechanism when converting _amount to BigInt. If _amount is not a valid number string, this will throw an error which is not currently handled. indexers/gemini-3h/leaderboard/src/mappings/mappingHandlers.ts [16] -const amount = BigInt(_amount.toString()); +let amount; +try { + amount = BigInt(_amount.toString()); +} catch (error) { + console.error('Invalid amount format:', _amount); + return; +} Suggestion importance[1-10]: 9 Why: The suggestion correctly identifies a potential runtime error when converting _amount to BigInt and provides a robust error handling mechanism, which is crucial for preventing application crashes.	9
	Add a fallback or existence check for BLOCK_SORT_AND_RANK_INTERVAL to prevent potential runtime errors Consider checking the existence of BLOCK_SORT_AND_RANK_INTERVAL before using it in the filter. This will prevent runtime errors if the constant is not properly imported or defined. indexers/gemini-3h/leaderboard/project.ts [164-165] { kind: SubstrateHandlerKind.Block, handler: "handleSortAndRankLeaderboard", filter: { - modulo: BLOCK_SORT_AND_RANK_INTERVAL, + modulo: BLOCK_SORT_AND_RANK_INTERVAL ?? 100, // Default value if not defined }, } Suggestion importance[1-10]: 7 Why: Adding a fallback value for BLOCK_SORT_AND_RANK_INTERVAL can prevent runtime errors if the constant is not defined, improving the robustness of the code.	7
Enhancement	Implement try-catch for better error handling in HTTP requests Use async error handling with try-catch around the HTTP request to handle potential errors more effectively. indexers/gemini-3h/leaderboard/src/mappings/helper.ts [30-35] -const req = request(options, (res) => res.setEncoding("utf8")); -req.on("error", (e) => { - logger.error(`Problem with request: ${e.message}`); -}); -req.write(postData); -req.end(); +try { + const req = request(options, (res) => res.setEncoding("utf8")); + req.on("error", (e) => { + throw new Error(`Problem with request: ${e.message}`); + }); + req.write(postData); + req.end(); +} catch (error) { + logger.error(error.message); +} Suggestion importance[1-10]: 9 Why: Using try-catch for error handling in asynchronous operations like HTTP requests is crucial for capturing and managing errors effectively, thus improving the code's resilience.	9
Error handling	Add error handling for database connection failures Add error handling for the connectToDB function to manage potential connection failures gracefully. indexers/taskboard/tasks/leaderboardSortAndRank.js [14] -const pool = await connectToDB(); +let pool; +try { + pool = await connectToDB(); +} catch (error) { + console.error('Failed to connect to the database:', error); + throw new Error('Database connection failed'); +} Suggestion importance[1-10]: 9 Why: Adding error handling for database connections is crucial for robustness, preventing the application from crashing due to unhandled connection errors.	9
Best practice	Enforce value parameter as bigint to enhance type safety and reduce runtime type checks Instead of converting value to BigInt inside the function, consider enforcing the type at the function parameter level to ensure type safety throughout the function's usage. indexers/gemini-3h/leaderboard/src/mappings/db.ts [12-13] -if (typeof value === "number") { - value = BigInt(value); +export async function createAndSaveLeaderboardEntry( + leaderboard: string, + owner: string, + value: bigint, + timestamp: Date, + createdAt: number +): Promise<LeaderboardEntry> { + ... } Suggestion importance[1-10]: 8 Why: Enforcing the value parameter as bigint enhances type safety and eliminates the need for runtime type conversion, which is a best practice for maintaining code reliability.	8
	Replace the custom UUID generation with standard UUID v5 generation Consider using a more robust UUID generation method that adheres to the UUID standard instead of creating a custom format based on a hash. This ensures compatibility with systems and libraries expecting standard UUID formats. indexers/taskboard/utils/db.js [26-35] +const { v5: uuidv5 } = require('uuid'); +const NAMESPACE = '1b671a64-40d5-491e-99b0-da01ff1f3341'; // Example namespace UUID + const stringToUUID = (input) => { - const hash = createHash("sha256").update(input).digest("hex"); - return [ - hash.substring(0, 8), - hash.substring(8, 12), - hash.substring(12, 16), - hash.substring(16, 20), - hash.substring(20, 32), - ].join("-"); + return uuidv5(input, NAMESPACE); }; Suggestion importance[1-10]: 8 Why: Using a standard UUID generation method ensures compatibility with other systems and libraries, which is a significant improvement over a custom implementation.	8
	Use a consistent method for default timestamp handling To ensure consistency and avoid potential bugs, consider using a single method to handle the default timestamp (new Date(0)) across different event handlers. indexers/gemini-3h/leaderboard/src/mappings/mappingHandlers.ts [18] -const timestamp = event.block.timestamp ?? new Date(0); +const timestamp = getDefaultTimestamp(event.block.timestamp); +// Elsewhere in the codebase define the getDefaultTimestamp function: +function getDefaultTimestamp(timestamp: Date | undefined): Date { + return timestamp ?? new Date(0); +} Suggestion importance[1-10]: 6 Why: The suggestion promotes consistency and maintainability by centralizing the logic for handling default timestamps, which is a good practice but not critical.	6
Maintainability	Refactor repeated leaderboard entry creation to a single function Refactor the repeated logic for creating leaderboard entries into a single function to improve code maintainability and reduce redundancy. indexers/gemini-3h/leaderboard/src/mappings/mappingHandlers.ts [20-26] -await createAndSaveLeaderboardEntry( - LEADERBOARD_ENTRY_TYPE.ACCOUNT_TRANSFER_SENDER_TOTAL_COUNT, - from, - 1, - timestamp, - blockNumber -); +await saveLeaderboardEntry(LEADERBOARD_ENTRY_TYPE.ACCOUNT_TRANSFER_SENDER_TOTAL_COUNT, from, 1, timestamp, blockNumber); ... -await createAndSaveLeaderboardEntry( - LEADERBOARD_ENTRY_TYPE.ACCOUNT_TRANSFER_RECEIVER_TOTAL_COUNT, - to, - 1, - timestamp, - blockNumber -); +await saveLeaderboardEntry(LEADERBOARD_ENTRY_TYPE.ACCOUNT_TRANSFER_RECEIVER_TOTAL_COUNT, to, 1, timestamp, blockNumber); +// Define the saveLeaderboardEntry function: +async function saveLeaderboardEntry(type, participant, count, timestamp, blockNumber) { + await createAndSaveLeaderboardEntry(type, participant, count, timestamp, blockNumber); +} Suggestion importance[1-10]: 8 Why: Refactoring repeated logic into a single function improves code maintainability and reduces redundancy, making the codebase easier to manage and extend.	8
	Improve constant naming for clarity and consistency Consider using a more descriptive and consistent naming convention for constants to enhance code readability and maintainability. indexers/gemini-3h/leaderboard/src/mappings/contants.ts [29] -export const BLOCK_SORT_AND_RANK_INTERVAL = 50; +export const BLOCK_SORT_AND_RANK_INTERVAL_SECONDS = 50; // Assuming the interval is in seconds Suggestion importance[1-10]: 6 Why: While the suggestion improves readability by making the constant's purpose clearer, it is a minor enhancement and does not address a critical issue.	6
	Use template literals for constructing error messages Use template literals for better readability and maintainability when constructing error messages. indexers/taskboard/tasks/leaderboardSortAndRank.js [39-44] -throw new Error( - "No leaderboard entries found for interval: " + - lastBlockNumber + - " - " + - blockNumber -); +throw new Error(`No leaderboard entries found for interval: ${lastBlockNumber} - ${blockNumber}`); Suggestion importance[1-10]: 6 Why: Using template literals improves code readability and maintainability, although it is a minor improvement.	6
Performance	Cache repeated method calls to improve performance Consider caching the result of event.block.block.header.number.toNumber() since it is used multiple times, which can help in reducing redundant method calls and improve performance. indexers/gemini-3h/leaderboard/src/mappings/mappingHandlers.ts [17] const blockNumber = event.block.block.header.number.toNumber(); const timestamp = event.block.timestamp ?? new Date(0); -... -const blockNumber = event.block.block.header.number.toNumber(); +// Use blockNumber from the cached value Suggestion importance[1-10]: 7 Why: Caching the result of event.block.block.header.number.toNumber() is a valid optimization that can reduce redundant method calls, although the performance gain might be minor.	7