Fix Accounts Sub Query Node #854

marc-aurele-besner · 2024-09-23T02:04:39Z

User description

Fix Accounts Sub Query Node

This adds/fix the logic around indexing accounts, transfer and account balance changes

I also added Redis, Express and BullMQ to have a tasks board, allowing to create a task that will then be executed outside of the indexer worker

(especially useful for updating existing db entry, or to run long process

PR Type

enhancement, bug_fix

Description

Refactored event handlers in project.ts to streamline logic and improve efficiency.
Added new helper functions in db.ts for creating and saving account, balance history, and transfer records.
Implemented utility functions in utils.ts for JSON handling and date management.
Updated GraphQL schema to include new types and indices for better data management.
Defined new table schemas and permissions in YAML configuration files for accounts, balance histories, and transfers.
Added new dependencies to support account management and balance updates.

Changes walkthrough 📝

Relevant files

Enhancement

6 files

project.ts `Refactor event handlers and add call handler` indexers/gemini-3h/accounts/project.ts Removed multiple event handlers for various modules and methods. Added a new handler for `SubstrateHandlerKind.Call`. Updated handlers for `VoteReward` and `BlockReward` events.	+8/-52
db.ts `Add database helper functions for accounts and transfers` indexers/gemini-3h/accounts/src/mappings/db.ts Added functions to create and save account, balance history, and transfer records. Implemented logic to check existence before saving new records.	+82/-0
helper.ts `Implement account balance update helper function` indexers/gemini-3h/accounts/src/mappings/helper.ts Added a function to update account balance. Utilized external balance API for fetching account balance.	+19/-0
mappingHandlers.ts `Refactor mapping handlers and add new event handlers` indexers/gemini-3h/accounts/src/mappings/mappingHandlers.ts Refactored `handleTransferEvent` to use new helper functions. Added new handlers for extrinsic and reward events. Removed redundant functions for account and transfer checks.	+108/-80
utils.ts `Add utility functions for JSON and date handling` indexers/gemini-3h/accounts/src/mappings/utils.ts Added utility functions for JSON stringification and date entry creation.	+9/-0
schema.graphql `Update GraphQL schema with new types and indices` indexers/gemini-3h/accounts/schema.graphql Updated `Account` type with new fields and indices. Added `BalanceHistory` type with fields and indices. Updated `Transfer` type with new indices.	+13/-3

Configuration changes

4 files

accounts_accounts.yaml `Define accounts table schema and permissions` indexers/db/metadata/databases/gemini_3h/tables/accounts_accounts.yaml Defined table schema and select permissions for accounts.	+20/-0
accounts_balance_histories.yaml `Define balance histories table schema and permissions` indexers/db/metadata/databases/gemini_3h/tables/accounts_balance_histories.yaml Defined table schema and select permissions for balance histories.	+18/-0
accounts_transfers.yaml `Define transfers table schema and permissions` indexers/db/metadata/databases/gemini_3h/tables/accounts_transfers.yaml Defined table schema and select permissions for transfers.	+23/-0
tables.yaml `Update tables configuration to include new schemas` indexers/db/metadata/databases/gemini_3h/tables/tables.yaml Included new tables for accounts, balance histories, and transfers.	+3/-0

Dependencies

1 files

package.json `Add new dependencies for account management` indexers/gemini-3h/accounts/package.json Added new dependencies `@autonomys/auto-consensus` and `@autonomys/auto-utils`.	+2/-0

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

github-actions · 2024-09-23T02:05:48Z

PR Reviewer Guide 🔍

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Key issues to review Handler Redundancy The new handlers `handleFarmerVoteRewardEvent` and `handleFarmerBlockRewardEvent` seem to perform similar operations as `handleTransferEvent`. Consider refactoring to a common function to avoid code duplication. Error Handling The functions `createAndSaveAccountIfNotExists`, `createAndSaveBalanceHistory`, and `createAndSaveTransfer` do not handle errors that might occur during database operations. Consider adding try-catch blocks to handle potential exceptions. Timestamp Handling The calculation of `timestamp` in `handleTransferEvent` might lead to incorrect values due to the multiplication by 1000. Verify if the original timestamp is in seconds and needs conversion to milliseconds.

github-actions · 2024-09-23T02:05:51Z

PR Code Suggestions ✨

Category Suggestion Score

Error handling

Add error handling for asynchronous operations in the handleTransferEvent function

Consider handling potential exceptions for the updateAccountBalance and
createAndSaveAccountIfNotExists calls within the handleTransferEvent function. This
can be done using try-catch blocks to ensure that errors are caught and handled
appropriately, preventing the function from failing silently.

indexers/gemini-3h/accounts/src/mappings/mappingHandlers.ts [19-38]

-const fromBalance = await updateAccountBalance(from, blockNumber);
-const toBalance = await updateAccountBalance(to, blockNumber);
-...
-await createAndSaveAccountIfNotExists(
-  from,
-  blockNumber,
-  BigInt(0),
-  fromBalance.free,
-  fromBalance.reserved,
-  fromBalance.free + fromBalance.reserved
-);
-await createAndSaveAccountIfNotExists(
-  to,
-  blockNumber,
-  BigInt(0),
-  toBalance.free,
-  toBalance.reserved,
-  toBalance.free + toBalance.reserved
-);
+try {
+  const fromBalance = await updateAccountBalance(from, blockNumber);
+  const toBalance = await updateAccountBalance(to, blockNumber);
+  ...
+  await createAndSaveAccountIfNotExists(
+    from,
+    blockNumber,
+    BigInt(0),
+    fromBalance.free,
+    fromBalance.reserved,
+    fromBalance.free + fromBalance.reserved
+  );
+  await createAndSaveAccountIfNotExists(
+    to,
+    blockNumber,
+    BigInt(0),
+    toBalance.free,
+    toBalance.reserved,
+    toBalance.free + toBalance.reserved
+  );
+} catch (error) {
+  console.error('Failed to handle transfer event:', error);
+}

Suggestion importance[1-10]: 8

Why: The suggestion to add error handling using try-catch blocks for asynchronous operations in the handleTransferEvent function is valid and important. It improves the robustness of the code by ensuring that any errors during these operations are caught and logged, preventing silent failures.

8

socket-security · 2024-09-23T02:05:59Z

👍 Dependency issues cleared. Learn more about Socket for GitHub ↗︎

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report↗︎

…b-query

indexers/gemini-3h/accounts/src/mappings/helper.ts

netlify · 2024-09-24T20:17:57Z

✅ Deploy Preview for dev-astral canceled.

Name	Link
🔨 Latest commit	`ea8e09a`
🔍 Latest deploy log	https://app.netlify.com/sites/dev-astral/deploys/66f31e68cf5da300088c54af

socket-security · 2024-09-24T20:18:51Z

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@bull-board/express@5.23.0	None	`0`	11.7 kB	felixmosh
npm/@bull-board/ui@5.23.0	None	`0`	3.36 MB	felixmosh
npm/@msgpackr-extract/msgpackr-extract-darwin-arm64@3.0.3	None	`0`	54.2 kB	kriszyp
npm/@msgpackr-extract/msgpackr-extract-darwin-x64@3.0.3	None	`0`	53.7 kB	kriszyp
npm/@msgpackr-extract/msgpackr-extract-linux-arm@3.0.3	None	`0`	26 kB	kriszyp
npm/@msgpackr-extract/msgpackr-extract-linux-arm64@3.0.3	None	`0`	28.6 kB	kriszyp
npm/@msgpackr-extract/msgpackr-extract-linux-x64@3.0.3	None	`0`	61.3 kB	kriszyp
npm/@msgpackr-extract/msgpackr-extract-win32-x64@3.0.3	None	`0`	228 kB	kriszyp
npm/async@3.2.6	None	`0`	808 kB	aearly
npm/body-parser@1.20.3	network	`0`	62.6 kB	ulisesgascon
npm/bullmq@4.17.0	environment, filesystem, network, shell	`0`	1.47 MB	manast
npm/connect-ensure-login@0.1.1	None	`0`	7.5 kB	jaredhanson
npm/cron-parser@4.9.0	filesystem	`0`	54.2 kB	harrisiirak
npm/detect-libc@2.0.3	filesystem, shell	`0`	23.6 kB	lovell
npm/ejs@3.1.10	eval, filesystem	`0`	143 kB	mde
npm/encodeurl@2.0.0	None	`0`	6.98 kB	blakeembrey
npm/express-session@1.18.0	environment	`0`	86.3 kB	dougwilson
npm/express@4.21.0	environment, filesystem, network	`0`	221 kB	wesleytodd
npm/filelist@1.0.4	filesystem	`0`	18.6 kB	mde
npm/finalhandler@1.3.1	environment	`0`	19 kB	dougwilson, wesleytodd
npm/glob@8.1.0	filesystem	`0`	56.2 kB	isaacs
npm/ioredis@5.4.1	network	`0`	698 kB	ioredis-robot
npm/jake@10.9.2	environment, filesystem, shell	`0`	175 kB	mde
npm/lodash@4.17.21	None	`0`	1.41 MB	bnjmnt4n
npm/luxon@3.5.0	None	`0`	4.48 MB	icambron
npm/merge-descriptors@1.0.3	None	`0`	5.08 kB	sindresorhus
npm/minimatch@5.1.6	None	`0`	38.9 kB	isaacs
npm/msgpackr-extract@3.0.3	None	`0`	15.5 kB	kriszyp
npm/msgpackr@1.11.0	environment, eval, unsafe	`0`	1.92 MB	kriszyp
npm/node-gyp-build-optional-packages@5.2.2	environment, filesystem, unsafe	`0`	14.8 kB	kriszyp
npm/object-inspect@1.13.2	None	`0`	99.1 kB	ljharb
npm/on-headers@1.0.2	None	`0`	7.54 kB	dougwilson
npm/passport-local@1.0.0	None	`0`	8.07 kB	jaredhanson
npm/passport-strategy@1.0.0	None	`0`	5.1 kB	jaredhanson
npm/passport@0.6.0	network	`0`	81.6 kB	jaredhanson
npm/path-to-regexp@0.1.10	None	`0`	6.38 kB	blakeembrey
npm/pause@0.0.1	None	`0`	2.17 kB
npm/pg@8.13.0	environment, network	`+3`	344 kB	brianc
npm/qs@6.13.0	None	`0`	254 kB	ljharb
npm/random-bytes@1.0.0	None	`0`	6.24 kB	dougwilson
npm/redis-info@3.1.0	None	`0`	21.2 kB	fgribreau
npm/semver@7.6.3	None	`0`	95.8 kB	npm-cli-ops
npm/send@0.19.0	filesystem, network	`0`	50.2 kB	ulisesgascon
npm/serve-static@1.16.2	None	`0`	25.4 kB	wesleytodd
npm/tslib@2.7.0	None	`0`	86.2 kB	typescript-bot
npm/uid-safe@2.1.5	None	`0`	7.47 kB	dougwilson

View full report↗︎

marc-aurele-besner added 6 commits September 22, 2024 21:11

improve account sub query schema and add db helper

2831031

add and improve utils, helpers and db files

cfba227

improve mapping logic

f5f604a

fix project

279fee0

add accounts metadata

7136f29

add permissions

3ab2e29

github-actions bot added enhancement New feature or request bug_fix labels Sep 23, 2024

github-actions bot added the Review effort [1-5]: 4 label Sep 23, 2024

marc-aurele-besner changed the title ~~Fix Consensus Sub Query Node~~ Fix Accounts Sub Query Node Sep 23, 2024

marc-aurele-besner added 3 commits September 23, 2024 10:47

add redis, express and bullMQ

025371f

add bullMQ and update balance task

c11f2b9

improve balance history id

a7d22d0

marc-aurele-besner marked this pull request as ready for review September 23, 2024 19:55

marc-aurele-besner requested review from DaMandal0rian, jfrank-summit and clostao September 23, 2024 19:55

marc-aurele-besner added 2 commits September 23, 2024 17:31

Merge branch 'feat/fix-consensus-sub-query' into feat/fix-accounts-su…

0a8465c

…b-query

fix scripts

9929d9f

Base automatically changed from feat/fix-consensus-sub-query to main September 23, 2024 23:32

clostao approved these changes Sep 24, 2024

View reviewed changes

indexers/gemini-3h/accounts/src/mappings/helper.ts Show resolved Hide resolved

Merge branch 'main' into feat/fix-accounts-sub-query

ea8e09a

marc-aurele-besner merged commit de09818 into main Sep 24, 2024
13 checks passed

marc-aurele-besner deleted the feat/fix-accounts-sub-query branch September 24, 2024 20:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Accounts Sub Query Node #854

Fix Accounts Sub Query Node #854

marc-aurele-besner commented Sep 23, 2024 •

edited

Loading

github-actions bot commented Sep 23, 2024

github-actions bot commented Sep 23, 2024 •

edited

Loading

socket-security bot commented Sep 23, 2024 •

edited

Loading

netlify bot commented Sep 24, 2024 •

edited

Loading

socket-security bot commented Sep 24, 2024

Fix Accounts Sub Query Node #854

Fix Accounts Sub Query Node #854

Conversation

marc-aurele-besner commented Sep 23, 2024 • edited Loading

User description