Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Consensus Sub Query Node #853

Merged
merged 15 commits into from
Sep 23, 2024
Merged

Conversation

marc-aurele-besner
Copy link
Collaborator

@marc-aurele-besner marc-aurele-besner commented Sep 21, 2024

User description

Fix Consensus Sub Query Node

This add/fix the logic for basic indexing of all block, events, extrinsics and log with very similar schema as what we had before.


PR Type

enhancement, tests


Description

  • Refactored and enhanced the logic for handling blocks, extrinsics, and events in the consensus sub-query node.
  • Implemented new database operations for saving blockchain data, including blocks, logs, extrinsics, and events.
  • Updated project configuration with new repository link, node options, and custom types.
  • Modified GraphQL schema to reflect the new data model, removing unused entities and updating existing ones.
  • Updated package dependencies to align with the new implementation and removed unused dependencies.
  • Added new utility and helper functions to support data processing and block author retrieval.
  • Introduced new scripts for console and metadata operations in the package configuration.

Changes walkthrough 📝

Relevant files
Enhancement
mappingHandlers.ts
Refactor and enhance block, extrinsic, and event handling

indexers/gemini-3h/consensus/src/mappings/mappingHandlers.ts

  • Refactored block handling logic with new helper functions.
  • Added functions to handle extrinsics and events.
  • Removed unused imports and code.
  • Introduced logging for block processing.
  • +161/-220
    db.ts
    Implement database operations for blockchain data               

    indexers/gemini-3h/consensus/src/mappings/db.ts

  • Added functions to create and save blocks, logs, extrinsics, and
    events.
  • Implemented helper functions for database operations.
  • Introduced module and section management.
  • +214/-0 
    utils.ts
    Add utility functions for data processing                               

    indexers/gemini-3h/consensus/src/mappings/utils.ts

  • Added utility functions for data processing and logging.
  • Implemented functions to calculate space pledged and history size.
  • +68/-0   
    helper.ts
    Add helper function for block author retrieval                     

    indexers/gemini-3h/consensus/src/mappings/helper.ts

  • Introduced helper function to get block author.
  • Utilized utility functions for log decoding.
  • +29/-0   
    schema.graphql
    Update GraphQL schema for new data model                                 

    indexers/gemini-3h/consensus/schema.graphql

  • Modified schema to include new entities and fields.
  • Removed unused Account entity.
  • Updated fields for Block, Extrinsic, and Event entities.
  • +44/-62 
    package.json
    Add new scripts for development operations                             

    indexers/package.json

    • Added new scripts for console and metadata operations.
    +2/-1     
    Configuration changes
    project.ts
    Update project configuration and data sources                       

    indexers/gemini-3h/consensus/project.ts

  • Updated project configuration with new repository and node options.
  • Added custom types for blockchain data.
  • Configured data sources for block, call, and event handlers.
  • +37/-3   
    Dependencies
    package.json
    Update package dependencies                                                           

    indexers/gemini-3h/consensus/package.json

  • Updated dependencies and devDependencies.
  • Removed unused dependencies.
  • +2/-9     

    💡 PR-Agent usage:
    Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

    Copy link

    PR Reviewer Guide 🔍

    ⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
    🧪 No relevant tests
    🔒 No security concerns identified
    ⚡ Key issues to review

    Possible Bug
    The method getBlockAuthor uses a hardcoded DEFAULT_ACCOUNT_ID as a fallback, which could lead to incorrect author data being recorded. Consider handling this case more dynamically or ensuring that the correct data is always available.

    Performance Issue
    The functions createAndSaveBlock, saveLog, createAndSaveSectionIfNotExists, createAndSaveExtrinsicModuleNameIfNotExists, createAndSaveEventModuleNameIfNotExists, createAndSaveExtrinsic, and createAndSaveEvent all use await inside loops or map iterations. This could lead to performance bottlenecks due to sequential execution. Consider refactoring to allow parallel execution where possible.

    Code Smell
    The file contains large functions with multiple responsibilities (e.g., handleBlock, handleCall, handleEvent). Consider breaking these functions into smaller, more focused functions to improve maintainability and readability.

    Copy link

    github-actions bot commented Sep 21, 2024

    PR Code Suggestions ✨

    CategorySuggestion                                                                                                                                    Score
    Possible bug
    Handle potential null values from environment variables safely

    Avoid using the TypeScript non-null assertion operator ('!') for environment
    variables as it may lead to runtime errors if the variables are not set. Instead,
    handle the potential null or undefined values gracefully.

    indexers/gemini-3h/consensus/project.ts [50-51]

    -endpoint: process.env.GEMINI_3H_RPC!?.split(",") as string[] | string,
    -dictionary: process.env.DICTIONARY!,
    +endpoint: process.env.GEMINI_3H_RPC ? process.env.GEMINI_3H_RPC.split(",") as string[] | string : "default_endpoint",
    +dictionary: process.env.DICTIONARY || "default_dictionary",
     
    Suggestion importance[1-10]: 9

    Why: This suggestion addresses a possible runtime error by providing default values if environment variables are not set, which is crucial for application stability.

    9
    Add a check for 'process.env.GEMINI_3H_RPC' before using it to avoid runtime errors

    Consider checking the return value of 'process.env.GEMINI_3H_RPC' before calling
    'split' to prevent runtime errors in case the environment variable is not set.

    indexers/gemini-3h/consensus/project.ts [50]

    -endpoint: process.env.GEMINI_3H_RPC!?.split(",") as string[] | string,
    +endpoint: process.env.GEMINI_3H_RPC ? process.env.GEMINI_3H_RPC.split(",") as string[] | string : ["default_endpoint"],
     
    Suggestion importance[1-10]: 9

    Why: This suggestion prevents potential runtime errors by ensuring the environment variable is checked before use, which is important for robust error handling.

    9
    Best practice
    Specify a fixed version for '@subql/node' to ensure consistent behavior

    Replace the wildcard version '*' for '@subql/node' with a specific version to ensure
    consistent behavior across different environments and avoid potential breaking
    changes from unexpected updates.

    indexers/gemini-3h/consensus/project.ts [23]

    -version: "*",
    +version: "specific_version_number",  # Replace 'specific_version_number' with the desired version
     
    Suggestion importance[1-10]: 8

    Why: Using a specific version instead of a wildcard ensures consistent behavior across environments and prevents potential issues from unexpected updates.

    8
    Maintainability
    Ensure type safety by replacing '@ts-ignore' with specific type assertions

    Replace the use of '@ts-ignore' which suppresses TypeScript errors, with a more
    specific type assertion or handling to maintain type safety.

    indexers/gemini-3h/consensus/project.ts [53-61]

    -// @ts-ignore
     types: {
       Solution: {
    -    public_key: "AccountId32",
    -    reward_address: "AccountId32",
    +    public_key: "AccountId32" as any,  # Replace 'any' with the actual expected type if known
    +    reward_address: "AccountId32" as any,
       },
       SubPreDigest: {
    -    slot: "u64",
    -    solution: "Solution",
    +    slot: "u64" as any,
    +    solution: "Solution" as any,
       },
     },
     
    Suggestion importance[1-10]: 7

    Why: Replacing '@ts-ignore' with specific type assertions improves code maintainability and type safety, although it may require additional type information.

    7

    Copy link

    socket-security bot commented Sep 21, 2024

    👍 Dependency issues cleared. Learn more about Socket for GitHub ↗︎

    This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

    View full report↗︎

    DaMandal0rian
    DaMandal0rian previously approved these changes Sep 22, 2024
    Base automatically changed from feat/add-sub-query-dictionary to main September 22, 2024 15:02
    @marc-aurele-besner marc-aurele-besner dismissed DaMandal0rian’s stale review September 22, 2024 15:02

    The base branch was changed.

    Copy link

    netlify bot commented Sep 22, 2024

    Deploy Preview for dev-astral canceled.

    Name Link
    🔨 Latest commit f713303
    🔍 Latest deploy log https://app.netlify.com/sites/dev-astral/deploys/66f1da0c95cf82000898a17f

    Copy link

    New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

    Package New capabilities Transitives Size Publisher
    npm/@autonomys/auto-consensus@0.2.0 None 0 50.2 kB marcaurelebesner
    npm/@autonomys/auto-utils@0.2.0 filesystem Transitive: environment, network +70 88.2 MB marcaurelebesner
    npm/@subsquid/graphql-server@4.5.1 Transitive: environment, eval, filesystem, network, shell, unsafe +226 27.2 MB subsquid
    npm/@subsquid/ss58@2.0.2 None +7 604 kB subsquid
    npm/@subsquid/substrate-metadata-explorer@3.1.2 Transitive: environment, eval, filesystem, network, shell +38 11.5 MB subsquid
    npm/@subsquid/substrate-processor@8.3.0 Transitive: environment, eval, filesystem, network, unsafe +64 17.9 MB subsquid
    npm/@subsquid/substrate-typegen@8.1.0 Transitive: environment, eval, filesystem, network, shell +33 13.1 MB subsquid
    npm/@subsquid/typeorm-codegen@2.0.0 Transitive: environment, eval, filesystem, network, shell, unsafe +209 26 MB subsquid
    npm/@subsquid/typeorm-migration@1.3.0 Transitive: environment, filesystem, shell +13 502 kB subsquid
    npm/@subsquid/typeorm-store@1.4.0 Transitive: environment, filesystem +11 497 kB subsquid
    npm/@types/node@20.12.7 None +1 2.1 MB types
    npm/pg@8.11.5 environment, network Transitive: filesystem +13 456 kB brianc
    npm/typeorm@0.3.20 environment, eval, filesystem Transitive: network, shell, unsafe +79 29.3 MB pleerock
    npm/typescript@5.4.5 None 0 32.4 MB typescript-bot

    🚮 Removed packages: npm/@babel/helper-validator-identifier@7.24.7, npm/@babel/highlight@7.24.7, npm/@emnapi/core@1.2.0, npm/@emnapi/runtime@1.2.0, npm/@emnapi/wasi-threads@1.0.1, npm/@hutson/parse-repository-url@3.0.2, npm/@isaacs/string-locale-compare@1.1.0, npm/@jest/schemas@29.6.3, npm/@lerna/create@8.1.8, npm/@napi-rs/wasm-runtime@0.2.4, npm/@npmcli/agent@2.2.2, npm/@npmcli/arborist@7.5.4, npm/@npmcli/fs@3.1.1, npm/@npmcli/git@5.0.8, npm/@npmcli/installed-package-contents@2.1.0, npm/@npmcli/map-workspaces@3.0.6, npm/@npmcli/metavuln-calculator@7.1.1, npm/@npmcli/name-from-folder@2.0.0, npm/@npmcli/node-gyp@3.0.0, npm/@npmcli/package-json@5.2.0, npm/@npmcli/promise-spawn@7.0.2, npm/@npmcli/query@3.1.0, npm/@npmcli/redact@2.0.1, npm/@npmcli/run-script@8.1.0, npm/@nrwl/devkit@19.7.3, npm/@nrwl/tao@19.7.3, npm/@nx/devkit@19.7.3, npm/@nx/nx-darwin-arm64@19.7.3, npm/@nx/nx-darwin-x64@19.7.3, npm/@nx/nx-freebsd-x64@19.7.3, npm/@nx/nx-linux-arm-gnueabihf@19.7.3, npm/@nx/nx-linux-arm64-gnu@19.7.3, npm/@nx/nx-linux-arm64-musl@19.7.3, npm/@nx/nx-linux-x64-gnu@19.7.3, npm/@nx/nx-linux-x64-musl@19.7.3, npm/@nx/nx-win32-arm64-msvc@19.7.3, npm/@nx/nx-win32-x64-msvc@19.7.3, npm/@octokit/auth-token@3.0.4, npm/@octokit/core@4.2.4, npm/@octokit/endpoint@7.0.6, npm/@octokit/graphql@5.0.6, npm/@octokit/openapi-types@18.1.1, npm/@octokit/plugin-enterprise-rest@6.0.1, npm/@octokit/plugin-paginate-rest@6.1.2, npm/@octokit/plugin-request-log@1.0.4, npm/@octokit/plugin-rest-endpoint-methods@7.2.3, npm/@octokit/request-error@3.0.3, npm/@octokit/request@6.2.8, npm/@octokit/rest@19.0.11, npm/@octokit/tsconfig@1.0.2, npm/@octokit/types@9.3.2, npm/@sigstore/bundle@2.3.2, npm/@sigstore/core@1.1.0, npm/@sigstore/protobuf-specs@0.3.2, npm/@sigstore/sign@2.3.2, npm/@sigstore/tuf@2.3.4, npm/@sigstore/verify@1.2.1, npm/@sinclair/typebox@0.27.8, npm/@tufjs/canonical-json@2.0.0, npm/@tufjs/models@2.0.1, npm/@tybys/wasm-util@0.9.0, npm/@types/minimatch@3.0.5, npm/@types/minimist@1.2.5, npm/@yarnpkg/lockfile@1.1.0, npm/@yarnpkg/parsers@3.0.0-rc.46, npm/@zkochan/js-yaml@0.0.7, npm/abbrev@2.0.0, npm/add-stream@1.0.0, npm/agent-base@7.1.1, npm/aggregate-error@3.1.0, npm/ansi-colors@4.1.3, npm/aproba@2.0.0, npm/array-differ@3.0.0, npm/array-ify@1.0.0, npm/arrify@2.0.1, npm/axios@1.7.7, npm/before-after-hook@2.2.3, npm/bin-links@4.0.4, npm/byte-size@8.1.1, npm/cacache@18.0.4, npm/callsites@3.1.0, npm/camelcase-keys@6.2.2, npm/camelcase@5.3.1, npm/chalk@4.1.0, npm/chownr@2.0.0, npm/ci-info@4.0.0, npm/clean-stack@2.2.0, npm/cli-width@3.0.0, npm/cmd-shim@6.0.3, npm/color-support@1.1.3, npm/columnify@1.6.0, npm/common-ancestor-path@1.0.1, npm/compare-func@2.0.0, npm/concat-stream@2.0.0, npm/console-control-strings@1.1.0, npm/conventional-changelog-angular@7.0.0, npm/conventional-changelog-core@5.0.1, npm/conventional-changelog-preset-loader@3.0.0, npm/conventional-changelog-writer@6.0.1, npm/conventional-commits-filter@3.0.0, npm/conventional-commits-parser@4.0.0, npm/conventional-recommended-bump@7.0.1, npm/core-util-is@1.0.3, npm/cosmiconfig@8.3.6, npm/cssesc@3.0.0, npm/dargs@7.0.0, npm/dateformat@3.0.3, npm/decamelize-keys@1.1.1, npm/decamelize@1.2.0, npm/dedent@1.5.3, npm/define-lazy-prop@2.0.0, npm/deprecation@2.3.1, npm/detect-indent@5.0.0, npm/diff-sequences@29.6.3, npm/dotenv-expand@11.0.6, npm/duplexer@0.1.2, npm/encoding@0.1.13, npm/enquirer@2.3.6, npm/env-paths@2.2.1, npm/envinfo@7.13.0, npm/err-code@2.0.3, npm/error-ex@1.3.2, npm/escape-string-regexp@1.0.5, npm/ethers@6.13.2, npm/eventemitter3@4.0.7, npm/execa@5.0.0, npm/exponential-backoff@3.1.1, npm/figures@3.2.0, npm/find-up@2.1.0, npm/front-matter@4.0.2, npm/fs-constants@1.0.0, npm/fs-minipass@3.0.3, npm/fs.realpath@1.0.0, npm/get-pkg-repo@4.2.1, npm/get-port@5.1.1, npm/get-stream@6.0.1, npm/git-raw-commits@3.0.0, npm/git-remote-origin-url@2.0.0, npm/git-semver-tags@5.0.1, npm/git-up@7.0.0, npm/git-url-parse@14.0.0, npm/gitconfiglocal@1.0.0, npm/glob-parent@6.0.2, npm/handlebars@4.7.8, npm/hard-rejection@2.1.0, npm/has-unicode@2.0.1, npm/hosted-git-info@7.0.2, npm/http-proxy-agent@7.0.2, npm/https-proxy-agent@7.0.5, npm/human-signals@2.1.0, npm/ignore-walk@6.0.5, npm/import-fresh@3.3.0, npm/import-local@3.1.0, npm/init-package-json@6.0.3, npm/inquirer@8.2.6, npm/ip-address@9.0.5, npm/is-arrayish@0.2.1, npm/is-ci@3.0.1, npm/is-core-module@2.15.1, npm/is-lambda@1.0.1, npm/is-plain-obj@1.1.0, npm/is-ssh@1.4.0, npm/is-stream@2.0.1, npm/is-text-path@1.0.1, npm/isarray@1.0.0, npm/isexe@3.1.1, npm/jest-diff@29.7.0, npm/jest-get-type@29.6.3, npm/js-tokens@4.0.0, npm/jsbn@1.1.0, npm/json-parse-even-better-errors@3.0.2, npm/json-stringify-nice@1.1.4, npm/jsonc-parser@3.2.0, npm/jsonparse@1.3.1, npm/jsonstream@1.3.5, npm/just-diff-apply@5.5.0, npm/just-diff@6.0.2, npm/lerna@8.1.8, npm/libnpmaccess@8.0.6, npm/libnpmpublish@9.0.9, npm/lines-and-columns@1.2.4, npm/load-json-file@6.2.0, npm/locate-path@2.0.0, npm/lodash.ismatch@4.4.0, npm/make-dir@4.0.0, npm/make-fetch-happen@13.0.1, npm/map-obj@1.0.1, npm/meow@8.1.2, npm/min-indent@1.0.1, npm/minimatch@9.0.3, npm/minimist-options@4.1.0, npm/minipass-collect@2.0.1, npm/minipass-fetch@3.0.5, npm/minipass-flush@1.0.5, npm/minipass-pipeline@1.2.4, npm/minipass-sized@1.0.3, npm/minizlib@2.1.2, npm/mkdirp@1.0.4, npm/modify-values@1.0.1, npm/multimatch@5.0.0, npm/node-fetch@2.6.7, npm/node-gyp@10.2.0, npm/node-machine-id@1.1.12, npm/nopt@7.2.1, npm/normalize-package-data@6.0.2, npm/npm-bundled@3.0.1, npm/npm-install-checks@6.3.0, npm/npm-normalize-package-bin@3.0.1, npm/npm-package-arg@11.0.3, npm/npm-packlist@8.0.2, npm/npm-pick-manifest@9.1.0, npm/npm-registry-fetch@17.1.0, npm/npm-run-path@4.0.1, npm/nx@19.7.3, npm/open@8.4.2, npm/ora@5.3.0, npm/p-finally@1.0.0, npm/p-limit@1.3.0, npm/p-locate@2.0.0, npm/p-map-series@2.1.0, npm/p-map@4.0.0, npm/p-pipe@3.1.0, npm/p-queue@6.6.2, npm/p-reduce@2.1.0, npm/p-timeout@3.2.0, npm/p-try@1.0.0, npm/p-waterfall@2.1.1, npm/pacote@18.0.6, npm/parent-module@1.0.1, npm/parse-conflict-json@3.0.1, npm/parse-json@5.2.0, npm/parse-path@7.0.0, npm/parse-url@8.1.0, npm/path-exists@3.0.0, npm/path-type@3.0.0, npm/pify@5.0.0, npm/pkg-dir@4.2.0, npm/postcss-selector-parser@6.1.2, npm/pretty-format@29.7.0, npm/proc-log@4.2.0, npm/process-nextick-args@2.0.1, npm/proggy@2.0.0, npm/promise-all-reject-late@1.0.1, npm/promise-call-limit@3.0.2, npm/promise-inflight@1.0.1, npm/promise-retry@2.0.1, npm/promzard@1.0.2, npm/protocols@2.0.1, npm/quick-lru@4.0.1, npm/react-is@18.3.1, npm/read-cmd-shim@4.0.0, npm/read-package-json-fast@3.0.2, npm/read-pkg-up@3.0.0, npm/read-pkg@3.0.0, npm/read@3.0.1, npm/redent@3.0.0, npm/resolve-cwd@3.0.0, npm/resolve-from@5.0.0, npm/retry@0.12.0, npm/rimraf@4.4.1, npm/run-async@2.4.1, npm/set-blocking@2.0.0, npm/sigstore@2.3.1, npm/smart-buffer@4.2.0, npm/socks-proxy-agent@8.0.4, npm/socks@2.8.3, npm/sort-keys@2.0.0, npm/spdx-correct@3.2.0, npm/spdx-exceptions@2.5.0, npm/spdx-expression-parse@3.0.1, npm/spdx-license-ids@3.0.20, npm/split2@3.2.2, npm/split@1.0.1, npm/sprintf-js@1.1.3, npm/ssri@10.0.6, npm/string-width-cjs@4.2.3, npm/strip-ansi-cjs@6.0.1, npm/strip-bom@4.0.0, npm/strip-final-newline@2.0.0, npm/strip-indent@3.0.0, npm/strong-log-transformer@2.1.0, npm/tar-stream@2.2.0, npm/tar@6.2.1, npm/temp-dir@1.0.0, npm/text-extensions@1.9.0, npm/through2@2.0.5, npm/through@2.3.8, npm/tmp@0.2.3, npm/treeverse@3.0.0, npm/trim-newlines@3.0.1, npm/tsconfig-paths@4.2.0, npm/tuf-js@2.2.1, npm/typedarray@0.0.6, npm/uglify-js@3.19.3, npm/unique-filename@3.0.0, npm/unique-slug@4.0.0, npm/universal-user-agent@6.0.1, npm/upath@2.0.1, npm/uuid@10.0.0, npm/validate-npm-package-license@3.0.4, npm/validate-npm-package-name@5.0.1, npm/walk-up-path@3.0.1, npm/which@4.0.0, npm/wide-align@1.1.5, npm/wrap-ansi-cjs@7.0.0, npm/write-file-atomic@5.0.1, npm/write-json-file@3.2.0, npm/write-pkg@4.0.0

    View full report↗︎

    @marc-aurele-besner marc-aurele-besner merged commit 766bea2 into main Sep 23, 2024
    13 checks passed
    @marc-aurele-besner marc-aurele-besner deleted the feat/fix-consensus-sub-query branch September 23, 2024 23:32
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    3 participants