Skip to content

Latest commit

 

History

History
104 lines (82 loc) · 7.25 KB

DESIGN.md

File metadata and controls

104 lines (82 loc) · 7.25 KB
Raw

VC-Ethereum Data Service Architecture

The pivotal decision in our architecture aimed at storing information for the most recent 50 blocks centers around our database choice. Given the project's scope and requirements detailed here, Redis was selected as our local store. The rationale behind this decision is thoroughly outlined here. This design strives to efficiently combine real-time and historical Ethereum data processing (most recent 50 blocks at the time of launching the application), leveraging Redis for both message brokering and data storage. The API Service ensures that clients have quick and reliable access to the latest blockchain data.

graph TD
    classDef ethereumNodeStyle fill:#7FB3D5,stroke:#333,stroke-width:1px,color:white;
    classDef blockNotificationStyle fill:#96C7C0,stroke:#333,stroke-width:1px,color:white;
    classDef bootstrapperStyle fill:#FAC15E,stroke:#333,stroke-width:1px,color:white;
    classDef redisChannelStyle fill:#F08A5D,stroke:#333,stroke-width:1px,color:white;
    classDef blockSubscriberStyle fill:#7AA6A6,stroke:#333,stroke-width:1px,color:white;
    classDef dataFormatterStyle fill:#9b59b6,stroke:#333,stroke-width:1px,color:white;
    classDef redisDBStyle fill:#F6CF71,stroke:#333,stroke-width:1px,color:black;
    classDef apiServiceStyle fill:#B6D7A8,stroke:#333,stroke-width:1px,color:black;
    classDef clientsStyle fill:#B2B2B2,stroke:#333,stroke-width:1px,color:black;

    subgraph VC-ETH-Data-Service
        direction TB
        ethereumNode["Ethereum Node<br/>(HTTPS RPC & WebSocket)"]:::ethereumNodeStyle
        blockNotification["Block<br/>Notification"]:::blockNotificationStyle
        bootstrapper["Bootstrapper"]:::bootstrapperStyle
        redisChannel(["Redis Channel"]):::redisChannelStyle
        blockSubscriber["Block<br/>Subscriber"]:::blockSubscriberStyle
        dataFormatter["Data<br/>Formatter"]:::dataFormatterStyle
        redisDB[("Redis DB")]:::redisDBStyle
        redisInsight("Redis Insight"):::redisDBStyle
        apiService["API Service"]:::apiServiceStyle
        clients["Clients"]:::clientsStyle

        ethereumNode <--> |WebSocket| blockNotification
        bootstrapper -.-> |HTTPS<br/>Request| ethereumNode
        ethereumNode -.-> |HTTPS<br/>Response| bootstrapper
        blockNotification --> |New Block<br/>Info| redisChannel
        bootstrapper --> |Latest 50<br/>Block Info| dataFormatter
        redisChannel --> blockSubscriber
        blockSubscriber --> |Processed<br/>Block Data| dataFormatter
        dataFormatter --> |Formatted<br/>Block Data| redisDB
        redisDB --> |Data<br/>Retrieval| apiService
        redisDB --> |Analysis| redisInsight
        clients --> |HTTP<br/>Request| apiService
        apiService --> |HTTP<br/>Response| clients
    end
Loading

Overall Flow of VC-Ethereum Data Service Architecture

The VC-Ethereum Data Service architecture streamlines the retrieval, processing, and distribution of Ethereum blockchain data through these components:

Ethereum Node

  • Role: Primary interface to the Ethereum blockchain, providing real-time updates via WebSocket and historical data (most recent 50 blocks) via HTTPS RPC.
  • Documentation: For more details on different JSON RPC calls supported, please refer to the Ethereum JSON-RPC.

Bootstrapper

  • Role: Retrieves the most recent 50 block data from the Ethereum Node using HTTPS RPC.
  • Flow: Initiates an HTTPS request to fetch the latest 50 blocks for initial synchronization, loads the data to Redis and then exits.
  • Documentation: For more details, please refer to the bootstrapper README.

Block Notification

  • Role: Listens for new blocks in real-time via WebSocket subscription from the Ethereum Node.
  • Flow: Establishes a bi-directional WebSocket connection to receive live updates on new blocks.
  • Documentation: For more details, please refer to the pub README.

Redis Channel

  • Role: Event-driven message broker facilitating asynchronous communication.
  • Flow: Receives new block information from Block Notification and forwards it to downstream components.

Block Subscriber

  • Role: Subscribes to the Redis Channel to process incoming block data.
  • Flow: Listens to the Redis Channel, processes block updates, and prepares data for further handling.
  • Documentation: For more details, please refer to the sub README.

Data Formatter

  • Role: Formats raw block data into a structured format as per model.Data struct definition.
  • Integration: Modular integration into Bootstrapper and Block Subscriber ensures consistent data formatting before storage.
  • Documentation: For more details, please refer to the model README.

Redis DB

  • Role: Central storage for processed Ethereum blockchain data.
  • Flow: Stores formatted block data received from upstream services with pre-define expiry time and enables efficient data retrieval.
  • Documentation: For more details, please refer to the storage README.

API Service

  • Role: External interface for clients to query Ethereum blockchain data.
  • Flow: Receives HTTP requests, retrieves requested data from Redis DB, and returns HTTP responses containing queried blockchain data.
  • Documentation: For more details, please refer to the v1 README.

Clients

  • Role: External consumers accessing Ethereum blockchain data.
  • Flow: Initiates HTTP requests to API Service to fetch specific blockchain data of interest.

Redis Insight

  • Role: Analyzes and visualizes data stored in Redis DB, offering insights into key creation, memory consumption and CPU usage.
  • Flow: Provides real-time GUI monitoring of Redis operations, complementing redis-cli.

Remarks

TTL for Blockdata in Redis

In the design, we start the Bootstrapper and BlockNotification services simultaneously. The Bootstrapper fetches the latest block height (h), retrieves data from h-50 to h, and exits gracefully. We set the TTL for all these retreived block information in Redis to 650 seconds (50 * 13 seconds, the average ETH block time). Concurrently, the block notifier publishes real-time block info to Redis with the same TTL. Initially, our datastore holds more than 50 blocks, but it eventually stabilizes at 50 blocks. Despite the initial load, Redis memory usage remains well within its capabilities.

Data Formatter

The Data Formatter module integrates as a component rather than a standalone service, ensuring uniform data formatting across Bootstrapper and Block Subscriber. This approach maximizes code reuse and data integrity within the VC-Ethereum Data Service architecture.