Cosmos Indexer Ecosystem
In this wiki is compiled research on various Cosmos Blockchain indexers developed by the community or private companies.
| Indexer | Link | Description |
|---|---|---|
| Cosmos Indexer | GitHub | Open-source Cosmos indexer. Useful for building applications like Sycamore. |
| GraphProtocol | GitHub | Components and infrastructure for the Graph Protocol indexer. |
| Numia | Website | The Cosmos Data & Development Platform. |
| SpaceBox | GitHub | Open-source tools for data indexation and storage. |
|
The Graph indexer defines a "decentralized protocol for indexing and querying blockchain data."
It is developed primarily for the Ethereum network, but recently began supporting Cosmos chains as well.
It works in the general following manner:
- Allow anyone to define data mappings for custom indexing requirements. The indexer will figure out what data they need and store it in the format that they need it in.
- Allowing anyone to define a "subgraph" - which is an open API that returns data that returns the above data mapping.
- A decentralized protocol allows feeding your own "subgraph" into the Graph ecosystem. Users can then query your subgraph using GraphQL, allowing them to add your indexed data into datasets provided by other "subgraphs"
- Decentralized indexers that will index the data requested by Subgraphs on the network.
The Graph Network is a blockchain of its own that is used to index Ethereum/Cosmos blockchain data.
It consists of the following general on-chain operators and concepts -
Indexers are node operators on the Graph Network. They are the primary operators of the indexing and querying services.
They choose Subgraphs to index based on the subgraph's curation signal, which is chosen by curators.
A custom API built on blockchain data that can be queried using GraphQL. Developers can build, deploy and publish subgraphs to The Graph's decentralized network. Then, Indexers can begin indexing subgraphs to make them available to be queried by subgraph consumers.
Delegators are network participants who delegate (i.e., "stake") GRT to one or more Indexers. Delegators contribute to securing the network without running a Graph Node themselves.
Curators signal which subgraphs are accurate and reliable. Doing so will net them some of the query fees paid to subgraphs. Signaling will indicate which subgrpahs should be indexed by Indexers.
The GraphProtocol is a decentralized network of indexers. Applications making use of the network will be relying on the network to index the data they need according to the Indexer/Subgraph/Curator model.
Cosmos Indexer maintains a single application model where the developer controls the code that runs, where it stores the data, and what is indexed.
Unlike The Graph, which exists as a decentralized network of indexers in an ecosystem with various stakeholders and investors, the Cosmos Indexer follows a more centralized application model. The users of the application control how it runs, where it runs and where it stores the data. The user of Cosmos Indexer also controls who has access to the data that is indexed.
Usage of the Graph Protocol network requires reliance on Curators to signal on your developed Subgraph. Signaling tells Indexers that the Subgraph is good, leading to the Indexers actually indexing the Subgraph.
This leads to a problem that using the Cosmos Indexer will not have: what happens to a Subgraph you really need indexed if no one is signaling on it? The Subgraph will not be picked up by Indexers and you will not be able to query the data your application needs.
Querying your Subgraphs requires an API Key for the Graph Network. This API Key is tied to billing information. Each query will incur fees that must be paid in the Network Token GRT.
On the other hand, you may setup the Cosmos Indexer on whatever infrastructure that can handle the application and you want to pay for and you may query it in any way you setup. The users have full control over the application.
Numia is a Cosmos Indexer that works like the following:
- Index multiple blockchains into a single database
- Index the following: source
- Blocks
- Transactions on blocks
- Messages on transactions
- Events and Attributes for messages
All of the indexed data makes its way into a Raw Table dataset, which contains all of the above indexed tables.
They also provide blockchain-specific Processed Tables. This is where they store specific datasets based on the needs of a blockchain (e.g. Osmosis Liquidity events are stored in downstream tables with specific data modeling - source). This is more akin to what I am proposing above, and basically how Cosmos Indexer works with its Taxable Transactions and Taxable Events.
Numia is very generalized. They have a specific general format all data raw data is passed to. They allow you to query for Blocks, Transactions, Messages and Events/Attributes. You would be able to find all particular attribute types for a given message. They also provide Processed Tables for Numia developed models of particular, block-chain specific datasets.
Cosmos Indexer provides a small subset of the same data that Numia provides in general, with some slight model differences. We index Blocks, Transactions and Messages. We do not index Events and Attributes. We turn Events and Attributes (as well as other raw data on the Messages themselves) into our proprietary Taxable Transactions model dataset. We also only do this for particular messages we have determined are "taxable" (as of the time of writing).
Numia is deployed by the Numia developers with users of the Numia application having to interface with the Numia application. As previously stated in the Graph comparison, Cosmos Indexer allows the user of the application to control how/where the application runs and who has access to the data it produces. Users have complete control of the application.
Numia provides public datasets in GCP BigQuery. For this, you will be dependent on the Numia developers to index the data into their proprietary formats and you will need to pay GCP costs to use BigQuery.
Numia indexes all Cosmos block chains according to a consistent table (as described above). For more blockchain-specific, custom models, they provide Processed Tables.
This is similar to the intended workflow for Cosmos Indexer with some caveats. The Cosmos Indexer will provide consistent models across blockchains (similar to the Raw Tables provided by Numia). It will also provide blockchain-specific models for certain data that does not fit the base models (for example, see the way the Cosmos Indexer handles Denom indexing). One further benefit of using Cosmos Indexer will be the ability to transform indexed data into alternate formats for easier querying of certain datasets (for example, see how the original iteration of the Cosmos Indexer handled Taxable Transactions).
Spacebox is a micro-service based indexer of Cosmos SDK-based blockchains. It aims to provide a stable, scalable architecture that guarantees data consistency.
It consists of a set of (as of time of writing) 4 main services that pass data along the following path:
- Crawler -> An application that gathers ALL data for each block, passing it to;
- A Kafka data broker -> A buffer that holds each block's data before it gets pulled by;
- Writer -> An application that gathers Kafka topics and stores specific datasets in;
- Clickhouse Database
There are also 2 MongoDB databases that sit next to:
- Crawler MongoDB: Stores block crawling processing statuses
- Writer MongoDB: Stores data on blocks that failed to be consumed
As of time of writing, Spacebox states they have support for the following base Cosmos SDK modules:
- bank
- gov
- auth
- mint
- staking
- distribution
- core
Spacebox follows a multi-service architecture with multiple intermediary data storage locations. Spacebox's indexer seems to store each Block module in its own table.
Cosmos Indexer is more of a monolith architecture, at least for its intended use-case of transaction indexing. The application is run in one location and interfaces with a single database. The application also aims to store module messages in generalized tables according to the type of the message.
Spacebox follows a multi-service architecture with multiple intermediary data storage locations, which may introduce complexity. In contrast, the Cosmos Indexer operates as a monolith application with a single purpose (move data from RPC nodes to a DB). The Cosmos Indexer also has partial support for many cosmos-sdk modules, including custom modules introduced by osmosis.
