Skip to content

Commit

Permalink
Create README_alpha-release_Marconi.md (#591)
Browse files Browse the repository at this point in the history
* Create README_alpha-release_Marconi.md

* Update README_alpha-release_Marconi.md

* Update plutus-chain-index/README_alpha-release_Marconi.md

Co-authored-by: koslambrou <konstantinos.lambrou@iohk.io>

* Update plutus-chain-index/README_alpha-release_Marconi.md

Co-authored-by: koslambrou <konstantinos.lambrou@iohk.io>

* Update plutus-chain-index/README_alpha-release_Marconi.md

Co-authored-by: Radu Ometita <radu.ometita@gmail.com>

* Update plutus-chain-index/README_alpha-release_Marconi.md

Co-authored-by: Radu Ometita <radu.ometita@gmail.com>

* Update plutus-chain-index/README_alpha-release_Marconi.md

Co-authored-by: Radu Ometita <radu.ometita@gmail.com>

* Apply suggestions from code review

Co-authored-by: Radu Ometita <radu.ometita@gmail.com>

* Update plutus-chain-index/README_alpha-release_Marconi.md

Co-authored-by: Radu Ometita <radu.ometita@gmail.com>

* Update README_alpha-release_Marconi.md

Incorporating comments received so far. Some comments/suggested content from Radu seems to have disappeared, so I've asked Radu to post them again.

* Update plutus-chain-index/README_alpha-release_Marconi.md

Radu's most recent additional content

Co-authored-by: Radu Ometita <radu.ometita@gmail.com>

* Update README_alpha-release_Marconi.md

Adding content from Radu, additional piece

Co-authored-by: koslambrou <konstantinos.lambrou@iohk.io>
Co-authored-by: Radu Ometita <radu.ometita@gmail.com>
  • Loading branch information
3 people committed Jul 14, 2022
1 parent 30bc533 commit 4cc6c84
Showing 1 changed file with 125 additions and 0 deletions.
125 changes: 125 additions & 0 deletions plutus-chain-index/README_alpha-release_Marconi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
![Marconi-Logo_sm](https://user-images.githubusercontent.com/104791413/178748103-077ed7e0-6071-461f-b9ed-97b3b7bd5b24.png)

# Marconi Alpha Release

## The Cardano blockchain indexer for dApp developers

Marconi is a lightweight, customizable chain follower application and library for dApp developers who need to index and query the Cardano blockchain.

## Introduction

Marconi is a blockchain follower application and library that you can configure for the types of information that you need to index and according to how you need to structure your database schema. Marconi provides a way for dApps to have access to data that has already been stored on-chain by indexing portions of the blockchain.

Marconi gives you the ability to customize the set of data that is indexed. It provides a library of functionality with which you can do the following:
* build your own indexers,
* work only with the data that you care about for your application,
* use whatever storage engine you prefer, and
* support only the queries that you need to support.

### Scalability through Customized Solutions for Each Use Case

The philosophy behind Marconi is to maintain scalability by customizing the solution for each use case. You can filter the streams of information to transmit only the data you need for your dApp, minimizing network traffic and database storage. You can also customize database schemas and queries so that the dApps have the most efficient and scalable application-specific API to interact with the data.

### Indexing solution ###

Any indexing solution must deal with blockchain rollbacks, which can happen as the network achieves consensus. Marconi is designed to deal with rollbacks as elegantly as possible while simultaneously finding an effective balance between memory, disk and CPU usage.

Managing rollbacks is very simple and fast. Marconi drops the events that were rolled back. Because the events that can be rolled back are stored in memory, no complicated logic is required to undo the projection of the list of events on disk, which Marconi would need if it stored everything on disk as quickly as possible.

Making `K` configurable makes the design quite scalable. Developers do not usually need to guard themselves against rollbacks by `K` blocks, so they can choose to store only 10 events in memory, for example, thereby sacrificing the correctness of handling rollbacks for memory usage.

In the event of a restart, recovery is very simple. Marconi stores only fully confirmed transactions, so there is nothing to do other than to load the state persisted on disk and to follow the blockchain from the corresponding chain-point.

## Queries

The indexed data is accessible through queries. There are no constraints on the format of queries or on the results. Both are identified by a type variable that the indexer exposes. The user, such as a dApp developer, provides the code for the query, result types and the functions used to query and store events on disk.

This is currently the only area where you can customize the indexer by providing functions that handle queries and storage. Note that the queries have to run on both the in-memory data and the data that is stored.

# Installation

Prerequisite: You already have a Cardano node running.

## Example command for launching Cardano node

```
cardano-node run \
--config mainnet-config.json \
--topology mainnet-topology.json \
--database-path db \
--socket-path /tmp/node-mainnet.sock \
--port 3003
```

The configuration files are available here:

[`configuration-files`](https://hydra.iohk.io/build/7654130/download/1/index.html)

## Marconi executable

The `marconi` executable is available as a nix flake.

You may either clone the [`plutus-apps`](https://github.com/input-output-hk/plutus-apps)
repository and run from the top-level:
```
nix build .#marconi
```
Or you may run from anywhere:
```
nix build github:input-output-hk/plutus-apps#marconi
```

Both commands will produce a `result` directory containing the executable
`result/bin/marconi`.

## Example command for launching Marconi
The synchronisation should take about 3.5h and it will consume around 9Gb of RAM, when synchronising both indexers at the same time.
```
./result/bin/marconi --socket-path /tmp/node-mainnet.sock --mainnet --datum-db datum.sqlite3 --utxo-db utxo.sqlite3
```
## Querying the SQL Database
A. Datum indexer

The database contains only one table defined as:

```
CREATE TABLE kv_datumhsh_datum (datumHash TEXT PRIMARY KEY, datum BLOB, slotNo INT);
```

The basic query that you can run is:

```
SELECT datum from kv_datumhsh_datum WHERE datumHash = ?;
```

B. Utxo indexer

The database contains two tables, one for unspent transactions and one for spent inputs. They are defined as follows:

```
CREATE TABLE utxos (address TEXT NOT NULL, txId TEXT NOT NULL, inputIx INT NOT NULL);
CREATE TABLE spent (txId TEXT NOT NULL, inputIx INT NOT NULL);
```

After you are fully synchronised you may run a couple of SQL statements to make sure that the query speed is optimised.

First, to create indices:

```
CREATE INDEX utxo_address ON utxos (address);
CREATE INDEX utxo_refs ON utxos (txId, inputIx);
CREATE INDEX spent_refs ON spent (txId, inputIx);
```

Then, to match inputs and outputs and trim the database:

```
DELETE FROM utxos WHERE utxos.rowid IN (SELECT utxos.rowid FROM utxos LEFT JOIN spent on utxos.txId = spent.txId AND utxos.inputIx = spent.inputIx WHERE spent.txId IS NOT NULL);
VACUUM;
```

And we can run the query using:

```
SELECT address, txId, inputIx FROM utxos LEFT JOIN spent ON utxos.txId = spent.txId AND utxos.inputIx = spent.inputIx WHERE utxos.txId IS NULL AND utxos.address = ?
```

0 comments on commit 4cc6c84

Please sign in to comment.