From 8249ae0c41fea2fa39db4a5ed0b3d07bd0a0ebdf Mon Sep 17 00:00:00 2001 From: zeapoz Date: Wed, 17 Apr 2024 13:25:11 +0200 Subject: [PATCH 1/7] doc(readme): fix a few lints --- .gitignore | 3 +++ README.md | 75 ++++++++++++++++++++++++++++++++---------------------- 2 files changed, 48 insertions(+), 30 deletions(-) diff --git a/.gitignore b/.gitignore index c96deb9..c707bd6 100644 --- a/.gitignore +++ b/.gitignore @@ -17,3 +17,6 @@ StateSnapshot.json # Direnv files. .direnv/ .envrc + +# Markdown linting rules. +.markdownlint.json diff --git a/README.md b/README.md index 36613ef..38050b6 100644 --- a/README.md +++ b/README.md @@ -1,66 +1,80 @@ # zkSync State Reconstruction Tool + + +- [zkSync State Reconstruction Tool](#zksync-state-reconstruction-tool) + - [Prerequisites & setup](#prerequisites--setup) + - [Usage](#usage) + - [Additional commands](#additional-commands) + + > Tool / Library to reconstruct zkSync state from commit blocks published on L1 ## Prerequisites & setup + Currently there are three ways to setup the environment: using the provided Nix flake, the container image, or installing the dependencies manually.
- Nix Flake (Linux only) - To use the supplied Nix development environment you need to have Nix installed, This can be done by following the official instructions here.

+Nix Flake (Linux only) - Once Nix is installed, the development environment can be activated via the following command: +To use the supplied Nix development environment you need to have Nix installed, This can be done by following the official instructions here.

- ```nix - nix develop --experimental-features 'nix-command flakes' - ``` +Once Nix is installed, the development environment can be activated via the following command: - If you instead want to permanently enable the experimental flakes feature, you can do so by following the instructions detailed here. The environment can then be activated via: +```nix +nix develop --experimental-features 'nix-command flakes' +``` - ```nix - nix develop - ``` +If you instead want to permanently enable the experimental flakes feature, you can do so by following the instructions detailed here. The environment can then be activated via: + +```nix +nix develop +```
- Container Image - To build the container image, use: -

+Container Image +To build the container image, use: +

+ +```fish +podman build -t state-reconstruction:latest . +``` - ```fish - podman build -t state-reconstruction:latest . - ``` +And, to run it with `podman`, please use: - And, to run it with `podman`, please use: +```fish +podman run -it state-reconstruction:latest +``` - ```fish - podman run -it state-reconstruction:latest - ```
- Manually - This tool is written in nightly Rust; you can install Rust by following the official instructions here, and then running the following command to switch to the nightly toolchain: -

+Manually +This tool is written in nightly Rust; you can install Rust by following the official instructions here, and then running the following command to switch to the nightly toolchain: +

- ```fish - rustup toolchain install nightly - ``` +```fish +rustup toolchain install nightly +``` - You also need to have `protobuf`, version `3.20` or above, installed and accessible via `PATH`. Use your preferred package manager to do this. For example, using brew: +You also need to have `protobuf`, version `3.20` or above, installed and accessible via `PATH`. Use your preferred package manager to do this. For example, using brew: + +```fish +brew install protobuf +``` - ```fish - brew install protobuf - ```
> [!IMPORTANT] > It is highly recommend to override the maximum number of open file descriptors. Without doing so you may eventually run into an error, halting progress. On Unix machines this can be done by using `ulimit` along with the `-n` argument: +> > ```fish > ulimit -n 8192 > ``` ## Usage + To start reconstructing the state, run the following command with any valid HTTP/HTTPS Ethereum JSON-RPC endpoint, for example using `https://eth.llamarpc.com`: ```fish @@ -77,6 +91,7 @@ Once the tool is running it will continuously output the state reconstruction pr ``` On each block insert, the tool will compare the new state root hash with that published on L1. Should they differ, the tool will panic. You can then use the `query` command to get additional information, as such: + ```fish cargo run -- query root-hash From a75a79686e45f859965d4a61aa197838637de5fb Mon Sep 17 00:00:00 2001 From: zeapoz Date: Wed, 17 Apr 2024 13:56:24 +0200 Subject: [PATCH 2/7] doc(readme): add section about snapshots --- README.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/README.md b/README.md index 38050b6..23bedfd 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,10 @@ - [zkSync State Reconstruction Tool](#zksync-state-reconstruction-tool) - [Prerequisites & setup](#prerequisites--setup) - [Usage](#usage) + - [Reconstruction](#reconstruction) + - [Snapshots](#snapshots) + - [Generating snapshots](#generating-snapshots) + - [Importing snapshots](#importing-snapshots) - [Additional commands](#additional-commands) @@ -75,6 +79,8 @@ brew install protobuf ## Usage +### Reconstruction + To start reconstructing the state, run the following command with any valid HTTP/HTTPS Ethereum JSON-RPC endpoint, for example using `https://eth.llamarpc.com`: ```fish @@ -104,6 +110,35 @@ Metrics reference: - `CUR BLOCK`: The last block height that was processed. - `TOTAL BLOCKS PROCESSED`: The total number of blocks that has been processed since starting. +### Snapshots + +Additionally, the state reconstruction tool provides ways to interact with the upcoming [zkSync Era](https://github.com/matter-labs/zksync-era) snapshot system. + +#### Generating snapshots + +Before being able to generate snapshots, it is necessary to first fetch the data from L1 and process it. This can be done by running the following command which will also show progress similarly to when reconstructing state from L1. + +```fish +cargo run -- prepare-snapshot --http-url https://eth.llamarpc.com +``` + +Once the tool has gathered a number of storage logs, snapshots can then be exported by using the following command, specifying where to export the snapshot directory to. + +```fish +cargo run -- export-snapshot +``` + +#### Importing snapshots + +Snapshots can also be imported directly to use as a base when reconstructing the state from L1. This can be done by using the `--snapshot ` argument when starting reconstruction, like so: + +> [!WARNING] +> Importing a snapshot when reconstruction has progressed further than the snapshot has the same effect as truncating the reconstructed state to that of the snapshot. + +```fish +cargo run -- reconstruct --snapshot l1 --http-url +``` + ### Additional commands To view all available options, you can use the `help` command: From 86c9ad54b49f2b5e88857d9752d8787a6c14a0d1 Mon Sep 17 00:00:00 2001 From: zeapoz Date: Wed, 17 Apr 2024 13:59:28 +0200 Subject: [PATCH 3/7] doc(readme): add `--release` flag to all commands --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 23bedfd..266f581 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ brew install protobuf To start reconstructing the state, run the following command with any valid HTTP/HTTPS Ethereum JSON-RPC endpoint, for example using `https://eth.llamarpc.com`: ```fish -cargo run -- reconstruct l1 --http-url https://eth.llamarpc.com +cargo run --release -- reconstruct l1 --http-url https://eth.llamarpc.com ``` Once the tool is running it will continuously output the state reconstruction progress in the following format: @@ -99,7 +99,7 @@ Once the tool is running it will continuously output the state reconstruction pr On each block insert, the tool will compare the new state root hash with that published on L1. Should they differ, the tool will panic. You can then use the `query` command to get additional information, as such: ```fish -cargo run -- query root-hash +cargo run --release -- query root-hash Batch: Root Hash: @@ -119,13 +119,13 @@ Additionally, the state reconstruction tool provides ways to interact with the u Before being able to generate snapshots, it is necessary to first fetch the data from L1 and process it. This can be done by running the following command which will also show progress similarly to when reconstructing state from L1. ```fish -cargo run -- prepare-snapshot --http-url https://eth.llamarpc.com +cargo run --release -- prepare-snapshot --http-url https://eth.llamarpc.com ``` Once the tool has gathered a number of storage logs, snapshots can then be exported by using the following command, specifying where to export the snapshot directory to. ```fish -cargo run -- export-snapshot +cargo run --release -- export-snapshot ``` #### Importing snapshots @@ -136,7 +136,7 @@ Snapshots can also be imported directly to use as a base when reconstructing the > Importing a snapshot when reconstruction has progressed further than the snapshot has the same effect as truncating the reconstructed state to that of the snapshot. ```fish -cargo run -- reconstruct --snapshot l1 --http-url +cargo run --release -- reconstruct --snapshot l1 --http-url ``` ### Additional commands @@ -144,7 +144,7 @@ cargo run -- reconstruct --snapshot l1 --http-url To view all available options, you can use the `help` command: ```fish -cargo run -- --help +cargo run --release -- --help zkSync state reconstruction tool @@ -163,7 +163,7 @@ Options: You can also view all the options for the subcommands in a similar fashion: ```fish -cargo run -- reconstruct --help +cargo run --release -- reconstruct --help Reconstruct L2 state from a source From bb72e922ffc67e9b1a46c514a9966f936ca852c7 Mon Sep 17 00:00:00 2001 From: zeapoz Date: Wed, 17 Apr 2024 14:12:41 +0200 Subject: [PATCH 4/7] doc(readme): update help pages --- README.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 266f581..6f8b56d 100644 --- a/README.md +++ b/README.md @@ -151,9 +151,12 @@ zkSync state reconstruction tool Usage: state-reconstruct Commands: - reconstruct Reconstruct L2 state from a source - query Query the local storage, and optionally, return a JSON-payload of the data - help Print this message or the help of the given subcommand(s) + download Download L2 state from L1 to JSON file + reconstruct Reconstruct L2 state from a source + query Query the local storage, and optionally, return a JSON-payload of the data + prepare-snapshot + export-snapshot + help Print this message or the help of the given subcommand(s) Options: -h, --help Print help @@ -175,6 +178,7 @@ Commands: help Print this message or the help of the given subcommand(s) Options: - -d, --db-path The path to the storage solution [env: ZK_SYNC_DB_PATH=] - -h, --help Print help + -d, --db-path The path to the storage solution [env: ZK_SYNC_DB_PATH=] + --snapshot If present, try to restore state from snapshot files contained in the specified directory. Note that this will only work when supplied with a fresh database + -h, --help Print help ``` From ba0533c35e6c61a239093b7a1f85706e15548419 Mon Sep 17 00:00:00 2001 From: zeapoz Date: Wed, 17 Apr 2024 14:12:58 +0200 Subject: [PATCH 5/7] doc(readme): move metrics reference under collapsable --- README.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 6f8b56d..e120ec7 100644 --- a/README.md +++ b/README.md @@ -96,6 +96,14 @@ Once the tool is running it will continuously output the state reconstruction pr 2024-01-02T13:30:06.031946Z INFO PROGRESS: [ 0%] CUR BLOCK L1: 16644868 L2: 27 TOTAL BLOCKS PROCESSED L1: 16378 L2: 27 ``` +
+Metrics reference + +- `CUR BLOCK`: The last block height that was processed. +- `TOTAL BLOCKS PROCESSED`: The total number of blocks that has been processed since starting. + +
+ On each block insert, the tool will compare the new state root hash with that published on L1. Should they differ, the tool will panic. You can then use the `query` command to get additional information, as such: ```fish @@ -105,11 +113,6 @@ Batch: Root Hash: ``` -Metrics reference: - -- `CUR BLOCK`: The last block height that was processed. -- `TOTAL BLOCKS PROCESSED`: The total number of blocks that has been processed since starting. - ### Snapshots Additionally, the state reconstruction tool provides ways to interact with the upcoming [zkSync Era](https://github.com/matter-labs/zksync-era) snapshot system. From 99e3fe66a5e13bf8221d76eaea26b80ad1742c71 Mon Sep 17 00:00:00 2001 From: zeapoz Date: Wed, 17 Apr 2024 14:16:43 +0200 Subject: [PATCH 6/7] doc(readme): reorder --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e120ec7..f90b017 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # zkSync State Reconstruction Tool +> Tool to reconstruct zkSync state from commit blocks published on L1. + +## Table of contents + - [zkSync State Reconstruction Tool](#zksync-state-reconstruction-tool) - [Prerequisites & setup](#prerequisites--setup) @@ -11,8 +15,6 @@ - [Additional commands](#additional-commands) -> Tool / Library to reconstruct zkSync state from commit blocks published on L1 - ## Prerequisites & setup Currently there are three ways to setup the environment: using the provided Nix flake, the container image, or installing the dependencies manually. From 634907dd0a31115cee9dbba52d21bc8ca95d3505 Mon Sep 17 00:00:00 2001 From: zeapoz Date: Wed, 17 Apr 2024 14:54:20 +0200 Subject: [PATCH 7/7] doc(readme): replace comma with period --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f90b017..ae71be2 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ Currently there are three ways to setup the environment: using the provided Nix
Nix Flake (Linux only) -To use the supplied Nix development environment you need to have Nix installed, This can be done by following the official instructions here.

+To use the supplied Nix development environment you need to have Nix installed. This can be done by following the official instructions here.

Once Nix is installed, the development environment can be activated via the following command: