From ae10eafb5729524c790f06986563bb57e90aafb9 Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 25 Oct 2023 12:38:34 +1000 Subject: [PATCH 01/11] Update Dockerfile features comment (#7815) --- docker/Dockerfile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docker/Dockerfile b/docker/Dockerfile index f27ff99fa71..8966ff12c9a 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -12,8 +12,9 @@ # Each stage must define the build arguments (ARGs) it uses. # # Build zebrad with these features -# Keep these in sync with: -# https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/sub-build-docker-image.yml#L37 +# +# Keep these argument defaults in sync with GitHub vars.RUST_PROD_FEATURES and vars.RUST_TEST_FEATURES +# https://github.com/ZcashFoundation/zebra/settings/variables/actions ARG FEATURES="default-release-binaries" ARG TEST_FEATURES="lightwalletd-grpc-tests zebra-checkpoints" From c173198d4f75f23ebe54d5b61f23e6ca5cdbc6c7 Mon Sep 17 00:00:00 2001 From: Alfredo Garcia Date: Wed, 25 Oct 2023 00:27:16 -0300 Subject: [PATCH 02/11] docs(build): Add ECC dependencies documentation (#7794) * add ecc dependencies documentation * add `cargo deny` suggestion to deny.toml section Co-authored-by: Arya * change sententence syntax Co-authored-by: Arya * Apply suggestions from code review Co-authored-by: teor * move consistency check --------- Co-authored-by: Arya Co-authored-by: teor --- book/src/SUMMARY.md | 1 + book/src/dev/ecc-updates.md | 70 +++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) create mode 100644 book/src/dev/ecc-updates.md diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index aa49967efcc..9f43a554cb1 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -29,6 +29,7 @@ - [Continuous Delivery](dev/continuous-delivery.md) - [Generating Zebra Checkpoints](dev/zebra-checkpoints.md) - [Doing Mass Renames](dev/mass-renames.md) + - [Updating the ECC dependencies](dev/ecc-updates.md) - [Zebra RFCs](dev/rfcs.md) - [Pipelinable Block Lookup](dev/rfcs/0001-pipelinable-block-lookup.md) - [Parallel Verification](dev/rfcs/0002-parallel-verification.md) diff --git a/book/src/dev/ecc-updates.md b/book/src/dev/ecc-updates.md new file mode 100644 index 00000000000..f99805d1734 --- /dev/null +++ b/book/src/dev/ecc-updates.md @@ -0,0 +1,70 @@ +# Updating the ECC dependencies + +Zebra relies on numerous Electric Coin Company ([ECC](https://electriccoin.co/)) dependencies, and updating them can be a complex task. This guide will help you navigate the process. + + +The main dependency that influences that is [zcash](https://github.com/zcash/zcash) itself. This is because [zebra_script](https://github.com/ZcashFoundation/zcash_script) links to specific files from it (zcash_script.cpp and all on which it depends). Due to the architecture of zcash, this requires linking to a lot of seemingly unrelated dependencies like orchard, halo2, etc (which are all Rust crates). + +## Steps for upgrading + +Let's dive into the details of each step required to perform an upgrade: + +### Before starting + +- Zebra developers often dismiss ECC dependency upgrade suggestions from dependabot. For instance, see [this closed PR](https://github.com/ZcashFoundation/zebra/pull/7745) in favor of the [5.7.0 zcashd upgrade PR](https://github.com/ZcashFoundation/zebra/pull/7784), which followed this guide. + +- Determine the version of `zcashd` to use. This version will determine which versions of other crates to use. Typically, this should be a [tag](https://github.com/zcash/zcash/tags), but in some cases, it might be a reference to a branch (e.g., nu5-consensus) for testing unreleased developments. + +- Upgrading the `zcash_script` crate can be challenging, depending on changes in the latest `zcashd` release. Follow the instructions in the project's [README](https://github.com/ZcashFoundation/zcash_script/blob/master/README.md) for guidance. + +- Upgrade and release `zcash_script` before upgrading other ECC dependencies in Zebra. + +### Upgrade versions + +- Use the `cargo upgrade` command to upgrade all the ECC dependency versions in Zebra. For example, in [this PR](https://github.com/ZcashFoundation/zebra/pull/7784), the following command was used: + +``` +cargo upgrade --incompatible -p bridgetree -p incrementalmerkletree -p orchard -p zcash_primitives -p zcash_proofs -p zcash_address -p zcash_encoding -p zcash_note_encryption -p zcash_script +``` + +Notes: + +- Insert all the crate names to be updated to the command. + +- Use `crate-name@version` to upgrade to a specific version of that crate, instead of just the highest version. + +- You need to have [cargo upgrade](https://crates.io/crates/cargo-upgrades) and [cargo edit](https://crates.io/crates/cargo-edit) installed for this command to work. + +### Version consistency check + +- Ensure that the crate versions in the `Cargo.toml` of the zcashd release, `Cargo.toml` of `zcash_script`, and the `Cargo.toml` files of Zebra crates are all the same. Version consistency is crucial. + +### Build/Test zebra & fix issues + +- Build zebra and make sure it compiles. + +``` +cargo build +``` + +- Test Zebra and make sure all test code compiles and all tests pass: + +``` +cargo test +``` + +- When upgrading, it's common for things to break, such as deprecated or removed functionality. Address these issues by referring to the broken dependency's changelog, which often provides explanations and workarounds. + +- If you encounter issues that you can't resolve, consider reaching out to ECC team members who worked on the upgrade, as they may have more context. + +### Check `deny.toml` + +- Review Zebra's `deny.toml` file for potential duplicates that can be removed due to the upgrade. You may also need to add new entries to `deny.toml`. +- You can identify issues with the dependencies using `cargo deny check bans` command, need to have [cargo deny](https://crates.io/crates/cargo-deny) installed. +- Push your changes and let the CI identify any additional problems. + +### Push the Pull Request (PR) + +- Push the pull request with all the changes and ensure that the full CI process passes. +- Seek approval for the PR. +- Merge to `main` branch. From 0e00ef4f4f20e40d61f9b7788af8bffb169cbed7 Mon Sep 17 00:00:00 2001 From: teor Date: Wed, 25 Oct 2023 19:37:52 +1000 Subject: [PATCH 03/11] Increase Docker unit test timeout (#7814) --- .github/workflows/sub-test-zebra-config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/sub-test-zebra-config.yml b/.github/workflows/sub-test-zebra-config.yml index a7288d5f733..6ed561cbd7e 100644 --- a/.github/workflows/sub-test-zebra-config.yml +++ b/.github/workflows/sub-test-zebra-config.yml @@ -31,7 +31,7 @@ on: jobs: test-docker-config: name: Test ${{ inputs.test_id }} in Docker - timeout-minutes: 15 + timeout-minutes: 30 runs-on: ubuntu-latest steps: - uses: actions/checkout@v4.1.1 From 06f381230a7c32e99579ac6fa7f77dc321d4ecbe Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 25 Oct 2023 21:59:53 +0000 Subject: [PATCH 04/11] build(deps): bump Swatinem/rust-cache from 2.7.0 to 2.7.1 (#7825) Bumps [Swatinem/rust-cache](https://github.com/swatinem/rust-cache) from 2.7.0 to 2.7.1. - [Release notes](https://github.com/swatinem/rust-cache/releases) - [Changelog](https://github.com/Swatinem/rust-cache/blob/master/CHANGELOG.md) - [Commits](https://github.com/swatinem/rust-cache/compare/v2.7.0...v2.7.1) --- updated-dependencies: - dependency-name: Swatinem/rust-cache dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci-lint.yml | 4 ++-- .github/workflows/ci-unit-tests-os.yml | 4 ++-- .github/workflows/docs-deploy-firebase.yml | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci-lint.yml b/.github/workflows/ci-lint.yml index 02881a0280e..12561b019a2 100644 --- a/.github/workflows/ci-lint.yml +++ b/.github/workflows/ci-lint.yml @@ -86,7 +86,7 @@ jobs: run: | curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain=stable --profile=default - - uses: Swatinem/rust-cache@v2.7.0 + - uses: Swatinem/rust-cache@v2.7.1 with: shared-key: "clippy-cargo-lock" @@ -131,7 +131,7 @@ jobs: # We don't cache `fmt` outputs because the job is quick, # and we want to use the limited GitHub actions cache space for slower jobs. - #- uses: Swatinem/rust-cache@v2.7.0 + #- uses: Swatinem/rust-cache@v2.7.1 - run: | cargo fmt --all -- --check diff --git a/.github/workflows/ci-unit-tests-os.yml b/.github/workflows/ci-unit-tests-os.yml index 69ed6ad40bd..bbfa6dc1d6e 100644 --- a/.github/workflows/ci-unit-tests-os.yml +++ b/.github/workflows/ci-unit-tests-os.yml @@ -109,7 +109,7 @@ jobs: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain=${{ matrix.rust }} --profile=minimal - - uses: Swatinem/rust-cache@v2.7.0 + - uses: Swatinem/rust-cache@v2.7.1 # TODO: change Rust cache target directory on Windows, # or remove this workaround once the build is more efficient (#3005). #with: @@ -218,7 +218,7 @@ jobs: run: | curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain=stable --profile=minimal - - uses: Swatinem/rust-cache@v2.7.0 + - uses: Swatinem/rust-cache@v2.7.1 with: shared-key: "clippy-cargo-lock" diff --git a/.github/workflows/docs-deploy-firebase.yml b/.github/workflows/docs-deploy-firebase.yml index 806211131c1..29b85def5db 100644 --- a/.github/workflows/docs-deploy-firebase.yml +++ b/.github/workflows/docs-deploy-firebase.yml @@ -140,7 +140,7 @@ jobs: run: | curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain=beta --profile=default - - uses: Swatinem/rust-cache@v2.7.0 + - uses: Swatinem/rust-cache@v2.7.1 - name: Build external docs run: | @@ -198,7 +198,7 @@ jobs: run: | curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain=beta --profile=default - - uses: Swatinem/rust-cache@v2.7.0 + - uses: Swatinem/rust-cache@v2.7.1 - name: Build internal docs run: | From b244d5628cdf567edf3d061727173c33f757ce89 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 25 Oct 2023 22:00:04 +0000 Subject: [PATCH 05/11] build(deps): bump tj-actions/changed-files from 39.2.3 to 39.2.4 (#7826) Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 39.2.3 to 39.2.4. - [Release notes](https://github.com/tj-actions/changed-files/releases) - [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md) - [Commits](https://github.com/tj-actions/changed-files/compare/v39.2.3...v39.2.4) --- updated-dependencies: - dependency-name: tj-actions/changed-files dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci-lint.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci-lint.yml b/.github/workflows/ci-lint.yml index 12561b019a2..1a0d44362e5 100644 --- a/.github/workflows/ci-lint.yml +++ b/.github/workflows/ci-lint.yml @@ -37,7 +37,7 @@ jobs: - name: Rust files id: changed-files-rust - uses: tj-actions/changed-files@v39.2.3 + uses: tj-actions/changed-files@v39.2.4 with: files: | **/*.rs @@ -49,7 +49,7 @@ jobs: - name: Workflow files id: changed-files-workflows - uses: tj-actions/changed-files@v39.2.3 + uses: tj-actions/changed-files@v39.2.4 with: files: | .github/workflows/*.yml From a126acb16039a908c33a0b23cf3303f355294582 Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 26 Oct 2023 08:18:41 +1000 Subject: [PATCH 06/11] change(doc): Document state format upgrade implementation and testing (#7799) * Explain verification modes * Explain block write modes * Expand upgrade goals * Document testing requirements * Update current format docs * Move the detailed checklist to the ticket template * Move an example back * Reword confusing paragraphs Co-authored-by: Arya * Explain the difference between major and minor/patch versions * Simplify upgrade wording Co-authored-by: Marek --------- Co-authored-by: Arya Co-authored-by: Marek --- book/src/dev/state-db-upgrades.md | 141 +++++++++++++++++++++++++----- 1 file changed, 118 insertions(+), 23 deletions(-) diff --git a/book/src/dev/state-db-upgrades.md b/book/src/dev/state-db-upgrades.md index db5e4bce868..1c162c8ae29 100644 --- a/book/src/dev/state-db-upgrades.md +++ b/book/src/dev/state-db-upgrades.md @@ -1,18 +1,64 @@ # Zebra Cached State Database Implementation +## Current Implementation + +### Verification Modes +[verification]: #verification + +Zebra's state has two verification modes: +- block hash checkpoints, and +- full verification. + +This means that verification uses two different codepaths, and they must produce the same results. + +By default, Zebra uses as many checkpoints as it can, because they are more secure against rollbacks +(and some other kinds of chain attacks). Then it uses full verification for the last few thousand +blocks. + +When Zebra gets more checkpoints in each new release, it checks the previously verified cached +state against those checkpoints. This checks that the two codepaths produce the same results. + ## Upgrading the State Database +[upgrades]: #upgrades For most state upgrades, we want to modify the database format of the existing database. If we change the major database version, every user needs to re-download and re-verify all the blocks, which can take days. +### Writing Blocks to the State +[write-block]: #write-block + +Blocks can be written to the database via two different code paths, and both must produce the same results: + +- Upgrading a pre-existing database to the latest format +- Writing newly-synced blocks in the latest format + +This code is high risk, because discovering bugs is tricky, and fixing bugs can require a full reset +and re-write of an entire column family. + +Most Zebra instances will do an upgrade, because they already have a cached state, and upgrades are +faster. But we run a full sync in CI every week, because new users use that codepath. (And it is +their first experience of Zebra.) + +When Zebra starts up and shuts down (and periodically in CI tests), we run checks on the state +format. This makes sure that the two codepaths produce the same state on disk. + +To reduce code and testing complexity: +- when a previous Zebra version opens a newer state, the entire state is considered to have that lower version, and +- when a newer Zebra version opens an older state, each required upgrade is run on the entire state. + ### In-Place Upgrade Goals +[upgrade-goals]: #upgrade-goals +Here are the goals of in-place upgrades: - avoid a full download and rebuild of the state -- the previous state format must be able to be loaded by the new state +- Zebra must be able to upgrade the format from previous minor or patch versions of its disk format + (Major disk format versions are breaking changes. They create a new empty state and re-sync the whole chain.) - this is checked the first time CI runs on a PR with a new state version. After the first CI run, the cached state is marked as upgraded, so the upgrade doesn't run again. If CI fails on the first run, any cached states with that version should be deleted. +- the upgrade and full sync formats must be identical + - this is partially checked by the state validity checks for each upgrade (see above) - previous zebra versions should be able to load the new format - this is checked by other PRs running using the upgraded cached state, but only if a Rust PR runs after the new PR's CI finishes, but before it merges @@ -30,17 +76,26 @@ This means that: - it can't give incorrect results, because that can affect verification or wallets - it can return an error - it can only return an `Option` if the caller handles it correctly -- multiple upgrades must produce a valid state format +- full syncs and upgrades must write the same format + - the same write method should be called from both the full sync and upgrade code, + this helps prevent data inconsistencies +- repeated upgrades must produce a valid state format - if Zebra is restarted, the format upgrade will run multiple times - if an older Zebra version opens the state, data can be written in an older format -- the format must be valid before and after each database transaction or API call, because an upgrade can be cancelled at any time +- the format must be valid before and after each database transaction or API call, because an + upgrade can be cancelled at any time - multi-column family changes should made in database transactions - - if you are building new column family, disable state queries, then enable them once it's done + - if you are building new column family: + - disable state queries, then enable them once it's done, or + - do the upgrade in an order that produces correct results + (for example, some data is valid from genesis forward, and some from the tip backward) - if each database API call produces a valid format, transactions aren't needed -If there is an upgrade failure, it can panic and tell the user to delete their cached state and re-launch Zebra. +If there is an upgrade failure, panic and tell the user to delete their cached state and re-launch +Zebra. -### Performance Constraints +#### Performance Constraints +[performance]: #performance Some column family access patterns can lead to very poor performance. @@ -62,17 +117,50 @@ But we need to use iterators for some operations, so our alternatives are (in pr Currently only UTXOs require key deletion, and only `utxo_loc_by_transparent_addr_loc` requires deletion and iterators. -### Implementation Steps +### Required Tests +[testing]: #testing + +State upgrades are a high-risk change. They permanently modify the state format on production Zebra +instances. Format issues are tricky to diagnose, and require extensive testing and a new release to +fix. Deleting and rebuilding an entire column family can also be costly, taking minutes or hours the +first time a cached state is upgraded to a new Zebra release. -- [ ] update the [database format](https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/state-db-upgrades.md#current) in the Zebra docs -- [ ] increment the state minor version -- [ ] write the new format in the block write task -- [ ] update older formats in the format upgrade task -- [ ] test that the new format works when creating a new state, and updating an older state +Some format bugs can't be fixed, and require an entire rebuild of the state. For example, deleting +or corrupting transactions or block headers. -See the [upgrade design docs](https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/state-db-upgrades.md#design) for more details. +So testing format upgrades is extremely important. Every state format upgrade should test: +- new format serializations +- new calculations or data processing +- the upgrade produces a valid format +- a full sync produces a valid format -These steps can be copied into tickets. +Together, the tests should cover every code path. For example, the subtrees needed mid-block, +end-of-block, sapling, and orchard tests. They mainly used the validity checks for coverage. + +Each test should be followed by a restart, a sync of 200+ blocks, and another restart. This +simulates typical user behaviour. + +And ideally: +- An upgrade from the earliest supported Zebra version + (the CI sync-past-checkpoint tests do this on every PR) + +#### Manually Triggering a Format Upgrade +[manual-upgrade]: #manual-upgrade + +Zebra stores the current state minor and patch versions in a `version` file in the database +directory. This path varies based on the OS, major state version, network, and config. + +For example, the default mainnet state version on Linux is at: +`~/.cache/zebra/state/v25/mainnet/version` + +To upgrade a cached Zebra state from `v25.0.0` to the latest disk format, delete the version file. +To upgrade from a specific version `v25.x.y`, edit the file so it contains `x.y`. + +Editing the file and running Zebra will trigger a re-upgrade over an existing state. +Re-upgrades can hide format bugs. For example, if the old code was correct, and the +new code skips blocks, the validity checks won't find that bug. + +So it is better to test with a full sync, and an older cached state. ## Current State Database Format [current]: #current @@ -124,8 +212,13 @@ We use the following rocksdb column families: | `history_tree` | `()` | `NonEmptyHistoryTree` | Update | | `tip_chain_value_pool` | `()` | `ValueBalance` | Update | -Zcash structures are encoded using `ZcashSerialize`/`ZcashDeserialize`. -Other structures are encoded using `IntoDisk`/`FromDisk`. +### Data Formats +[rocksdb-data-format]: #rocksdb-data-format + +We use big-endian encoding for keys, to allow database index prefix searches. + +Most Zcash protocol structures are encoded using `ZcashSerialize`/`ZcashDeserialize`. +Other structures are encoded using custom `IntoDisk`/`FromDisk` implementations. Block and Transaction Data: - `Height`: 24 bits, big-endian, unsigned (allows for ~30 years worth of blocks) @@ -145,17 +238,21 @@ Block and Transaction Data: - `NoteCommitmentSubtreeIndex`: 16 bits, big-endian, unsigned - `NoteCommitmentSubtreeData<{sapling, orchard}::tree::Node>`: `Height \|\| {sapling, orchard}::tree::Node` -We use big-endian encoding for keys, to allow database index prefix searches. - Amounts: - `Amount`: 64 bits, little-endian, signed - `ValueBalance`: `[Amount; 4]` -Derived Formats: +Derived Formats (legacy): - `*::NoteCommitmentTree`: `bincode` using `serde` - stored note commitment trees always have cached roots -- `NonEmptyHistoryTree`: `bincode` using `serde`, using `zcash_history`'s `serde` implementation +- `NonEmptyHistoryTree`: `bincode` using `serde`, using our copy of an old `zcash_history` `serde` + implementation + +`bincode` is a risky format to use, because it depends on the exact order and type of struct fields. +Do not use it for new column families. +#### Address Format +[rocksdb-address-format]: #rocksdb-address-format The following figure helps visualizing the address index, which is the most complicated part. Numbers in brackets are array sizes; bold arrows are compositions (i.e. `TransactionLocation` is the @@ -200,6 +297,7 @@ Each column family handles updates differently, based on its specific consensus - Each key-value entry is created once. - Keys can be deleted, but values are never updated. - Code called by ReadStateService must ignore deleted keys, or use a read lock. + - We avoid deleting keys, and avoid using iterators on `Delete` column families, for performance. - TODO: should we prevent re-inserts of keys that have been deleted? - Update: - Each key-value entry is created once. @@ -353,8 +451,6 @@ So they should not be used for consensus-critical checks. the Merkle tree nodes as required to insert new items. For each block committed, the old tree is deleted and a new one is inserted by its new height. - **TODO:** store the sprout note commitment tree by `()`, - to avoid ReadStateService concurrent write issues. - The `{sapling, orchard}_note_commitment_tree` stores the note commitment tree state for every height, for the specific pool. Each tree is stored @@ -368,7 +464,6 @@ So they should not be used for consensus-critical checks. state. There is always a single entry for it. The tree is stored as the set of "peaks" of the "Merkle mountain range" tree structure, which is what is required to insert new items. - **TODO:** store the history tree by `()`, to avoid ReadStateService concurrent write issues. - Each `*_anchors` stores the anchor (the root of a Merkle tree) of the note commitment tree of a certain block. We only use the keys since we just need the set of anchors, From 71a9865b6cca905f6ed26843ee600480789354bf Mon Sep 17 00:00:00 2001 From: teor Date: Thu, 26 Oct 2023 09:42:27 +1000 Subject: [PATCH 07/11] remove(cmd): Remove `zebrad download` command, because it no longer does anything (#7819) * Remove the `zebrad download` command * Reorder command.rs imports * Remove zcash-params/Dockerfile and the main Dockerfile commands that use it * Stop building zcash-params Docker images in CI * Update CHANGELOG for `zebrad download` removal * Clarify why the image is smaller Co-authored-by: Marek --------- Co-authored-by: Marek --- .github/workflows/sub-build-zcash-params.yml | 45 -------------------- CHANGELOG.md | 10 +++-- docker/Dockerfile | 7 +-- docker/zcash-params/Dockerfile | 19 --------- zebrad/src/commands.rs | 37 +++++++--------- zebrad/src/commands/download.rs | 35 --------------- 6 files changed, 24 insertions(+), 129 deletions(-) delete mode 100644 .github/workflows/sub-build-zcash-params.yml delete mode 100644 docker/zcash-params/Dockerfile delete mode 100644 zebrad/src/commands/download.rs diff --git a/.github/workflows/sub-build-zcash-params.yml b/.github/workflows/sub-build-zcash-params.yml deleted file mode 100644 index ee08b6fd3c2..00000000000 --- a/.github/workflows/sub-build-zcash-params.yml +++ /dev/null @@ -1,45 +0,0 @@ -name: Build zcash-params - -# Ensures that only one workflow task will run at a time. Previous deployments, if -# already in process, won't get cancelled. Instead, we let the first to complete -# then queue the latest pending workflow, cancelling any workflows in between -concurrency: - group: ${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: false - -on: - workflow_dispatch: - inputs: - no_cache: - description: 'Disable the Docker cache for this build' - required: false - type: boolean - default: false - - push: - branches: - - 'main' - paths: - # parameter download code - - 'zebra-consensus/src/primitives/groth16/params.rs' - - 'zebra-consensus/src/router.rs' - - 'zebrad/src/commands/download.rs' - - 'zebrad/src/commands/start.rs' - # workflow definitions - - 'docker/zcash-params/Dockerfile' - - '.dockerignore' - - '.github/workflows/sub-build-zcash-params.yml' - - '.github/workflows/sub-build-docker-image.yml' - -jobs: - build: - name: Build Zcash Params Docker - uses: ./.github/workflows/sub-build-docker-image.yml - with: - dockerfile_path: ./docker/zcash-params/Dockerfile - dockerfile_target: release - image_name: zcash-params - no_cache: ${{ inputs.no_cache || false }} - rust_backtrace: full - rust_lib_backtrace: full - rust_log: info diff --git a/CHANGELOG.md b/CHANGELOG.md index d46c3ec28a6..beda30e185b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org). ## [Zebra 1.4.0](https://github.com/ZcashFoundation/zebra/releases/tag/v1.4.0) - TODO: DATE -Zebra's mining RPCs are now available in release builds. TODO: rest of intro +Zebra's mining RPCs are now available in release builds. Our Docker images are significantly smaller, +because the smaller Zcash verification parameters are now built into the `zebrad` binary. +TODO: rest of intro This release contains the following changes: @@ -21,10 +23,12 @@ read our [mining blog post](https://zfnd.org/experimental-mining-support-in-zebr Please [let us know](https://github.com/ZcashFoundation/zebra/issues/new?assignees=&labels=C-enhancement%2CS-needs-triage&projects=&template=feature_request.yml&title=feature%3A+) if your mining pool needs extra RPC methods or fields. -### Parameters in Binary +### Zcash Parameters in `zebrad` Binary `zebrad` now bundles zk-SNARK parameters directly into its binary. This increases the binary size -by a few megabytes, but these parameters do not need to be downloaded or stored separately. +by a few megabytes, but reduces the size of the Docker image by around 600 MB because +the parameters don't contain the Sprout proving key anymore. The `zebrad download` +command does nothing, so it has been removed. Previously, parameters were stored by default in these locations: diff --git a/docker/Dockerfile b/docker/Dockerfile index 8966ff12c9a..a0a4b9a9466 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -95,9 +95,7 @@ ENV CARGO_HOME="/opt/zebrad/.cargo/" # We also download needed dependencies for tests to work, from other images. # An entrypoint.sh is only available in this step for easier test handling with variables. FROM deps AS tests -# TODO: do not hardcode the user /root/ even though is a safe assumption -# Pre-download Zcash Sprout, Sapling parameters and Lightwalletd binary -COPY --from=us-docker.pkg.dev/zfnd-dev-zebra/zebra/zcash-params:edge /root/.zcash-params /root/.zcash-params + COPY --from=us-docker.pkg.dev/zfnd-dev-zebra/zebra/lightwalletd:edge /opt/lightwalletd /usr/local/bin # cargo uses timestamps for its cache, so they need to be in this order: @@ -176,11 +174,10 @@ RUN chmod u+x /entrypoint.sh # This stage is only used when deploying nodes or when only the resulting zebrad binary is needed # # To save space, this step starts from scratch using debian, and only adds the resulting -# binary from the `release` stage, and the Zcash Sprout & Sapling parameters from ZCash +# binary from the `release` stage FROM debian:bullseye-slim AS runtime COPY --from=release /opt/zebrad/target/release/zebrad /usr/local/bin COPY --from=release /entrypoint.sh / -COPY --from=us-docker.pkg.dev/zfnd-dev-zebra/zebra/zcash-params:edge /root/.zcash-params /root/.zcash-params RUN apt-get update && \ apt-get install -y --no-install-recommends \ diff --git a/docker/zcash-params/Dockerfile b/docker/zcash-params/Dockerfile deleted file mode 100644 index dce8153d185..00000000000 --- a/docker/zcash-params/Dockerfile +++ /dev/null @@ -1,19 +0,0 @@ -# This image is for caching Zcash Sprout and Sapling parameters. -# We don't test it automatically in CI due to download server rate-limiting. -# To manually run it on the PR branch before merging, go to: -# https://github.com/ZcashFoundation/zebra/actions/workflows/zcash-params.yml - -FROM debian:bullseye-slim AS release - -# Just use the precompiled zebrad binary from a recent release image. -# -# It doesn't matter what build or commit of Zebra we use, because it just calls into the -# zcash_proofs download code. (Which doesn't change much.) -# Test image zebrad binaries would also work, but it's harder to get a recent tag for them. -# -# Compiling the download-params example using `cargo ` is another alternative: -# `cargo run --locked --release --features default-docker --example download-params` -COPY --from=zfnd/zebra:latest /usr/local/bin/zebrad /usr/local/bin - -# Pre-download Zcash Sprout and Sapling parameters -RUN zebrad download diff --git a/zebrad/src/commands.rs b/zebrad/src/commands.rs index d7f4fa337be..70728ce9302 100644 --- a/zebrad/src/commands.rs +++ b/zebrad/src/commands.rs @@ -1,7 +1,16 @@ //! Zebrad Subcommands +use std::path::PathBuf; + +use abscissa_core::{config::Override, Command, Configurable, FrameworkError, Runnable}; + +use crate::config::ZebradConfig; + +pub use self::{entry_point::EntryPoint, start::StartCmd}; + +use self::{copy_state::CopyStateCmd, generate::GenerateCmd, tip_height::TipHeightCmd}; + mod copy_state; -mod download; mod entry_point; mod generate; mod start; @@ -10,18 +19,7 @@ mod tip_height; #[cfg(test)] mod tests; -use self::ZebradCmd::*; -use self::{ - copy_state::CopyStateCmd, download::DownloadCmd, generate::GenerateCmd, - tip_height::TipHeightCmd, -}; - -pub use self::{entry_point::EntryPoint, start::StartCmd}; - -use crate::config::ZebradConfig; - -use abscissa_core::{config::Override, Command, Configurable, FrameworkError, Runnable}; -use std::path::PathBuf; +use ZebradCmd::*; /// Zebrad Configuration Filename pub const CONFIG_FILE: &str = "zebrad.toml"; @@ -33,10 +31,6 @@ pub enum ZebradCmd { // TODO: hide this command from users in release builds (#3279) CopyState(CopyStateCmd), - // The `download` subcommand - /// Pre-download required Zcash Sprout and Sapling parameter files - Download(DownloadCmd), - /// Generate a default `zebrad.toml` configuration Generate(GenerateCmd), @@ -60,7 +54,7 @@ impl ZebradCmd { CopyState(_) | Start(_) => true, // Utility commands that don't use server components - Download(_) | Generate(_) | TipHeight(_) => false, + Generate(_) | TipHeight(_) => false, } } @@ -74,14 +68,14 @@ impl ZebradCmd { Start(_) => true, // Utility commands - CopyState(_) | Download(_) | Generate(_) | TipHeight(_) => false, + CopyState(_) | Generate(_) | TipHeight(_) => false, } } /// Returns true if this command should ignore errors when /// attempting to load a config file. pub(crate) fn should_ignore_load_config_error(&self) -> bool { - matches!(self, ZebradCmd::Generate(_) | ZebradCmd::Download(_)) + matches!(self, ZebradCmd::Generate(_)) } /// Returns the default log level for this command, based on the `verbose` command line flag. @@ -96,7 +90,7 @@ impl ZebradCmd { Generate(_) | TipHeight(_) => true, // Commands that generate informative logging output by default. - CopyState(_) | Download(_) | Start(_) => false, + CopyState(_) | Start(_) => false, }; if only_show_warnings && !verbose { @@ -113,7 +107,6 @@ impl Runnable for ZebradCmd { fn run(&self) { match self { CopyState(cmd) => cmd.run(), - Download(cmd) => cmd.run(), Generate(cmd) => cmd.run(), Start(cmd) => cmd.run(), TipHeight(cmd) => cmd.run(), diff --git a/zebrad/src/commands/download.rs b/zebrad/src/commands/download.rs deleted file mode 100644 index 4feefcb9e58..00000000000 --- a/zebrad/src/commands/download.rs +++ /dev/null @@ -1,35 +0,0 @@ -//! `download` subcommand - pre-download required parameter files -//! -//! `zebrad download` automatically downloads required parameter files the first time it is run. -//! -//! This command should be used if you're launching lots of `zebrad start` instances for testing, -//! or you want to include the parameter files in a distribution package. - -use abscissa_core::{Command, Runnable}; - -/// Pre-download required Zcash Sprout and Sapling parameter files -#[derive(Command, Debug, Default, clap::Parser)] -pub struct DownloadCmd {} - -impl DownloadCmd { - /// Download the Sapling and Sprout Groth16 parameters if needed, - /// check they were downloaded correctly, and load them into Zebra. - /// - /// # Panics - /// - /// If the downloaded or pre-existing parameter files are invalid. - fn download_and_check(&self) { - // The lazy static initializer does the download, if needed, - // and the file hash checks. - lazy_static::initialize(&zebra_consensus::groth16::GROTH16_PARAMETERS); - } -} - -impl Runnable for DownloadCmd { - /// Run the download command. - fn run(&self) { - info!("checking if Zcash Sapling and Sprout parameters have been downloaded"); - - self.download_and_check(); - } -} From 06ab5298778142872a74591ba9a17f4cb8ad4c7a Mon Sep 17 00:00:00 2001 From: Marek Date: Thu, 26 Oct 2023 02:06:40 +0200 Subject: [PATCH 08/11] docs: Refactor the installation instructions for `s-nomp` (#7835) * Remove references to `getblocktemplate-rpcs` * Refactor the sync description * Refactor the instructions for installing `s-nomp` * Mention Node 10 instead of 8.11 * Refactor the description of Zebra's sync Co-authored-by: teor --------- Co-authored-by: teor --- book/src/user/mining-testnet-s-nomp.md | 50 ++++++++++++++------------ 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/book/src/user/mining-testnet-s-nomp.md b/book/src/user/mining-testnet-s-nomp.md index 70fbc7a7647..fbffdb14d71 100644 --- a/book/src/user/mining-testnet-s-nomp.md +++ b/book/src/user/mining-testnet-s-nomp.md @@ -73,11 +73,11 @@ These fixes disable mining pool operator payments and miner payments: they just -2. [Build](https://github.com/ZcashFoundation/zebra#build-instructions) and [Run Zebra](https://zebra.zfnd.org/user/run.html) with the `getblocktemplate-rpcs` feature: +2. [Run Zebra](https://zebra.zfnd.org/user/run.html) with the config you created: ```sh - cargo run --release --features "getblocktemplate-rpcs" --bin zebrad -- -c zebrad.toml + zebrad -c zebrad.toml ``` -3. Wait a few hours for Zebra to sync to the testnet tip (on mainnet this takes 2-3 days) +3. Wait for Zebra to sync to the testnet tip. This takes 8-12 hours on testnet (or 2-3 days on mainnet) as of October 2023. ## Install `s-nomp` @@ -137,43 +137,49 @@ These fixes disable mining pool operator payments and miner payments: they just
Arch-specific instructions -#### Install dependencies +#### Install `s-nomp` -1. Install [`redis`](https://redis.io/docs/getting-started/) and run it on the default port: +1. Install Redis, and development libraries required by S-nomp ```sh - sudo pacman -S redis - sudo systemctl start redis + sudo pacman -S redis boost libsodium ``` - -2. Install and activate [`nvm`](https://github.com/nvm-sh/nvm#installing-and-updating): + +2. Install `nvm`, Python 3.10 and `virtualenv` ```sh - sudo pacman -S nvm - unset npm_config_prefix - source /usr/share/nvm/init-nvm.sh + paru -S python310 nvm + sudo pacman -S python-virtualenv ``` -3. Install `boost` and `libsodium` development libraries: - +3. Start Redis ```sh - sudo pacman -S boost libsodium + sudo systemctl start redis ``` -#### Install `s-nomp` +4. Clone the repository + + ```sh + git clone https://github.com/ZcashFoundation/s-nomp && cd s-nomp + ``` -1. `git clone https://github.com/ZcashFoundation/s-nomp && cd s-nomp` +5. Use Node 10: -2. Use the Zebra configs: `git checkout zebra-mining` + ```sh + unset npm_config_prefix + source /usr/share/nvm/init-nvm.sh + nvm install 10 + nvm use 10 + ``` -3. Use node 8.11.0: +6. Use Python 3.10 ```sh - nvm install 8.11.0 - nvm use 8.11.0 + virtualenv -p 3.10 s-nomp + source s-nomp/bin/activate ``` -4. Update dependencies and install: +7. Update dependencies and install: ```sh npm update From 879fd5f37e48a6b70a2113c01c25d56ebe6126f7 Mon Sep 17 00:00:00 2001 From: Alfredo Garcia Date: Thu, 26 Oct 2023 09:48:09 -0300 Subject: [PATCH 09/11] Revert #6825 (#7834) * revert #6825 * remove todo comment from CI --- .github/workflows/ci-unit-tests-os.patch.yml | 2 +- .github/workflows/ci-unit-tests-os.yml | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci-unit-tests-os.patch.yml b/.github/workflows/ci-unit-tests-os.patch.yml index 549f1809a51..4a97eb719f5 100644 --- a/.github/workflows/ci-unit-tests-os.patch.yml +++ b/.github/workflows/ci-unit-tests-os.patch.yml @@ -21,7 +21,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - os: [ubuntu-latest] + os: [ubuntu-latest, macos-latest] rust: [stable, beta] features: [""] exclude: diff --git a/.github/workflows/ci-unit-tests-os.yml b/.github/workflows/ci-unit-tests-os.yml index bbfa6dc1d6e..c433bf82163 100644 --- a/.github/workflows/ci-unit-tests-os.yml +++ b/.github/workflows/ci-unit-tests-os.yml @@ -74,8 +74,7 @@ jobs: fail-fast: false matrix: # TODO: Windows was removed for now, see https://github.com/ZcashFoundation/zebra/issues/3801 - # TODO: macOS tests were removed for now, see https://github.com/ZcashFoundation/zebra/issues/6824 - os: [ubuntu-latest] + os: [ubuntu-latest, macos-latest] rust: [stable, beta] # TODO: When vars.EXPERIMENTAL_FEATURES has features in it, add it here. # Or work out a way to trim the space from the variable: GitHub doesn't allow empty variables. From 64bf38c17ccf0fa4ce05a8a3339cf845e98b3570 Mon Sep 17 00:00:00 2001 From: Pili Guerra Date: Thu, 26 Oct 2023 21:17:47 +0200 Subject: [PATCH 10/11] Make macOS a tier 2 supported platform in the docs. closes #7824 (#7843) --- book/src/user/supported-platforms.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/book/src/user/supported-platforms.md b/book/src/user/supported-platforms.md index 152be56537e..8060b1b0054 100644 --- a/book/src/user/supported-platforms.md +++ b/book/src/user/supported-platforms.md @@ -34,6 +34,7 @@ For the full requirements, see [Tier 2 platform policy](platform-tier-policy.md# | -------|-------|-------|-------|------- | `x86_64-unknown-linux-gnu` | [GitHub ubuntu-latest](https://github.com/actions/virtual-environments#available-environments) | 64-bit | [latest stable release](https://github.com/rust-lang/rust/releases) | N/A | `x86_64-unknown-linux-gnu` | [GitHub ubuntu-latest](https://github.com/actions/virtual-environments#available-environments) | 64-bit | [latest beta release](https://github.com/rust-lang/rust/blob/beta/src/version) | N/A +| `x86_64-apple-darwin` | [GitHub macos-latest](https://github.com/actions/virtual-environments#available-environments) | 64-bit | [latest stable release](https://github.com/rust-lang/rust/releases) | N/A ## Tier 3 @@ -45,6 +46,4 @@ For the full requirements, see [Tier 3 platform policy](platform-tier-policy.md# | platform | os | notes | rust | artifacts | -------|-------|-------|-------|------- -| `aarch64-unknown-linux-gnu` | [Debian 11](https://www.debian.org/releases/bullseye/) | 64-bit | [latest stable release](https://github.com/rust-lang/rust/releases) | N/A -| `x86_64-apple-darwin` | [GitHub macos-latest](https://github.com/actions/virtual-environments#available-environments) | 64-bit | [latest stable release](https://github.com/rust-lang/rust/releases) | N/A - +| `aarch64-unknown-linux-gnu` | [Debian 11](https://www.debian.org/releases/bullseye/) | 64-bit | [latest stable release](https://github.com/rust-lang/rust/releases) | N/A \ No newline at end of file From ce22be37dcddd476737ccaef22bd5cfce9fc3667 Mon Sep 17 00:00:00 2001 From: teor Date: Fri, 27 Oct 2023 08:08:22 +1000 Subject: [PATCH 11/11] change(docs): Add macOS M1 as a tier 3 supported platform (#7851) * Add macOS M1 as a tier 3 supported platform * Add apple machine brand name that users will know --- book/src/user/supported-platforms.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/book/src/user/supported-platforms.md b/book/src/user/supported-platforms.md index 8060b1b0054..d42246d091f 100644 --- a/book/src/user/supported-platforms.md +++ b/book/src/user/supported-platforms.md @@ -46,4 +46,5 @@ For the full requirements, see [Tier 3 platform policy](platform-tier-policy.md# | platform | os | notes | rust | artifacts | -------|-------|-------|-------|------- -| `aarch64-unknown-linux-gnu` | [Debian 11](https://www.debian.org/releases/bullseye/) | 64-bit | [latest stable release](https://github.com/rust-lang/rust/releases) | N/A \ No newline at end of file +| `aarch64-unknown-linux-gnu` | [Debian 11](https://www.debian.org/releases/bullseye/) | 64-bit | [latest stable release](https://github.com/rust-lang/rust/releases) | N/A +| `aarch64-apple-darwin` | latest macOS | 64-bit, Apple M1 or M2 | [latest stable release](https://github.com/rust-lang/rust/releases) | N/A