Merge branch 'main' into issue7903

ZcashFoundation · Nov 8, 2023 · d8af410 · d8af410
2 parents 48a1d3a + 2fad357
commit d8af410
Show file tree

Hide file tree

Showing 101 changed files with 1,063 additions and 697 deletions.
diff --git a/.github/PULL_REQUEST_TEMPLATE/release-checklist.md b/.github/PULL_REQUEST_TEMPLATE/release-checklist.md
@@ -131,8 +131,10 @@ The end of support height is calculated from the current blockchain height:
 
 ## Test the Pre-Release
 
-- [ ] Wait until the [Docker binaries have been built on `main`](https://github.com/ZcashFoundation/zebra/actions/workflows/continous-integration-docker.yml), and the quick tests have passed.
-- [ ] Wait until the [pre-release deployment machines have successfully launched](https://github.com/ZcashFoundation/zebra/actions/workflows/continous-delivery.yml)
+- [ ] Wait until the Docker binaries have been built on `main`, and the quick tests have passed:
+    - [ ] [ci-unit-tests-docker.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-unit-tests-docker.yml?query=branch%3Amain)
+    - [ ] [ci-integration-tests-gcp.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-integration-tests-gcp.yml?query=branch%3Amain)
+- [ ] Wait until the [pre-release deployment machines have successfully launched](https://github.com/ZcashFoundation/zebra/actions/workflows/cd-deploy-nodes-gcp.yml?query=event%3Arelease)
 
 ## Publish Release
 
@@ -148,7 +150,7 @@ The end of support height is calculated from the current blockchain height:
       and put the output in a comment on the PR.
 
 ## Publish Docker Images
-- [ ] Wait for the [the Docker images to be published successfully](https://github.com/ZcashFoundation/zebra/actions/workflows/release-binaries.yml).
+- [ ] Wait for the [the Docker images to be published successfully](https://github.com/ZcashFoundation/zebra/actions/workflows/release-binaries.yml?query=event%3Arelease).
 - [ ] Un-freeze the [`batched` queue](https://dashboard.mergify.com/github/ZcashFoundation/repo/zebra/queues) using Mergify.
 - [ ] Remove `do-not-merge` from the PRs you added it to
 

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -1,49 +1,72 @@
 ## Motivation
-
 <!--
 Thank you for your Pull Request.
-How does this change improve Zebra?
+Does it close any issues?
 -->
+_What are the most important goals of the ticket or PR?_
+
+
+### PR Author Checklist
+
+#### Check before marking the PR as ready for review:
+  - [ ] Will the PR name make sense to users?
+  - [ ] Does the PR have a priority label?
+  - [ ] Have you added or updated tests?
+  - [ ] Is the documentation up to date?
+
+##### For significant changes:
+  - [ ] Is there a summary in the CHANGELOG?
+  - [ ] Can these changes be split into multiple PRs?
+
+_If a checkbox isn't relevant to the PR, mark it as done._
 
 ### Specifications
 
 <!--
 If this PR changes consensus rules, quote them, and link to the Zcash spec or ZIP:
 https://zips.z.cash/#nu5-zips
-If this PR changes network behaviour, quote and link to the Bitcoin network reference:
-https://developer.bitcoin.org/reference/p2p_networking.html
 -->
 
+
 ### Complex Code or Requirements
 
 <!--
 Does this PR change concurrency, unsafe code, or complex consensus rules?
-If it does, ask for multiple reviewers on this PR.
+If it does, label this PR with `extra-reviews`.
 -->
 
+
 ## Solution
 
 <!--
 Summarize the changes in this PR.
-Does it close any issues?
-If this is a large change, list commits of key functional changes here.
 -->
 
+
+### Testing
+
+<!--
+Which tests were changed or added in this PR? Were there manual tests?
+-->
+
+
 ## Review
 
 <!--
 Is this PR blocking any other work?
 If you want specific reviewers for this PR, tag them here.
 -->
 
+
 ### Reviewer Checklist
 
-  - [ ] Will the PR name make sense to users?
-    - [ ] Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
-  - [ ] Are the PR labels correct?
-  - [ ] Does the code do what the ticket and PR says?
-    - [ ] Does it change concurrent code, unsafe code, or consensus rules?
-  - [ ] How do you know it works? Does it have tests?
+Check before approving the PR:
+  - [ ] Does the PR scope match the ticket?
+  - [ ] Are there enough tests to make sure it works? Do the tests cover the PR motivation?
+  - [ ] Are all the PR blockers dealt with?
+        PR blockers can be dealt with in new tickets or PRs.
+
+_And check the PR Author checklist is complete._
 
 ## Follow Up Work
 

diff --git a/.github/workflows/cd-deploy-nodes-gcp.yml b/.github/workflows/cd-deploy-nodes-gcp.yml
@@ -148,7 +148,7 @@ jobs:
     with:
       test_id: 'custom-conf'
       docker_image: ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
-      grep_patterns: '-e "v1.0.0-rc.2.toml"'
+      grep_patterns: '-e "loaded zebrad config.*config_path.*=.*v1.0.0-rc.2.toml"'
       test_variables: '-e NETWORK -e ZEBRA_CONF_PATH="zebrad/tests/common/configs/v1.0.0-rc.2.toml"'
       network: ${{ inputs.network || vars.ZCASH_NETWORK }}
 

diff --git a/.github/workflows/ci-lint.yml b/.github/workflows/ci-lint.yml
@@ -37,7 +37,7 @@ jobs:
 
       - name: Rust files
         id: changed-files-rust
-        uses: tj-actions/changed-files@v39.2.4
+        uses: tj-actions/changed-files@v40.0.2
         with:
           files: |
             **/*.rs
@@ -49,7 +49,7 @@ jobs:
 
       - name: Workflow files
         id: changed-files-workflows
-        uses: tj-actions/changed-files@v39.2.4
+        uses: tj-actions/changed-files@v40.0.2
         with:
           files: |
             .github/workflows/*.yml

diff --git a/.github/workflows/ci-unit-tests-docker.patch.yml b/.github/workflows/ci-unit-tests-docker.patch.yml
@@ -26,13 +26,6 @@ on:
       - '.github/workflows/sub-build-docker-image.yml'
 
 jobs:
-  # We don't patch the testnet job, because testnet isn't required to merge (it's too unstable)
-  get-available-disks:
-    name: Check if cached state disks exist for Mainnet / Check if cached state disks exist
-    runs-on: ubuntu-latest
-    steps:
-      - run: 'echo "No build required"'
-
   build:
     name: Build CI Docker / Build images
     runs-on: ubuntu-latest

diff --git a/.github/workflows/ci-unit-tests-docker.yml b/.github/workflows/ci-unit-tests-docker.yml
@@ -237,7 +237,7 @@ jobs:
     with:
       test_id: 'custom-conf'
       docker_image: ${{ vars.GAR_BASE }}/${{ vars.CI_IMAGE_NAME }}@${{ needs.build.outputs.image_digest }}
-      grep_patterns: '-e "v1.0.0-rc.2.toml"'
+      grep_patterns: '-e "loaded zebrad config.*config_path.*=.*v1.0.0-rc.2.toml"'
       test_variables: '-e NETWORK -e ZEBRA_CONF_PATH="zebrad/tests/common/configs/v1.0.0-rc.2.toml"'
       network: ${{ inputs.network || vars.ZCASH_NETWORK }}
 

diff --git a/.github/workflows/ci-unit-tests-os.yml b/.github/workflows/ci-unit-tests-os.yml
@@ -51,7 +51,6 @@ on:
       - '**/clippy.toml'
       # workflow definitions
       - '.github/workflows/ci-unit-tests-os.yml'
-      - '.github/workflows/sub-build-docker-image.yml'
 
 env:
   CARGO_INCREMENTAL: ${{ vars.CARGO_INCREMENTAL }}

diff --git a/.github/workflows/docs-deploy-firebase.patch.yml b/.github/workflows/docs-deploy-firebase.patch.yml
@@ -27,13 +27,6 @@ jobs:
     steps:
       - run: 'echo "No build required"'
 
-  build-docs-external:
-    name: Build and Deploy Zebra External Docs
-    timeout-minutes: 45
-    runs-on: ubuntu-latest
-    steps:
-      - run: 'echo "No build required"'
-
   build-docs-internal:
     name: Build and Deploy Zebra Internal Docs
     timeout-minutes: 45

diff --git a/.github/workflows/docs-deploy-firebase.yml b/.github/workflows/docs-deploy-firebase.yml
@@ -114,62 +114,6 @@ jobs:
           projectId: ${{ vars.GCP_FIREBASE_PROJECT }}
           target: docs-book
 
-  build-docs-external:
-    name: Build and Deploy Zebra External Docs
-    timeout-minutes: 45
-    runs-on: ubuntu-latest
-    permissions:
-      checks: write
-      contents: 'read'
-      id-token: 'write'
-      pull-requests: write
-    steps:
-      - name: Checkout the source code
-        uses: actions/checkout@v4.1.1
-        with:
-          persist-credentials: false
-
-      - name: Install last version of Protoc
-        uses: arduino/setup-protoc@v2.1.0
-        with:
-          version: '23.x'
-          repo-token: ${{ secrets.GITHUB_TOKEN }}
-
-      # Setup Rust with beta toolchain and default profile (to include rust-docs)
-      - name: Setup Rust
-        run: |
-          curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain=beta --profile=default
-
-      - uses: Swatinem/rust-cache@v2.7.1
-
-      - name: Build external docs
-        run: |
-          # Exclude zebra-utils and zebra-test, they are not for library or app users
-          cargo doc --no-deps --workspace --all-features --exclude zebra-utils --exclude zebra-test --target-dir "$(pwd)"/target/external
-
-      # Setup gcloud CLI
-      - name: Authenticate to Google Cloud
-        id: auth
-        uses: google-github-actions/auth@v1.1.1
-        with:
-          retries: '3'
-          workload_identity_provider: '${{ vars.GCP_WIF }}'
-          service_account: '${{ vars.GCP_FIREBASE_SA }}'
-
-      # TODO: remove this step after issue https://github.com/FirebaseExtended/action-hosting-deploy/issues/174 is fixed
-      - name: Add $GCP_FIREBASE_SA_PATH to env
-        run: |
-          # shellcheck disable=SC2002
-          echo "GCP_FIREBASE_SA_PATH=$(cat ${{ steps.auth.outputs.credentials_file_path }} | tr -d '\n')" >> "$GITHUB_ENV"
-
-      - name: Deploy external docs to firebase
-        uses: FirebaseExtended/action-hosting-deploy@v0.7.1
-        with:
-          firebaseServiceAccount: ${{ env.GCP_FIREBASE_SA_PATH }}
-          channelId: ${{ env.FIREBASE_CHANNEL }}
-          target: docs-external
-          projectId: ${{ vars.GCP_FIREBASE_PROJECT }}
-
   build-docs-internal:
     name: Build and Deploy Zebra Internal Docs
     timeout-minutes: 45

diff --git a/.github/workflows/release-binaries.yml b/.github/workflows/release-binaries.yml
@@ -18,29 +18,34 @@ jobs:
   # Each time this workflow is executed, a build will be triggered to create a new image
   # with the corresponding tags using information from git
 
-  # The image will be named `zebra:<semver>`
-  build:
-    name: Build Release Docker
+  # The image will be named `zebra:<semver>.experimental`
+  build-experimental:
+    name: Build Experimental Features Release Docker
     uses: ./.github/workflows/sub-build-docker-image.yml
     with:
       dockerfile_path: ./docker/Dockerfile
       dockerfile_target: runtime
       image_name: zebra
-      features: ${{ vars.RUST_PROD_FEATURES }}
+      tag_suffix: .experimental
+      features: ${{ format('{0} {1}', vars.RUST_PROD_FEATURES, vars.RUST_EXPERIMENTAL_FEATURES) }}
       rust_log: ${{ vars.RUST_LOG }}
     # This step needs access to Docker Hub secrets to run successfully
     secrets: inherit
 
-  # The image will be named `zebra:<semver>.experimental`
-  build-experimental:
-    name: Build Experimental Features Release Docker
+  # The image will be named `zebra:<semver>`
+  # It should be built last, so tags with the same name point to the production build, not the experimental build.
+  build:
+    name: Build Release Docker
+    # Run this build last, regardless of whether experimental worked
+    needs: build-experimental
+    if: always()
     uses: ./.github/workflows/sub-build-docker-image.yml
     with:
       dockerfile_path: ./docker/Dockerfile
       dockerfile_target: runtime
       image_name: zebra
-      tag_suffix: .experimental
-      features: ${{ format('{0} {1}', vars.RUST_PROD_FEATURES, vars.RUST_EXPERIMENTAL_FEATURES) }}
+      latest_tag: true
+      features: ${{ vars.RUST_PROD_FEATURES }}
       rust_log: ${{ vars.RUST_LOG }}
     # This step needs access to Docker Hub secrets to run successfully
     secrets: inherit

diff --git a/.github/workflows/sub-build-docker-image.yml b/.github/workflows/sub-build-docker-image.yml
@@ -33,6 +33,10 @@ on:
       test_features:
         required: false
         type: string
+      latest_tag:
+        required: false
+        type: boolean
+        default: false
       tag_suffix:
         required: false
         type: string
@@ -88,19 +92,34 @@ jobs:
           # appends inputs.tag_suffix to image tags/names
           flavor: |
             suffix=${{ inputs.tag_suffix }}
+            latest=${{ inputs.latest_tag }}
           # generate Docker tags based on the following events/attributes
           tags: |
-            type=schedule
-            # semver and ref,tag automatically add a "latest" tag, but only on stable releases
+            # These DockerHub release tags support the following use cases:
+            # - `latest`: always use the latest Zebra release when you pull or update
+            # - `1`: use the latest Zebra release, but require manual intervention for the next network upgrade
+            # - `1.x`: update to bug fix releases, but don't add any new features or incompatibilities
+            # - `v1.x.y` or `1.x.y`: always use the exact version, don't automatically upgrade
+            # - `sha-zzzzzz`: always use the exact commit (the same as `1.x.y`, but also used in CI and production image tests)
+            #
+            # Stopping publishing some tags is a silently breaking change:
+            # - `1`: doesn't get expected new consensus-compatible releases or bug fixes
+            # - `1.x`: doesn't get expected bug fixes
+            #
+            # `semver` adds a "latest" tag if `inputs.latest_tag` is `true`.
             type=semver,pattern={{version}}
             type=semver,pattern={{major}}.{{minor}}
             type=semver,pattern={{major}}
             type=ref,event=tag
-            type=ref,event=branch
-            type=ref,event=pr
+            # DockerHub release and CI tags.
+            # This tag makes sure tests are using exactly the right image, even when multiple PRs run at the same time.
             type=sha
-            # edge is the latest commit on the default branch.
+            # These CI-only tags support CI on PRs, the main branch, and scheduled full syncs. 
+            # These tags do not appear on DockerHub, because DockerHub images are only published on the release event.
+            type=ref,event=pr
+            type=ref,event=branch
             type=edge,enable={{is_default_branch}}
+            type=schedule
 
       # Setup Docker Buildx to allow use of docker cache layers from GH
       - name: Set up Docker Buildx