Release 2504 distribution

[jpraynaud]

Why

In order to deploy new features, improvements and bug fixes, we need to release a new distribution.

What

Create and release a new distribution 2504

Important

• 🔥 Blocked by Missing debug and info logs in Mithril nodes #2227
• 🔥 Blocked by Signer does not handle properly signature signed entity timeout #2229
• 🔥 Blocked by Grafana aggregator dashboard is not working on release networks #2230
• Upgrade release-preprod VM from e2-highmem-2 to e2-highmem-4
• Upgrade to Cardano node 10.1.4
  • in pre-release-preview
  • in release-preprod
  • in release-mainnet
• Activate the HTTP traffic compression in:
  • in pre-release-preview
  • in release-preprod
  • in release-mainnet
• Import the immutable file digests in the aggregator database
  • in pre-release-preview
  • in release-preprod
  • in release-mainnet

Distribution +1:

Note

• TBD

How

Phase 1: Prepare distribution

• Run a backward compatibility test in the CI
• Upgrade repository dependencies (with runbook)
• Check that all minor versions of all crates have been bumped for breaking change (if needed)
• Check if versions of repository linked crates are correct for publication to crates.io(mithril-common, mithril-client)
• Check if Cardano minimum supported versions in networks.json must be updated following the runbook
• Merge a PR for rotating the CHANGELOG.md (set the distribution version)
• Create a draft PR for rotating the documentation

Phase 2: Rollout pre-release distribution

• Activate the HTTP traffic compression in:
  • in pre-release-preview
• Tag and deploy a 2504.0-pre pre-release distribution on the pre-release-preview network
• Fill highlights, distributions compatibility and release notes sections in the pre-release page on GitHub
• Import the immutable file digests in the aggregator database in pre-release-preview with process
• Make sure pre-release-preview is working as expected
• Make sure libraries version next have been published on npm registry
• Run an all green client multi-platform test in the CI
• Run an all green aggregator stress test in the CI
• Run a Cardano transaction prover benchmark:
  • Store results in the issue
  • Make sure that there is no performance drop since last release
• Announce the pre-release on discord channel
• Announce any breaking change in:
  • #cardano-scaling slack channel for pre-release
  • #mithril-lace slack channel for pre-release
  • #blockfrost-scaling slack channel for pre-release
  • Cardano Foundation IBC bridge team
• Wait for at least 3 signers running the new version (could be more if era change is tested) over at least 2 epochs

Phase 3: Rollout release distribution

• Activate the HTTP traffic compression in:
  • in release-preprod
  • in release-mainnet
• Tag and deploy a 2504.0 release distribution on the pre-release-preview network
• Fill highlights, distributions compatibility and release notes sections in the release page on GitHub
• Make sure pre-release-preview is working as expected
• Update networks compatibility table in release notes
• Stop all the cardano nodes of release-preprod as the VM will be updated with process
• Release the 2504.0
• Import the immutable file digests in the aggregator database in release-mainnet with process
• Make sure release-mainnet is working as expected
• Make sure that the VM of release-preprod has been updated properly with process
• Import the immutable file digests in the aggregator database in release-preprod with process
• Make sure release-preprod is working as expected
• Make sure libraries have been published on crates.io
• Make sure libraries version latest have been published on npm registry
• Merge the PR for rotating the documentation
• Merge a PR for rotating the CHANGELOG.md (updating the release date of the distribution)
• Announce the release on discord channel (with breaking-change, upcoming features activation, ...)
• Announce any breaking/new feature change in:
  • #cardano-scaling slack channel for release
  • #mithril-lace slack channel for release
  • #blockfrost-scaling slack channel for release
  • Cardano Foundation IBC bridge team
• Update Hydra tutorial if client breaking change for database restoration
• Dev blog post:
  • Distribution 2504

[jpraynaud]

Upgrade VM in place process

Prepare the upgrade

Make a manual snapshot of the data disk in GCP:

export SNAPSHOT_ID=mithril-release-preprod-...
export SOURCE_DISK=mithril-release-preprod-data
gcloud compute snapshots create $SNAPSHOT_ID --source-disk $SOURCE_DISK --snapshot-type STANDARD --source-disk-zone europe-west1-b

Then connect to the VM:

export VM=aggregator.release-preprod.api.mithril.network
ssh curry@$VM

Once connected, create a temp directory:

export TMP_DIR=/home/curry/tmp
mkdir -p $TMP_DIR

Make a clean stop of the Cardano nodes:

docker ps --format '{{.Names}}' | grep cardano > $TMP_DIR/docker-ps-cardano.txt
cat $TMP_DIR/docker-ps-cardano.txt | xargs -I '{}' docker stop '{}'

Run the terraform deployment to upgrade the VM

Run the deployment with the correct GitHub actions workflow that will upgrade the VM in place.

Check that the data disk is properly mounted

Once the VM has been modified in place, check that all the data disk is properly mounted:

lsblk | grep sdb | grep "/home/curry/data"

Case 1: the disk is properly mounted

If the previous command returns an entry, the data disk is mounted to the data folder and there is nothing more to do.

Case 2: the disk is not properly mounted

Otherwise, the data disk must be mounted.

First, stop all the remaining containers:

docker stop reverse-proxy
docker ps --format '{{.Names}}' > $TMP_DIR/docker-ps-all.txt
cat $TMP_DIR/docker-ps-all.txt | xargs -I '{}' docker stop '{}'

Then mount the data disk to the data folder:

sudo mount -o discard,defaults /dev/sdb /home/curry/data/

And make sure that the disk is now correctly mounted:

lsblk | grep sdb | grep "/home/curry/data"

Then restart all the containers:

cat $TMP_DIR/docker-ps-cardano.txt | xargs -I '{}' docker start '{}'
docker start reverse-proxy

Cleanup

Delete the temporary folder:

rm -rf $TMP_DIR