[ZK Stack] - How to migrate self deployed ZkSync hyperchain to new machine?

[sachanayush47]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.80.1

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

We are running zksync hyperchain on limit system specs and we know in some time we need to move to a new bigger machine. We are right now exploring ways to migrate whole ecosystem + chain with Postgres DB to a new machine. So when after migration is completed, we can run the server from last state.

Expected Behavior

NA

Current Behavior

NA

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[gleenai[bot]]

Migrating your self-deployed ZKsync hyperchain to a new machine involves several steps, primarily focused on transferring your current system's data and configurations to the new environment. Here’s a general approach to achieve this:

1. Backup Current Data: Ensure you have comprehensive backups of your current system. This includes:

   • Database dumps of your Postgres DB.
   • Configuration files and any custom scripts or software related to your ZKsync hyperchain.

2. Prepare the New Machine:

   • Install the necessary software, including the same version of Rust (1.80.1) and other dependencies that match your current setup.
   • Set up Postgres and ensure it is configured similarly to your existing setup.

3. Transfer Data:

   • Transfer the database dumps and configuration files from the old machine to the new one.
   • Restore the database from the dumps to the new Postgres installation.

4. Reconfigure and Test:

   • Update any configuration files with new machine-specific settings (e.g., IP addresses, paths).
   • Test the system thoroughly to ensure everything is functioning as expected before going live.

5. Switch Over:

   • Once testing is complete and you’re confident in the new setup, you can switch your operations to the new machine.

For detailed steps tailored to your specific setup, you might want to consult with a system administrator or a professional with experience in migrating complex systems. Additionally, consider visiting the ZKsync developers discussions on Github for more in-depth guidance or community support.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[bxpana]

@sachanayush47 I'm checking up on this for you and will let you know when I hear back from the team.

[popzxc]

Hey! We don't really have a step-by-step guide for that and it depends on the actual setup a lot, so it's hard to provide a precise answer.
It's also not clear if some downtime is acceptable for you. I'll assume yes; if not -- the situation may become significantly more complex.
However, here are some general advices:

1. Is postgres running on the same machine as the sequencer? If not, the first step would be to migrate Postgres to a separate machine. Probably, the easiest way to do it would be to prepare a clean Postgres deployment on the machine of your choice (or actually; if you're running in cloud, you might want to have managed Postgres, e.g. CloudSQL in GCP) shutdown the server, take the dump of the database, and apply it on the newly deployed instance. Then you should change configuration for the server to talk with the new Postgres, and you can run it.
2. Are you running all the components together (e.g. in a monolithic mode) or are you using different machines for different components (e.g. api/core/tree)? I assume that you're running it in a monolithic mode. In general, you might be interested in researching distributed mode, as it allows, for example, horizontal scaling for the API servers (http/ws). For that, take a look at the --components flag for the server node.

Regardless of the mode, the only stateful part of the server (besides Postgres) is RocksDB data (Merkle tree and state keeper cache). You need to shutdown the server, move this data to a new machine, update the configuration if any paths have changed, and then run the sequencer on the new machine.

Finally, I would really recommend doing that on some test network first to get experience. The process isn't complex, and any problem except Postgres data loss should be recoverable (e.g. even Merkle tree can be rebuilt from scratch), but this way you can make it easier for yourself and avoid prolonged downtime and frustration.

[sachanayush47]

Thank you so much for the detailed guide.

and yeah, currently everything is running in the same machine including postgres. Downtime is acceptable.

[sachanayush47]

There is a snapshot command available with zkstack cli https://github.com/matter-labs/zksync-era/tree/main/zkstack_cli#snapshot-commands. What's the purpose of it?

[popzxc]

It's for managing network snapshots, which is used by the external nodes. Documentation for it can be found here.

Snapshots are not suitable for restoring the sequencer node, however.