Support upgradability of the consensus parameters and state transitio…

…n bytecode in genesis (#1826)

The changes make the state transition bytecode part of the
`ChainConfig`. It guarantees the state transition's availability for the
network's first blocks.

The change has many minor improvements in different areas related to the
state transition bytecode:
- The state transition bytecode lies in its own
file(`state_transition_bytecode.wasm`) along with the chain config file.
The `ChainConfig` loads it automatically when `ChainConfig::load` is
called and pushes it back when `ChainConfig::write` is called.
- The `fuel-core` release bundle also contains the
`fuel-core-wasm-executor.wasm` file of the corresponding executor
version.
- The regenesis process now considers the last block produced by the
previous network. When we create a (re)genesis block of a new network,
it has the `height = last_block_of_old_netowkr + 1`. It continues the
old network and doesn't overlap blocks(before, we had `old_block.height
== new_genesis_block.hegiht`).
- Along with the new block height, the regenesis process also increases
the state transition bytecode and consensus parameters versions. It
guarantees that a new network doesn't use values from the previous
network and allows us not to migrate `StateTransitionBytecodeVersions`
and `ConsensusParametersVersions` tables.
- Added a new CLI argument, `native-executor-version,` that allows
overriding of the default version of the native executor. It can be
useful for side rollups that have their own history of executor
upgrades.
- Replaced:
  
  ```rust
           let file = std::fs::File::open(path)?;
           let mut snapshot: Self = serde_json::from_reader(&file)?;
  ```
  
  with a:
  
  ```rust
           let mut json = String::new();
           std::fs::File::open(&path)
.with_context(|| format!("Could not open snapshot file: {path:?}"))?
               .read_to_string(&mut json)?;
let mut snapshot: Self = serde_json::from_str(json.as_str())?;
  ```
  because it is faster in 100 times for big JSON files.
- Updated all tests to use `Config::local_node_*` instead of working
with the `SnapshotReader` directly. It is the preparation of the tests
for the futures bumps of the `Executor::VERSION`. When we increase the
version, all tests continue to use
`GenesisBlock.state_transition_btyecode = 0` while the version is
different, which forces the usage of the WASM executor, while for tests,
we still prefer to test native execution. The `Config::local_node_*`
handles it and forces the executor to use the native version.
- Reworked the `build.rs` file of the upgradable executor. The script
now has a cache used between builds to avoid recompilation of the WASM
bytecode. Also, fixed the issue with outdated WASM bytecode. The script
reacts on any modifications of the `fuel-core-wasm-executor` and forces
recompilation(it is why we need cache), so WASM bytecode always is
actual now.

## Checklist
- [x] Breaking changes are clearly marked as such in the PR description
and changelog

### Before requesting review
- [x] I have reviewed the code myself

---------

Co-authored-by: Hannes Karppila <hannes.karppila@gmail.com>

.github/workflows/ci.yml

@@ -555,6 +555,7 @@ jobs:
           mkdir -pv "$ARTIFACT"
           cp "target/${{ matrix.job.target }}/release/fuel-core" "$ARTIFACT"
           cp "target/${{ matrix.job.target }}/release/fuel-core-keygen" "$ARTIFACT"
+          cp "target/${{ matrix.job.target }}/release/fuel-core-upgradable-executor-cache/wasm32-unknown-unknown/release/fuel-core-wasm-executor.wasm" "$ARTIFACT"
           tar -czvf "$ZIP_FILE_NAME" "$ARTIFACT"
 
       - name: Upload Binary Artifact

CHANGELOG.md

@@ -30,6 +30,32 @@ Description of the upcoming release here.
 
 #### Breaking
 
+- [#1826](https://github.com/FuelLabs/fuel-core/pull/1826): The changes make the state transition bytecode part of the `ChainConfig`. It guarantees the state transition's availability for the network's first blocks.
+    The change has many minor improvements in different areas related to the state transition bytecode:
+    - The state transition bytecode lies in its own file(`state_transition_bytecode.wasm`) along with the chain config file. The `ChainConfig` loads it automatically when `ChainConfig::load` is called and pushes it back when `ChainConfig::write` is called.
+    - The `fuel-core` release bundle also contains the `fuel-core-wasm-executor.wasm` file of the corresponding executor version.
+    - The regenesis process now considers the last block produced by the previous network. When we create a (re)genesis block of a new network, it has the `height = last_block_of_old_netowkr + 1`. It continues the old network and doesn't overlap blocks(before, we had `old_block.height == new_genesis_block.hegiht`).
+    - Along with the new block height, the regenesis process also increases the state transition bytecode and consensus parameters versions. It guarantees that a new network doesn't use values from the previous network and allows us not to migrate `StateTransitionBytecodeVersions` and `ConsensusParametersVersions` tables.
+    - Added a new CLI argument, `native-executor-version,` that allows overriding of the default version of the native executor. It can be useful for side rollups that have their own history of executor upgrades.
+    - Replaced:
+
+      ```rust
+               let file = std::fs::File::open(path)?;
+               let mut snapshot: Self = serde_json::from_reader(&file)?;
+      ```
+
+      with a:
+
+      ```rust
+               let mut json = String::new();
+               std::fs::File::open(&path)
+                   .with_context(|| format!("Could not open snapshot file: {path:?}"))?
+                   .read_to_string(&mut json)?;
+               let mut snapshot: Self = serde_json::from_str(json.as_str())?;
+      ```
+      because it is 100 times faster for big JSON files.
+    - Updated all tests to use `Config::local_node_*` instead of working with the `SnapshotReader` directly. It is the preparation of the tests for the futures bumps of the `Executor::VERSION`. When we increase the version, all tests continue to use `GenesisBlock.state_transition_bytecode = 0` while the version is different, which forces the usage of the WASM executor, while for tests, we still prefer to test native execution. The `Config::local_node_*` handles it and forces the executor to use the native version.
+    - Reworked the `build.rs` file of the upgradable executor. The script now caches WASM bytecode to avoid recompilation. Also, fixed the issue with outdated WASM bytecode. The script reacts on any modifications of the `fuel-core-wasm-executor` and forces recompilation (it is why we need the cache), so WASM bytecode always is actual now.
 - [#1822](https://github.com/FuelLabs/fuel-core/pull/1822): Removed support of `Create` transaction from debugger since it doesn't have any script to execute.
 - [#1822](https://github.com/FuelLabs/fuel-core/pull/1822): Use `fuel-vm 0.49.0` with new transactions types - `Upgrade` and `Upload`. Also added `max_bytecode_subsections` field to the `ConsensusParameters` to limit the number of bytecode subsections in the state transition bytecode. 
 - [#1816](https://github.com/FuelLabs/fuel-core/pull/1816): Updated the upgradable executor to fetch the state transition bytecode from the database when the version doesn't match a native one. This change enables the WASM executor in the "production" build and requires a `wasm32-unknown-unknown` target.

Cargo.toml

@@ -77,7 +77,7 @@ fuel-core-trace = { version = "0.24.2", path = "./crates/trace" }
 fuel-core-types = { version = "0.24.2", path = "./crates/types", default-features = false }
 fuel-core-tests = { version = "0.0.0", path = "./tests" }
 fuel-core-upgradable-executor = { version = "0.24.2", path = "./crates/services/upgradable-executor" }
-fuel-core-wasm-executor = { version = "0.24.2", path = "./crates/services/upgradable-executor/wasm-executor" }
+fuel-core-wasm-executor = { version = "0.24.2", path = "./crates/services/upgradable-executor/wasm-executor", default-features = false }
 fuel-core-xtask = { version = "0.0.0", path = "./xtask" }
 
 # Fuel dependencies
@@ -88,6 +88,7 @@ anyhow = "1.0"
 async-trait = "0.1"
 cynic = { version = "2.2.1", features = ["http-reqwest"] }
 clap = "4.4"
+derivative = { version = "2" }
 derive_more = { version = "0.99" }
 enum-iterator = "1.2"
 hyper = { version = "0.14.26" }

benches/benches/block_target_gas.rs

@@ -35,7 +35,6 @@ use fuel_core_benches::{
 use fuel_core_chain_config::{
     ChainConfig,
     ContractConfig,
-    SnapshotReader,
     StateConfig,
 };
 use fuel_core_services::Service;
@@ -296,10 +295,7 @@ fn service_with_many_contracts(
         contracts: contract_configs,
         ..Default::default()
     };
-    let mut config = Config {
-        snapshot_reader: SnapshotReader::new_in_memory(chain_config, state_config),
-        ..Config::local_node()
-    };
+    let mut config = Config::local_node_with_configs(chain_config, state_config);
     config.utxo_validation = false;
     config.block_production = Trigger::Instant;
 

bin/e2e-test-client/src/tests/test_data/large_state/state_config.json

@@ -118151,6 +118151,5 @@
       ]
     }
   ],
-  "block_height": 0,
-  "da_block_height": 0
+  "last_block": null
 }

bin/e2e-test-client/tests/integration_tests.rs

@@ -95,8 +95,6 @@ async fn execute_suite(config_path: String) {
 }
 
 fn dev_config() -> Config {
-    let mut config = Config::local_node();
-
     let snapshot = SnapshotMetadata::read("../../bin/fuel-core/chainspec/dev-testnet")
         .expect("Should be able to open snapshot metadata");
     let reader =
@@ -112,9 +110,8 @@ fn dev_config() -> Config {
             >= 1 << 17 // 131072
     );
 
+    let mut config = Config::local_node_with_reader(reader);
     config.static_gas_price = 1;
-    config.snapshot_reader = reader;
-
     config.block_producer.coinbase_recipient = Some(
         ContractId::from_str(
             "0x7777777777777777777777777777777777777777777777777777777777777777",

bin/fuel-core/chainspec/dev-testnet/state_config.json

@@ -131,6 +131,5 @@
       "balances": []
     }
   ],
-  "block_height": 0,
-  "da_block_height": 0
+  "last_block": null
 }

bin/fuel-core/chainspec/testnet/state_config.json

@@ -13,6 +13,5 @@
       "balances": []
     }
   ],
-  "block_height": 0,
-  "da_block_height": 0
+  "last_block": null
 }

bin/fuel-core/src/cli/run.rs

@@ -42,6 +42,7 @@ use fuel_core_chain_config::{
     SnapshotMetadata,
     SnapshotReader,
 };
+use fuel_core_types::blockchain::header::StateTransitionBytecodeVersion;
 use pyroscope::{
     pyroscope::PyroscopeAgentRunning,
     PyroscopeAgent,
@@ -139,6 +140,10 @@ pub struct Command {
     #[arg(long = "utxo-validation", env)]
     pub utxo_validation: bool,
 
+    /// Overrides the version of the native executor.
+    #[arg(long = "native-executor-version", env)]
+    pub native_executor_version: Option<StateTransitionBytecodeVersion>,
+
     /// The minimum allowed gas price
     #[arg(long = "min-gas-price", default_value = "0", env)]
     pub min_gas_price: u64,
@@ -217,6 +222,7 @@ impl Command {
             vm_backtrace,
             debug,
             utxo_validation,
+            native_executor_version,
             min_gas_price,
             consensus_key,
             poa_trigger,
@@ -327,6 +333,7 @@ impl Command {
             combined_db_config,
             snapshot_reader,
             debug,
+            native_executor_version,
             utxo_validation,
             block_production: trigger,
             vm: VMConfig {

bin/fuel-core/src/cli/snapshot.rs

@@ -305,10 +305,9 @@ mod tests {
             builder.add(self.common.contract_state);
             builder.add(self.common.contract_balance);
 
-            let height = self.common.block.value.header().height();
-            let da_height = self.common.block.value.header().application().da_height;
-
-            builder.build(*height, da_height).unwrap()
+            builder
+                .build(Some(self.common.block.value.header().into()))
+                .unwrap()
         }
 
         fn read_from_snapshot(snapshot: SnapshotMetadata) -> Self {
@@ -331,11 +330,17 @@ mod tests {
 
             let block = {
                 let mut block = CompressedBlock::default();
-                let height = reader.block_height();
-                block.header_mut().application_mut().da_height = reader.da_block_height();
-                block.header_mut().set_block_height(height);
+                let last_block_config = reader
+                    .last_block_config()
+                    .cloned()
+                    .expect("Expects the last block config to be set");
+                block.header_mut().application_mut().da_height =
+                    last_block_config.da_block_height;
+                block
+                    .header_mut()
+                    .set_block_height(last_block_config.block_height);
                 TableEntry {
-                    key: height,
+                    key: last_block_config.block_height,
                     value: block,
                 }
             };
@@ -767,8 +772,7 @@ mod tests {
                 coins: vec![],
                 messages: vec![],
                 contracts: vec![randomly_chosen_contract],
-                block_height: original_state.block_height,
-                da_block_height: original_state.da_block_height,
+                last_block: original_state.last_block,
             }
         );
 

ci_checks.sh

@@ -25,7 +25,7 @@ cargo check -p fuel-core-storage --target wasm32-unknown-unknown --no-default-fe
 cargo check -p fuel-core-client --target wasm32-unknown-unknown --no-default-features &&
 cargo check -p fuel-core-chain-config --target wasm32-unknown-unknown --no-default-features &&
 cargo check -p fuel-core-executor --target wasm32-unknown-unknown --no-default-features &&
-cargo test --workspace &&
+OVERRIDE_CHAIN_CONFIGS=true cargo test --workspace &&
 FUEL_ALWAYS_USE_WASM=true cargo test --all-features --workspace &&
 cargo test -p fuel-core --no-default-features &&
 cargo test -p fuel-core-client --no-default-features &&

crates/chain-config/Cargo.toml

@@ -13,6 +13,7 @@ description = "Fuel Chain config types"
 [dependencies]
 anyhow = { workspace = true }
 bech32 = { version = "0.9.0", default-features = false, optional = true }
+derivative = { workspace = true }
 fuel-core-storage = { workspace = true }
 fuel-core-types = { workspace = true, default-features = false, features = [
   "serde",