Second node created a fork and stuck

[askucher]

Bug Description
We have launched version 1.9.0 of Optimism along with beta.2 of our smart contracts using the useCustomToken feature. Everything is functioning correctly on our servers, and blocks are being created as expected.

To enhance fault tolerance, we decided to deploy additional nodes, using the example from this guide.

However, after launching a second node (op-geth + op-node), it started syncing but eventually created a fork and got stuck. This was unexpected because the setup should have maintained synchronization across nodes without forking

Thats why we have some questions

1. How the optimism is protected from L1 forks as far we use some L1 rpc which can be unstable
2. Can we make so optimism can roll back some block and try to resolve itself automatically
3. How is fork possible in case when we have 1 block proposer?

Steps to Reproduce

1. There is not steps to reproduce but we have some questions

Expected behavior
It's should working perfectly without forks and we need to have possibility to switch from one node to another is something happened with a disk

Environment Information:

• Ubuntu 20.04
• optimism 1.9.0 + beta.2 smart contracts

Configurations:
We do not use the super chain registry all vars are following the documentation.

Logs:
7|op-node | t=2024-09-02T15:11:11+0000 lvl=info msg="Scheduled sequencer action" delta=1.999644808s
7|op-node | t=2024-09-02T15:11:12+0000 lvl=warn msg="unexpected block author" topic=blocksV3 err= peer=16Uiu2HAmLPbSkxyYGYiqaBHyCP1SdFwpVcLhqtTDnj64gEzq1Kmn addr=0x99d508fAd3d9aeE9eA4f206191c6dB915221cE12 expected=0x7b04384c0a67E27E95c508c8B8D79afA9080f5f5

Info about first block where is different state

Detailed INFO OF BLOCKS 217 - REMOTE


baseFeePerGas        422404275
difficulty           0
extraData            0x
gasLimit             30000000
gasUsed              43890
hash                 0xb54822428dbd243d1d61c4460c84a18b990178f6834e4092ba0b2df4e1831361
logsBloom            0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
miner                0x4200000000000000000000000000000000000011
mixHash              0xb909be732042d9972431e777d8e7c0468f2654961cb88a6330b993c5d73b03ae
nonce                0x0000000000000000
number               217
parentHash           0x9725e439d7eec1b659ffc22548b81597a33c3344f1c133f1dfb50bd92aa72bf3
transactionsRoot     0x81863caf566c2af30e72fe923297fb7a9cc7eb29136ab8d70e84c5f415d9616b
receiptsRoot         0x018603f8ce61ad4aa03e27b84fe3f996320b7a275d3cb996add4e628c7602a8f
sha3Uncles           0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347
size                 840
stateRoot            0x64ff565bfa20da241af389acd56c0c741077419354e89a5d9263616b34138d1f
timestamp            1724783006 (Tue, 27 Aug 2024 18:23:26 +0000)
withdrawalsRoot      0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421
totalDifficulty      0
transactions:        [
	0xb8c012f2202e66e31d82367446fb89435d543037c8ee92e2dca9dd2f03800c9b
]
Detailed INFO OF BLOCKS 217 - LOCAL


baseFeePerGas        422404275
difficulty           0
extraData            0x
gasLimit             30000000
gasUsed              55066
hash                 0x07552e9c0789c60951d3a229bbdad911de69a5912cac713939d8b2d860b6f244
logsBloom            0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
miner                0x4200000000000000000000000000000000000011
mixHash              0xd8404809117914e393a11a167c873e2e1145931cd311e08d3255e06704f82804
nonce                0x0000000000000000
number               217
parentHash           0x9725e439d7eec1b659ffc22548b81597a33c3344f1c133f1dfb50bd92aa72bf3
transactionsRoot     0xbc4b028985f76b1f5941c90dc7e9b6785d82b7a21d5c24681659a38e56d54d3b
receiptsRoot         0x8435f38e8666eb2f15cd135f9f46fedeed9265ae5b399d1f80c7b19c69d12909
sha3Uncles           0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347
size                 840
stateRoot            0x6757ae4ec81e7adcd5f8cfafea686aa0da3fdcb8a3be0d7c9fbff0bc4c792d6d
timestamp            1724783006 (Tue, 27 Aug 2024 18:23:26 +0000)
withdrawalsRoot      0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421
totalDifficulty      0
transactions:        [
	0xb7c4248afcf325c1339b370cafe2957d42a976814db17525c6984cb533d8ae40
]
Detailed INFO OF TX 0xb8c012f2202e66e31d82367446fb89435d543037c8ee92e2dca9dd2f03800c9b - REMOTE

blockHash            0xb54822428dbd243d1d61c4460c84a18b990178f6834e4092ba0b2df4e1831361
blockNumber          217
from                 0xDeaDDEaDDeAdDeAdDEAdDEaddeAddEAdDEAd0001
gas                  1000000
gasPrice             0
hash                 0xb8c012f2202e66e31d82367446fb89435d543037c8ee92e2dca9dd2f03800c9b
input                0x440a5e20000f42400000000000000000000000040000000066ce1954000000000064720e000000000000000000000000000000000000000000000000000000040cf4e7e40000000000000000000000000000000000000000000000000000006304a6b4ac86ae18a7696c27438365d637551fffd1c5ea6319e2be61ffd7b76d655699a184000000000000000000000000f36af909d472d966c49eb11217efc90fd070abd4
nonce                216
r                    0x0000000000000000000000000000000000000000000000000000000000000000
s                    0x0000000000000000000000000000000000000000000000000000000000000000
to                   0x4200000000000000000000000000000000000015
transactionIndex     0
v                    0
value                0
depositReceiptVersion 1
mint                 0
sourceHash           39140696042246058621864761218877383772630202131977715764031800604037221138676

Detailed INFO OF TX 0xb7c4248afcf325c1339b370cafe2957d42a976814db17525c6984cb533d8ae40 - REMOTE

blockHash            0x3a1ab5ca2462ce034636a95f729af89267c7419fc8b4c4a6566d1ac6320ae1bd
blockNumber          218
from                 0xDeaDDEaDDeAdDeAdDEAdDEaddeAddEAdDEAd0001
gas                  1000000
gasPrice             0
hash                 0xb7c4248afcf325c1339b370cafe2957d42a976814db17525c6984cb533d8ae40
input                0x440a5e20000f42400000000000000000000000000000000066ce1960000000000064720f00000000000000000000000000000000000000000000000000000004033f17100000000000000000000000000000000000000000000000000000006f653ba46682400829565b5287dfd4004df64e029b876b7532e1ca98947114e0c347b10367000000000000000000000000f36af909d472d966c49eb11217efc90fd070abd4
nonce                217
r                    0x0000000000000000000000000000000000000000000000000000000000000000
s                    0x0000000000000000000000000000000000000000000000000000000000000000
to                   0x4200000000000000000000000000000000000015
transactionIndex     0
v                    0
value                0
depositReceiptVersion 1
mint                 0
sourceHash           17760142539965865497085337990672279271964787131804171250706484526030666864411

Detailed INFO OF TX 0xb7c4248afcf325c1339b370cafe2957d42a976814db17525c6984cb533d8ae40 - LOCAL

blockHash            0x07552e9c0789c60951d3a229bbdad911de69a5912cac713939d8b2d860b6f244
blockNumber          217
from                 0xDeaDDEaDDeAdDeAdDEAdDEaddeAddEAdDEAd0001
gas                  1000000
gasPrice             0
hash                 0xb7c4248afcf325c1339b370cafe2957d42a976814db17525c6984cb533d8ae40
input                0x440a5e20000f42400000000000000000000000000000000066ce1960000000000064720f00000000000000000000000000000000000000000000000000000004033f17100000000000000000000000000000000000000000000000000000006f653ba46682400829565b5287dfd4004df64e029b876b7532e1ca98947114e0c347b10367000000000000000000000000f36af909d472d966c49eb11217efc90fd070abd4
nonce                216
r                    0x0000000000000000000000000000000000000000000000000000000000000000
s                    0x0000000000000000000000000000000000000000000000000000000000000000
to                   0x4200000000000000000000000000000000000015
transactionIndex     0
v                    0
value                0
depositReceiptVersion 1
mint                 0
sourceHash           17760142539965865497085337990672279271964787131804171250706484526030666864411

Detailed INFO OF TX 0xb8c012f2202e66e31d82367446fb89435d543037c8ee92e2dca9dd2f03800c9b - LOCAL
Error: 
tx not found: 0xb8c012f2202e66e31d82367446fb89435d543037c8ee92e2dca9dd2f03800c9b

Additional context
We can provide more information per request

---

⚠️ Notice: Issues that do not include the following sections will be subject to closure:

• Bug Description
• Steps to Reproduce
• Environment Information

Please ensure all required sections are filled out accurately to expedite the debugging process and improve issue resolution efficiency.

[ZakAyesh]

Hi, to answer your 3 questions we have to lay out some fundamentals on how the chain operates first:

Rollups derive their security from Ethereum. What that means is that the L2 blockchain can be derived from data posted on Ethereum. So this creates this L2 concept of unsafe, safe, and finalized blocks/transactions. The following numbers are based on OP Mainnet.

• unsafe (2-4s) - transaction in block created by the L2 sequencer, but the tx data has not been posted to the L1 yet. Its gets transmitted over the p2p network for op-nodes connected to the sequencer (and the nodes connected to those nodes).
• safe (5-10m) - transaction data has been posted to the L1 (by the sequencer's batcher). This transaction can be derived from any L2 node.
• finalized (15-20m) - same as safe, but enough L1 blocks have passed so the tx data on L1 will not be reorged.
• However the safe and finalized depend on how often the batcher posts to the L1. A common chain option is, 1_800 base layer blocks (6 hours for an L2 on Ethereum, assuming 12 second L1 blocktime) or lower. A chain operator would increase this interval so they won't have to post as often to L1; which saves them gas fees. So technically you could post batch data more frequently than OP Mainnet, so L2 txs end up on L1 and finalize faster.

So to answer your questions:

1. There are several things you can do to help prevent a replica node from forking.

• One is to make sure your connection to the L1 RPC is as strong as possible for deriving any L1 information, this will be important for the safe and finalized heads.
• Another is to follow the steps in the docs here to make sure your p2p network is set up robustly to update the unsafe head.
• In op-node you can set the verifier.l1-confs to a higher number to reduce the chance of the node bringing in a block that is reorged out. See docs here.

2. The finalized canonical chain is always on L1 as the chain of finalized heads. Therefore the best practice would be to have multiple replicas and if one forks begin resyncing that node either from scratch from the L1 via execution layer sync, from any snapshots you have created from before the fork, or via snap-sync in op-geth.

3. There could be several reasons, was the block that forked an unsafe, safe, or finalized block?

[askucher]

My replica node is syncing blocks from the very beginning, rather than using snap mode. The height of the main node is significantly ahead of the point where the unsafe block is created, but the replica node ends up creating a fork.
So, I think verifier.l1-confs is not a problem here

I intentionally disconnected from the super chain and manually copied genesis.json and rollup.json. I also ran the following command:

bash

./build/bin/geth init --datadir=datadir genesis.json

Could this be causing an issue, as the testnet may not be listed in the super chain registry after this action?

Also, it is really helpful if you tell how exactly op-node verifies the canonical order of transactions. Better if you point lines of codes. I would find the solution by debugging it. Logs are not so informative

Additional information

All rollups works properly

https://sepolia.etherscan.io/address/0xa8bc7df3b574352f09e99cca943f335f3d0c90ba
https://sepolia.etherscan.io/address/0xF36aF909D472d966c49Eb11217EFC90fd070ABd4

Bridge works properly on main node.

My replica launch scripts

  cat > op-geth.sh << EOM
  #!/bin/bash
  ./build/bin/geth \
      --datadir ./datadir \
      --bootnodes "$BOOTNODES" \
      --syncmode full \
      --http \
      --http.corsdomain="*" \
      --http.vhosts="*" \
      --http.addr=0.0.0.0 \
      --http.api=web3,debug,eth,txpool,net,engine \
      --ws \
      --ws.addr=0.0.0.0 \
      --ws.port=$OP_GETH_WS_PORT \
      --cache 4096 \
      --ws.api=debug,eth,txpool,net,engine \
      --rollup.sequencerhttp="$OP_GETH_SEQUENCER_HTTP" \
      --rollup.halt=major \
      --gcmode=archive \
      --nodiscover \
      --maxpeers=1000 \
      --networkid=$L2_CHAIN_ID \
      --authrpc.vhosts="*" \
      --authrpc.addr=0.0.0.0 \
      --authrpc.jwtsecret=./jwt.txt \
      --authrpc.port=$OP_GETH_AUTHRPC_PORT \
      --rollup.disabletxpoolgossip=true \
      --verbosity=4 \

EOM

op-node

cat > op-node.sh << EOM
  #!/bin/bash
  ./bin/op-node \
    --l2="http://localhost:$OP_GETH_AUTHRPC_PORT" \
    --l2.jwt-secret="./jwt.txt" \
    --sequencer.enabled \
    --sequencer.l1-confs="5" \
    --verifier.l1-confs="4" \
    --rollup.config="./rollup.json" \
    --rpc.addr="0.0.0.0" \
    --p2p.disable=false \
    --p2p.listen.ip="0.0.0.0" \
    --p2p.listen.tcp=$OP_NODE_P2P_PORT \
    --l1="http://localhost:$HAPROXY_PORT" \
    --l1.rpckind="$L1_RPC_KIND" \
    --l1.beacon="https://ethereum-sepolia-beacon-api.publicnode.com"
EOM