New synchronisation mechanism

[ppedziwiatr]

Current implementation

Currently, the sequencer generates transactions with a sortKey set for the current network height.
https://github.com/warp-contracts/gateway/blob/main/src/gateway/router/routes/sequencerRoute.ts#L76

The new Syncer waits a specified number of blocks to synchronize L1 transactions (to ensure they are properly confirmed and not from a fork).

Problem

There is a risk with how Sequencer is generating sortKeys. For example:

• The Sequencer starts writing transactions at block height 100.
• At the same time, the Syncer begins synchronizing transactions from block height 90 (assuming a 10-block delay).
• The Syncer saves transactions to the database with sortKey = 90 + 10, but after the sequencer has already saved its transactions at that block height.

Because the second part of the sortKey for sequencer transactions is always greater than those from L1, there is a risk that transactions from the sequencer at block height 100 will be processed (e.g. by DRE) before Syncer adds its transactions at that block height. This could cause the transactions from the Syncer to be effectively unused for evaluating state (because the client will have already calculated transactions for newer sortKeys - those from the sequencer).

Solution

Parameters

• CONFIRMATION_BLOCKS - how many blocks is Syncer waiting before downloading the block, e.g. 10. After this many blocks the risk of having a fork in the arweave blockchain is minimal
• READY_BLOCKS - how many blocks "into the future" is Syncer generating the sortKey with
• CURRENT_HEIGHT - block height returned by the Warp's Gateway
• SYNCER_FINISHED_HEIGHT - last block height that is saved by the Syncer
• SEQUENCER_CURRENT_HEIGHT - block height used in Sequencer to generate sortKeys

Block height used for SortKey generation

Syncer

It saves the block height it last synchronized (SYNCER_FINISHED_HEIGHT) in the database.
Syncer saves interactions with block height = SYNCER_FINISHED_HEIGHT + READY_BLOCKS.

For example:

• a sortKey for a transaction from block 85 (as reported by Arweave)
• with a 10-block CONFIRMATION_BLOCKS
• with a 15-block READY_BLOCKS

should have a block height of 100.

Sequencer

Sequencer assigns sortKeys using block height:

SEQUENCER_CURRENT_HEIGHT = SYNCER_FINISHED_HEIGHT + CONFIRMATION_BLOCKS

Sequencer doesn't know what is the current block height of Arweave network

Sequencer doesn't send interactions to Redis anymore.

Introduce Forwarder

Forwarder is a new component connected to the database and the Redis master node, it is responsible for:

• sending interactions from the database to Redis
• ensuring that after SEQUENCER_CURRENT_HEIGHT is updated in the database all L1 interactions are sent before any L2 interaction.

Forwarder gets interactions that are already bundled

graph TD
    A[Send L1 interactions <br/> to Redis ] -->|Success| B[Send L2 interactions <br/> as soon as they appear]
    B -->|Check| C{SEQUENCER_CURRENT_HEIGHT updated}
    C -->|Yes| A
    C -->|No| B

Loading

Implementation details:

Block height in interaciton table

• Syncer saves the original block height of the transaction
• Sequencer saves with the current block height as reported from the Warp's Gateway

Block height used in sortKey vs block key seen by the contract

	Sort Key	Block Height seen by the contract
Syncer	Original block height from tx + READY_BLOCKS	Original block height from tx
Sequencer	SYNCER_FINISHED_HEIGHT + CONFIRMATION_BLOCKS	Block height at request time

Gateway

Gateway needs to use SEQUENCER_CURRENT_HEIGHT when returning interactions. Only interactions with sortKeys generated with block height smaller than SEQUENCER_CURRENT_HEIGHT can be returned.

TODO (remove)

Additional SortKey

We should generate two sortKeys:

• first for contracts that only use L1 transactions (which should have the original block height, not the height where we synchronize) and
• second for contracts that use mixed or sequencer-only modes.

Data in the interaction column should reflect the block for which we are generating the sortKey (problem: what about double generation of sortKeys - do we save two interactions? that's a lot of data...).

[janekolszak]

sequenceDiagram
    autonumber
    participant Arweave Network
    participant Syncer
    participant DB
    participant Sequencer
    participant Forwarder
    loop For every CURRENT_HEIGHT 
        Note over Syncer: SYNCER_FINISHED_HEIGHT = <br/> CURRENT_HEIGHT - <br/> CONFIRMATION_BLOCKS 
        Syncer->>Arweave Network: Download block for height <br/> SYNCER_FINISHED_HEIGHT
        activate Syncer
        Syncer->>DB: Save interactions with <br/> sortKey1 = SYNCER_FINISHED_HEIGHT + <br/> READY_BLOCKS
        Syncer->>DB:  Save SYNCER_FINISHED_HEIGHT
        deactivate Syncer
    end
    loop For every SYNCER_FINISHED_HEIGHT
        activate Sequencer
        Sequencer->>DB: Get SYNCER_FINISHED_HEIGHT
        Note over Sequencer: SEQUENCER_CURRENT_HEIGHT = <br/> SYNCER_FINISHED_HEIGHT <br/> + CONFIRMATION_BLOCKS 
        Sequencer->>DB: Save SEQUENCER_CURRENT_HEIGHT
        Note over Sequencer: Use updated <br/> SEQUENCER_CURRENT_HEIGHT <br/> for generating sortKey
        deactivate Sequencer
    end 
    loop For every SEQUENCER_CURRENT_HEIGHT
        activate Forwarder
        Forwarder ->> DB: Get interactions <br/> already stored by Syncer
        Forwarder ->> DB: Get interactions <br/> added by Sequencer
        Note over Forwarder: Forward to Redis
        deactivate Forwarder
    end

Loading

[ppedziwiatr]

My dumb questions/comments:

1. READY_BLOCKS must be bigger than CONFIRMATION_BLOCKS, right? How much 'bigger' is safe/good enough?

2. IMHO the example for Syncer is a bit misleading, as it suggests that the 'base' height is taken from the original Arweave tx block height. However, the formula (SYNCER_FINISHED_HEIGHT + READY_BLOCKS) suggests that the 'base' height is taken from from the SYNCER_FINISHED_HEIGHT. I believe that (assuming that SYNCER_FINISHED_HEIGHT is set after storing all the transactions from the given height - ideally within the same database transaction that saves the arweave transactions from that height - as those two operations should be atomic) SYNCER_FINISHED_HEIGHT is always lower than tx.block.height.
   Or maybe - the formula is broken? ;-)

3. IMO the Sequencer example does not play well with the syncer example.
   Let's assume for a while, that the example for the Syncer is correct and that for a transaction at Arweave block.height = 85 - we've generated and stored an interaction with sortKey = 100,....
   So now we already have interactions at sortKey starting with 100,.... that are in our db - and someone can load them via through gateway and evaluate (and cache) the state.
   Since the original tx had the block height = 85 - the SYNCER_FINISHED_HEIGHT = 85.
   Now - the syncer will generate the sortKey according to formula:
   SEQUENCER_CURRENT_HEIGHT = SYNCER_FINISHED_HEIGHT + CONFIRMATION_BLOCKS
   - so in this case SEQUENCER_CURRENT_HEIGHT = 85 + 10 = 95
   => the sequencer generates the interactions at lower height that were already stored for the L1 transactions - I guess the problem here is obvious.

4. I don't get the SEQUENCER_CURRENT_HEIGHT updated part in the forwarder schema. The SEQUENCER_CURRENT_HEIGHT is (according to the specs) simply a sum of:
   SEQUENCER_CURRENT_HEIGHT = SYNCER_FINISHED_HEIGHT + CONFIRMATION_BLOCKS
   - why does it need to be stored/updated as a separate param?
   I believe the only params stored in the database should be
   SYNCER_FINISHED_HEIGHT, CONFIRMATION_BLOCKS, READ_BLOCKS

5. READY_BLOCKS - how many blocks is Sequencer waiting to take interactions from L1 into considerations, e.g. 15
   - what does exactly the to take interactions from L1 into considerations mean in this case?

6. Implementation details, 2nd. bullet point Sequencer saves with the current block height as reported from the Warp's Gateway - this suggests that Sequencer MUST use the current Arweave network block height (loaded from the Warp Gateway).
   But the paragraph for the Sequencer says (in bold!) - Sequencer doesn't know what is the current block height of Arweave network....

[ppedziwiatr]

One thing missing in the above specs is setting the 'previous sort key' (a.k.a. lastSortKey) for the mixed L1/L2 transactions - which I believe is non trivial and still not fully thought.

[janekolszak]

1. READY_BLOCKS should be bigger than CONFIRMATION_BLOCKS. It's not crucial for the system to work, but is keeps SEQUENCER_CURRENT_HEIGHT near CURRENT_HEIGHT and thus sortKey has the block_height component more or less equal to the real network height.

2. I'm not sure what "base height" means. It may be confusing that SYNCER_FINISHED_HEIGHT is used both as the height currently downloaded by the Syncer and the last fully downloaded L1 block, but I don't know how else to show it. Feel free to change the picture.

3. Transactions for the L1 interactions are stored in the database, but are used only after a specific SEQUENCER_CURRENT_HEIGHT is reached. Every select from the database will have a WHERE clause. This way it is impossible to mix L1 interactions that are stored waaaay into the future with the interactions added by the Sequencer.

L1 transactions are sent to the Redis only after the right SEQUENCER_CURRENT_HEIGHT is reached.

4. SEQUENCER_CURRENT_HEIGHT is updated to let the Forwarder know that Sequencer is using another block height. Forwarder will then ensure that L1 transactions get sent to Redis before L2. After SEQUENCER_CURRENT_HEIGHT is updated in the database Sequencer will not use a smaller one.

5. That's a mistake, Sequencer doesn't do anything with L1 transactions. It was a leftover (Forwarder used to be integrated in to the Sequencer). I re-worded, but I'm not sure if it's clear.

6. Block height used to generate sortKey is not the same as the block_height saved in the separate column in the interactions table. block_height from the table is passed to the contract.

[janekolszak]

lastSort key is set in Forwarder and specified per contract, only for L1. Sequencer handles lastSortKey for L2

[ppedziwiatr]

also - after the last call - no one really knows what is the point of READY_BLOCKS param :-), so we're leaving CONFIRMATION_BLOCKS only.
We need to be super careful with setting CONFIRMATION_BLOCKS, as this value must be known by the Warp SDK (so that it was able to properly generate sort keys for L1 transactions in case they were loaded directly from arweave.net gateway) - updating this value will be a pain in the ass.