"large" gaps between the finalized block and the best-block

[josepot]

I'm not sure if this is a bug. However, it is a bit annoying from a UX point of view.

The issue is that there are instances in which smoldot doesn't report the finalized event for a ~1 minute or so. It's worth pointing out that the bestBlock event is being reported during that span of time. Therefore, in those instances there is a significant gap between the finalized block and the best block. I have witnessed instances with gaps of ~10 blocks.

Some logs:
smlogs.json

For instance, in the attached logs there is a gap between: 2024-04-08T11:46:56.632Z and 2024-04-08T11:47:52.829Z where no finalized events were emitted.

Once again, I'm not sure if this is a bug or not.

Thanks,

[tomaka]

Smoldot passively receives messages from full nodes that indicate which blocks have been finalized. In the logs you give, smoldot simply didn't receive any message from the full nodes.

The situation might be helped by making smoldot actively query finality proofs.
cc #1638 (comment)

[josepot]

The situation might be helped by making smoldot actively query finality proofs.

That would be great.

We could also add a method for fetching block justifications and -unless I'm mistaken- that would have the side-effect that smoldot would be able to determine if said block is finalized, and if it is then we would receive the finalized event over the chainHead subscription... is this assumption correct?

In our case we would only use said method call if we saw that we have a large amount of pinned blocks due to the fact that we haven't received the finalized event in a "long" time. For instance, if we see that there is a gap of N blocks between the best-block and the finalized one, and we also see that there are a few forks, then we could fetch the justification for the block in the finalized + (N * (2/3)) block.

[lexnv]

Once this substrate fix: paritytech/polkadot-sdk#4135 is deployed, smoldot will receive the Grandpa Neighbor packet with every finalized block.

Then, in smoldot we could download the block justifications (and not wait for the Grandpa Commit message).
In doing so, the fetching of the justification will also help #1638 and unblock subxt from updating to the latest version.

I think it would be possible to use the p2p sync protocol instead of adding the chainHead_justification method. Is there another way smoldot can fetch the justification of a block? (cc @tomaka)

[tomaka]

I don't understand why you both think that this is in any way related to the JSON-RPC API. This is about the syncing algorithm that smoldot uses to synchronize with the rest of the network.

[lexnv]

IIUC, the p2p sync protocol is the only way smoldot can fetch the justifications? 🤔

[tomaka]

Yes. Querying justifications is the easy part, and is already implemented. The difficult part is the syncing algorithm, which determines when to query justifications and from who, as it is the code that must resist all sorts of attacks.

---

I was unfortunately not aware of the fact that full nodes only send neighbor packets on epoch change. There's unfortunately not much we can do before paritytech/polkadot-sdk#4135, because smoldot has absolutely no way to tell when a block has been finalized by other peers other than passively waiting for them to send a commit message.

One possible hack is to randomly (like, every 6 seconds) try to download justifications, or something like that, but I'd much prefer to not do that.
Another possible hack would be to periodically disconnect and reconnect from peers, because they always send a neighbor packet (and thus tell their finalized block) when the connection is established, but again I'd prefer not do that.

[tomaka]

Closing as duplicate of #1638