feature(state-processor): send only validators set diffs to consensus #2143

abi87 · 2024-11-09T22:18:32Z

Whenever we turn the epoch, we must inform consensus of the changes in the validator set composition.
It's best to send only diffs of the validator set wrt previous epoch, rather than sending the whole new validator set.
This is what this PR does.

coderabbitai · 2024-11-09T22:18:39Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

codecov · 2024-11-09T22:20:37Z

Codecov Report

Attention: Patch coverage is 68.57143% with 11 lines in your changes missing coverage. Please review.

Project coverage is 26.59%. Comparing base (e72fe31) to head (0a5edd9).

Files with missing lines	Patch %	Lines
...-transition/pkg/core/state_processor_validators.go	65.62%	9 Missing and 2 partials ⚠️

Additional details and impacted files

@@                        Coverage Diff                         @@
##           complete-validator-set-updates    #2143      +/-   ##
==================================================================
+ Coverage                           26.51%   26.59%   +0.07%     
==================================================================
  Files                                 360      360              
  Lines                               16244    16276      +32     
  Branches                               12       12              
==================================================================
+ Hits                                 4307     4328      +21     
- Misses                              11658    11667       +9     
- Partials                              279      281       +2

Files with missing lines	Coverage Δ
mod/state-transition/pkg/core/state_processor.go	`35.52% <100.00%> (+0.24%)`	⬆️
...ate-transition/pkg/core/state_processor_genesis.go	`49.39% <100.00%> (ø)`
...-transition/pkg/core/state_processor_validators.go	`72.00% <65.62%> (ø)`

abi87 · 2024-11-09T22:21:40Z

mod/state-transition/pkg/core/state_processor.go

+	// prevEpochValidators tracks the set of validators active during
+	// previous epoch. This is useful at the turn of the epoch to send to
+	// consensus only the diffs rather than the full updated validator set.
+	prevEpochValidators []*transition.ValidatorUpdate


note that we only keep active validators here, i.e. those whose effective balance is larger than ejection one.
I do not expect this list to be beefy, although we should probably track its size and decide whether this is the best way to calculate diffs.

A less memory consuming approach is to track chances wherever we do them (e.g. when updating effective balances, or creating validators or removing it). I prefer current approach is it is more easily verifiable for correctness for me, but we should measure how much of a performance improvement these two alternatives bring

abi87 · 2024-11-09T22:22:53Z

mod/state-transition/pkg/core/state_processor_genesis.go

@@ -166,5 +166,5 @@ func (sp *StateProcessor[
 		return nil, err
 	}

-	return sp.processSyncCommitteeUpdates(st)


processSyncCommitteeUpdates does not really make sense to me.
It seems to me we're trying to squeeze a pure BeaconKit operation (updating CometBFT val set) into an Eth 2.0 one

abi87 · 2024-11-09T22:23:04Z

mod/state-transition/pkg/core/state_processor_staking_test.go

-	for i := range len(genDeposits) - 1 {
-		require.Equal(t, genVals[i], newEpochVals[i], fmt.Sprintf("idx: %d", i))
-	}
+	require.Len(t, newEpochVals, 1) // just topped up one validator


only send diffs

abi87 · 2024-11-09T22:23:16Z

mod/state-transition/pkg/core/state_processor_staking_test.go

-	for i := range len(genDeposits) {
-		require.Equal(t, genVals[i], newEpochVals[i], fmt.Sprintf("idx: %d", i))
-	}
+	require.Len(t, newEpochVals, 1) // just added 1 validator


only send diff.

abi87 · 2024-11-10T00:21:32Z

mod/state-transition/pkg/core/helpers_test.go

+var (
+	emptyAddress     = common.ExecutionAddress{}
+	emptyCredentials = types.NewCredentialsFromExecutionAddress(
+		emptyAddress,
+	)
+
+	dummyExecutionPayload = &types.ExecutionPayload{
+		Timestamp:     0,
+		ExtraData:     []byte("testing"),
+		Transactions:  [][]byte{},
+		Withdrawals:   []*engineprimitives.Withdrawal{},
+		BaseFeePerGas: math.NewU256(0),
+	}
+)


minor UT consolidation

…tes-optimizations

abi87 · 2024-11-10T16:25:15Z

mod/state-transition/pkg/core/state_processor_validators.go

+	// prevValsSet now contains all evicted validators (and only those)
+	for pkBytes := range prevValsSet {
+		//#nosec:G703 // bytes comes from a pk
+		pk, _ := bytes.ToBytes48([]byte(pkBytes))
+		res = append(res, &transition.ValidatorUpdate{
+			Pubkey:           pk,
+			EffectiveBalance: 0, // signal val eviction to consensus
+		})
+	}


we do not evict validators yet, so this should not really happen. Still added for completeness

calbera

logically makes sense, agree with trying to optimize this in future

…tes-optimizations

minor processSyncCommitteeUpdates renaming

eda7909

abi87 self-assigned this Nov 9, 2024

abi87 commented Nov 9, 2024

View reviewed changes

send only validators set diffs

c6b5586

abi87 force-pushed the validator-set-updates-optimizations branch from 6d61cc7 to c6b5586 Compare November 9, 2024 22:28

This was referenced Nov 9, 2024

bug(abci): We are updating the entire valset every block (this is big bad). #1095

Open

perf(abci) StateProcessor should produce a set of valset changes to pass to Comet #1120

Open

minor UT code consolidation

e9f01e2

abi87 marked this pull request as ready for review November 10, 2024 00:19

abi87 requested a review from itsdevbear as a code owner November 10, 2024 00:19

abi87 commented Nov 10, 2024

View reviewed changes

appease linter

3bc7678

abi87 mentioned this pull request Nov 10, 2024

feature(state-processor): update validators EffectiveBalance only when epoch turns #2142

Open

4 tasks

abi87 added 2 commits November 10, 2024 16:42

Merge branch 'complete-validator-set-updates' into validator-set-upda…

f361fe4

…tes-optimizations

nit

260bddb

abi87 commented Nov 10, 2024

View reviewed changes

README nit

f3c2b50

abi87 added the Ready for Review label Nov 11, 2024

calbera approved these changes Nov 11, 2024

View reviewed changes

nidhi-singh02 approved these changes Nov 13, 2024

View reviewed changes

abi87 added 2 commits November 14, 2024 17:28

Merge branch 'complete-validator-set-updates' into validator-set-upda…

d69d966

…tes-optimizations

Merge branch 'complete-validator-set-updates' into validator-set-upda…

0a5edd9

…tes-optimizations

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature(state-processor): send only validators set diffs to consensus #2143

feature(state-processor): send only validators set diffs to consensus #2143

abi87 commented Nov 9, 2024 •

edited

Loading

coderabbitai bot commented Nov 9, 2024 •

edited

Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

codecov bot commented Nov 9, 2024 •

edited

Loading

abi87 Nov 9, 2024

abi87 Nov 9, 2024

abi87 Nov 9, 2024

abi87 Nov 9, 2024

abi87 Nov 10, 2024

abi87 Nov 10, 2024

calbera left a comment

feature(state-processor): send only validators set diffs to consensus #2143

Are you sure you want to change the base?

feature(state-processor): send only validators set diffs to consensus #2143

Conversation

abi87 commented Nov 9, 2024 • edited Loading

coderabbitai bot commented Nov 9, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

codecov bot commented Nov 9, 2024 • edited Loading

Codecov Report

abi87 Nov 9, 2024

Choose a reason for hiding this comment

abi87 Nov 9, 2024

Choose a reason for hiding this comment

abi87 Nov 9, 2024

Choose a reason for hiding this comment

abi87 Nov 9, 2024

Choose a reason for hiding this comment

abi87 Nov 10, 2024

Choose a reason for hiding this comment

abi87 Nov 10, 2024

Choose a reason for hiding this comment

calbera left a comment

Choose a reason for hiding this comment

abi87 commented Nov 9, 2024 •

edited

Loading

coderabbitai bot commented Nov 9, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

codecov bot commented Nov 9, 2024 •

edited

Loading