-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(blockifier): added regression test for gas computation #968
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #968 +/- ##
==========================================
- Coverage 74.18% 70.36% -3.82%
==========================================
Files 359 87 -272
Lines 36240 11606 -24634
Branches 36240 11606 -24634
==========================================
- Hits 26886 8167 -18719
+ Misses 7220 3070 -4150
+ Partials 2134 369 -1765
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
e4d5fb6
to
cf2c186
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 4 files reviewed, 1 unresolved discussion (waiting on @aner-starkware)
crates/blockifier/src/fee/gas_usage_test.rs
line 292 at r1 (raw file):
// Assert gas computation results are as expected. The goal of this test is to prevent unwanted // changes to the gas computation. fn test_gas_computation_regression_test(
does it make sense that your fixes didn't break this test? @aner-starkware
maybe I should add cases which are affected by rounding up / down?
Code quote:
fn test_gas_computation_regression_test(
cf2c186
to
55d2500
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 4 files reviewed, 2 unresolved discussions (waiting on @aner-starkware)
crates/blockifier/src/fee/receipt_test.rs
line 316 at r1 (raw file):
// TODO(Aner, 21/01/24) modify for 4844 (taking blob_gas into account). // TODO(Nimrod, 1/5/2024): Test regression w.r.t. all resources (including VM). (Only starknet // resources are taken into account).
should this TODO be deleted?
Code quote:
// TODO(Nimrod, 1/5/2024): Test regression w.r.t. all resources (including VM). (Only starknet
// resources are taken into account).
Why are you Code quote: GasVectorComputationMode::NoL2Gas => match use_kzg_da {
true => GasVector { l1_gas: 26, l1_data_gas: 0, l2_gas: 0 },
false => GasVector { l1_gas: 26, l1_data_gas: 0, l2_gas: 0 },
}, |
Why are you Code quote: GasVectorComputationMode::All => match use_kzg_da {
true => GasVector { l1_gas: 0, l1_data_gas: 0, l2_gas: 1040000 },
false => GasVector { l1_gas: 0, l1_data_gas: 0, l2_gas: 1040000 },
}, |
Previously, amosStarkware wrote…
It's very suspicious. |
Where are the magic numbers from? Code quote: n_steps: 10000,
n_memory_holes: 0,
builtin_instance_counter: HashMap::from([
(BuiltinName::pedersen, 10),
(BuiltinName::range_check, 24),
(BuiltinName::ecdsa, 1),
(BuiltinName::bitwise, 1),
(BuiltinName::poseidon, 1),
]), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 2 of 4 files at r1, all commit messages.
Reviewable status: 2 of 4 files reviewed, 5 unresolved discussions (waiting on @amosStarkware)
crates/blockifier/src/fee/gas_usage_test.rs
line 364 at r2 (raw file):
false => GasVector { l1_gas: 62834, l1_data_gas: 0, l2_gas: 1127040 }, }, };
Not sure which option is better, but the expected l1_data_gas
depends only on the use_kzg_da
, the expected l2_gas
depends only on the gas_vector_computation_mode
and the expected l1_gas
can be written as a conditional summation of 21543 {+ 27} {+ 41291}
.
Code quote:
let expected_gas_vector = match gas_vector_computation_mode {
GasVectorComputationMode::NoL2Gas => match use_kzg_da {
true => GasVector { l1_gas: 21570, l1_data_gas: 2720, l2_gas: 0 },
false => GasVector { l1_gas: 62861, l1_data_gas: 0, l2_gas: 0 },
},
GasVectorComputationMode::All => match use_kzg_da {
true => GasVector { l1_gas: 21543, l1_data_gas: 2720, l2_gas: 1127040 },
false => GasVector { l1_gas: 62834, l1_data_gas: 0, l2_gas: 1127040 },
},
};
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 2 of 4 files reviewed, 6 unresolved discussions (waiting on @amosStarkware)
a discussion (no related file):
Blocked until reviewed by @dorimedini-starkware.
55d2500
to
a574c2c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 2 of 4 files reviewed, 6 unresolved discussions (waiting on @aner-starkware, @yoni, and @Yoni-Starkware)
crates/blockifier/src/test_utils.rs
line 392 at r2 (raw file):
Previously, aner-starkware wrote…
Where are the magic numbers from?
I chose values at random (this is true for all the values in the test) - I used as much values as possible, to increase the chance the test will break on any change to gas computation.
@Yoni-Starkware WYT of the current values?
do you want me to add more cases?
crates/blockifier/src/fee/gas_usage_test.rs
line 292 at r1 (raw file):
Previously, aner-starkware wrote…
It's very suspicious.
@Yoni-Starkware WYT?
crates/blockifier/src/fee/gas_usage_test.rs
line 338 at r2 (raw file):
Previously, aner-starkware wrote…
Why are you
match
ing if it makes no difference? Do you expect thevm_resources
to depend on theuse_kzg_da
flag?
I didn't notice there was no difference, Done
crates/blockifier/src/fee/gas_usage_test.rs
line 342 at r2 (raw file):
Previously, aner-starkware wrote…
Why are you
match
ing if it makes no difference? Do you expect thevm_resources
to depend on theuse_kzg_da
flag?
I didn't notice there was no difference, Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 4 files reviewed, 7 unresolved discussions (waiting on @amosStarkware and @Yoni-Starkware)
crates/blockifier/src/fee/gas_usage_test.rs
line 33 at r1 (raw file):
use crate::versioned_constants::{ResourceCost, StarknetVersion, VersionedConstants}; pub fn create_event_for_testing(keys_size: usize, data_size: usize) -> OrderedEvent {
Docstring.
Code quote:
pub fn create_event_for_testing(keys_size: usize, data_size: usize) -> OrderedEvent
crates/blockifier/src/fee/gas_usage_test.rs
line 50 at r3 (raw file):
#[fixture] fn starknet_resources() -> StarknetResources {
Docstring.
Code quote:
fn starknet_resources() -> StarknetResources
crates/blockifier/src/fee/gas_usage_test.rs
line 76 at r3 (raw file):
n_modified_contracts: 17, }); StarknetResources::new(2_usize, 3_usize, 4_usize, state_resources, 6.into(), execution_summary)
I think it'd be clearer if you wrote the argument names as above. Are the specific numbers arbitrary?
Code quote:
2_usize, 3_usize, 4_usize, state_resources, 6.into(), execution_summary
crates/blockifier/src/fee/gas_usage_test.rs
line 329 at r3 (raw file):
false => GasVector { l1_gas: 62834, l1_data_gas: 0, l2_gas: 87040 }, }, };
Here and below, it might be clearer if expected_l1_data_gas
depends only on use_kzg_da
, expected_l2_gas
depends only on gas_vector_computation_mode
, and expected_l1_gas
would be a conditional summation.
Code quote:
let expected_starknet_resources_gas_vector = match gas_vector_computation_mode {
GasVectorComputationMode::NoL2Gas => match use_kzg_da {
true => GasVector { l1_gas: 21544, l1_data_gas: 2720, l2_gas: 0 },
false => GasVector::from_l1_gas(62835),
},
GasVectorComputationMode::All => match use_kzg_da {
true => GasVector { l1_gas: 21543, l1_data_gas: 2720, l2_gas: 87040 },
false => GasVector { l1_gas: 62834, l1_data_gas: 0, l2_gas: 87040 },
},
};
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 4 files reviewed, 8 unresolved discussions (waiting on @amosStarkware and @aner-starkware)
crates/blockifier/src/test_utils.rs
line 392 at r2 (raw file):
Previously, amosStarkware wrote…
I chose values at random (this is true for all the values in the test) - I used as much values as possible, to increase the chance the test will break on any change to gas computation.
@Yoni-Starkware WYT of the current values?
do you want me to add more cases?
It's good enough
crates/blockifier/src/fee/gas_usage_test.rs
line 292 at r1 (raw file):
Previously, amosStarkware wrote…
@Yoni-Starkware WYT?
Yes, try to find numbers for such case
crates/blockifier/src/fee/gas_usage_test.rs
line 338 at r3 (raw file):
// Test VM resources. let vm_resources = get_vm_resource_usage(); let actual_vm_resources_gas_vector = get_vm_resources_cost(
Use ComputationResources
instead
Code quote:
let actual_vm_resources_gas_vector = get_vm_resources_cost(
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 4 of 4 files at r3, all commit messages.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @amosStarkware and @aner-starkware)
a discussion (no related file):
Previously, aner-starkware wrote…
Blocked until reviewed by @dorimedini-starkware.
reviewed
crates/blockifier/src/fee/gas_usage_test.rs
line 328 at r3 (raw file):
true => GasVector { l1_gas: 21543, l1_data_gas: 2720, l2_gas: 87040 }, false => GasVector { l1_gas: 62834, l1_data_gas: 0, l2_gas: 87040 }, },
does it make sense that one unit of L1 gas is 87040 units of L2 gas?
I would expect the computed gas vector - for L2 gas mode - would have significantly less L1 gas cost than the L! gas mode computation
Code quote:
GasVectorComputationMode::NoL2Gas => match use_kzg_da {
true => GasVector { l1_gas: 21544, l1_data_gas: 2720, l2_gas: 0 },
false => GasVector::from_l1_gas(62835),
},
GasVectorComputationMode::All => match use_kzg_da {
true => GasVector { l1_gas: 21543, l1_data_gas: 2720, l2_gas: 87040 },
false => GasVector { l1_gas: 62834, l1_data_gas: 0, l2_gas: 87040 },
},
crates/blockifier/src/fee/gas_usage_test.rs
line 346 at r3 (raw file):
let expected_vm_resources_gas_vector = match gas_vector_computation_mode { GasVectorComputationMode::NoL2Gas => GasVector { l1_gas: 26, l1_data_gas: 0, l2_gas: 0 }, GasVectorComputationMode::All => GasVector { l1_gas: 0, l1_data_gas: 0, l2_gas: 1040000 },
or use ..Default::default()
Suggestion:
GasVectorComputationMode::NoL2Gas => GasVector::from_l1_gas(26),
GasVectorComputationMode::All => GasVector::from_l2_gas(1040000),
a574c2c
to
88582b8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 5 files reviewed, 8 unresolved discussions (waiting on @aner-starkware, @dorimedini-starkware, and @Yoni-Starkware)
crates/blockifier/src/fee/gas_usage_test.rs
line 33 at r1 (raw file):
Previously, aner-starkware wrote…
Docstring.
I don't think this simple test util warrants a docstring
crates/blockifier/src/fee/gas_usage_test.rs
line 292 at r1 (raw file):
Previously, Yoni-Starkware (Yoni) wrote…
Yes, try to find numbers for such case
I don't see any L2 <-> L1 gas conversions in StarknetResources
(which, IIUC, had the bug @aner-starkware fixed), so I'm not sure I can test it.
I changed vm_resource_fee_cost.n_steps
to 30/10000, so that L2 <-> L1 gas conversions result in a fraction (which is turned into an integer).
does this fill the requirement?
or is this bad, because I haven't changed deprecated_l2_resource_gas_costs
& archival_data_gas_costs
and this makes the ratio between them wrong?
crates/blockifier/src/fee/gas_usage_test.rs
line 50 at r3 (raw file):
Previously, aner-starkware wrote…
Docstring.
I don't think this simple test util warrants a docstring
crates/blockifier/src/fee/gas_usage_test.rs
line 76 at r3 (raw file):
Previously, aner-starkware wrote…
I think it'd be clearer if you wrote the argument names as above. Are the specific numbers arbitrary?
they are arbitrary. StarknetResources::new
includes some logic, IMO it's better to use it
crates/blockifier/src/fee/gas_usage_test.rs
line 328 at r3 (raw file):
Previously, dorimedini-starkware wrote…
does it make sense that one unit of L1 gas is 87040 units of L2 gas?
I would expect the computed gas vector - for L2 gas mode - would have significantly less L1 gas cost than the L! gas mode computation
crates/blockifier/src/fee/gas_usage_test.rs
line 338 at r3 (raw file):
Previously, Yoni-Starkware (Yoni) wrote…
Use
ComputationResources
instead
Done
crates/blockifier/src/fee/gas_usage_test.rs
line 346 at r3 (raw file):
Previously, dorimedini-starkware wrote…
or use
..Default::default()
Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 5 files reviewed, 9 unresolved discussions (waiting on @aner-starkware, @dorimedini-starkware, and @Yoni-Starkware)
crates/blockifier/src/versioned_constants.rs
line 197 at r4 (raw file):
// Fee related. pub vm_resource_fee_cost: Arc<VmResourceCosts>,
@noaov1 is it ok I made this public?
Code quote:
pub vm_resource_fee_cost: Arc<VmResourceCosts>,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 5 of 5 files at r4, all commit messages.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @amosStarkware, @aner-starkware, @noaov1, and @Yoni-Starkware)
crates/blockifier/src/versioned_constants.rs
line 197 at r4 (raw file):
Previously, amosStarkware wrote…
@noaov1 is it ok I made this public?
if you can make do with pub(crate)
for now it will be easier to approve
crates/blockifier/src/fee/gas_usage_test.rs
line 325 at r4 (raw file):
let mut versioned_constants = VersionedConstants::get(StarknetVersion::V0_13_2_1).clone(); // Change the VM resource fee cost so that the L2 / L1 gas ratio is a fraction.
why?
we would want this test to also catch changes in the actual L1<>L2 ratio on 0.13.2.1, no?
Code quote:
Change the VM resource fee cost so that the L2 / L1 gas ratio is a fraction.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @amosStarkware, @aner-starkware, @noaov1, and @Yoni-Starkware)
crates/blockifier/src/fee/receipt_test.rs
line 316 at r1 (raw file):
Previously, amosStarkware wrote…
should this TODO be deleted?
yes, you covered it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @amosStarkware, @aner-starkware, @dorimedini-starkware, and @noaov1)
crates/blockifier/src/fee/gas_usage_test.rs
line 328 at r3 (raw file):
Previously, amosStarkware wrote…
1 L1 gas == 40,000 L2 gas.
Seems like a rounding issue, but I would expect to see 2 units of L1 gas. @amosStarkware please try to debug this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @amosStarkware, @aner-starkware, @dorimedini-starkware, and @noaov1)
crates/blockifier/src/fee/gas_usage_test.rs
line 292 at r1 (raw file):
Previously, amosStarkware wrote…
I don't see any L2 <-> L1 gas conversions in
StarknetResources
(which, IIUC, had the bug @aner-starkware fixed), so I'm not sure I can test it.
I changedvm_resource_fee_cost.n_steps
to 30/10000, so that L2 <-> L1 gas conversions result in a fraction (which is turned into an integer).
does this fill the requirement?
or is this bad, because I haven't changeddeprecated_l2_resource_gas_costs
&archival_data_gas_costs
and this makes the ratio between them wrong?
The purpose of deprecated_l2_resource_gas_costs
is to be consistent when replying to old blocks.
No constraints on future blocks and VC
88582b8
to
3d6bcce
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @aner-starkware, @dorimedini-starkware, and @Yoni-Starkware)
crates/blockifier/src/versioned_constants.rs
line 197 at r4 (raw file):
Previously, dorimedini-starkware wrote…
if you can make do with
pub(crate)
for now it will be easier to approve
Done
crates/blockifier/src/fee/gas_usage_test.rs
line 328 at r3 (raw file):
Previously, Yoni-Starkware (Yoni) wrote…
1 L1 gas == 40,000 L2 gas.
Seems like a rounding issue, but I would expect to see 2 units of L1 gas. @amosStarkware please try to debug this
Talked in slack, looks like there is no bug, and that this is because of rounding down.
crates/blockifier/src/fee/gas_usage_test.rs
line 325 at r4 (raw file):
Previously, dorimedini-starkware wrote…
why?
we would want this test to also catch changes in the actual L1<>L2 ratio on 0.13.2.1, no?
0.13.2.1
is arbitrary - the purpose of this test is to catch any changes to gas computation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 5 of 5 files at r5, all commit messages.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @amosStarkware and @aner-starkware)
crates/blockifier/src/fee/gas_usage_test.rs
line 50 at r3 (raw file):
Previously, amosStarkware wrote…
I don't think this simple test util warrants a docstring
I think you can add a short description of the resources - "a little bit of every resource"?
or, rename the fixture to something more indicative
crates/blockifier/src/fee/gas_usage_test.rs
line 325 at r4 (raw file):
Previously, amosStarkware wrote…
0.13.2.1
is arbitrary - the purpose of this test is to catch any changes to gas computation
isn't it possible that gas consumption will change when using 0.13.1 constants, and not with 0.13.2.1?
will this test be hard to parametrize for all released versions?
3d6bcce
to
7ff737a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @aner-starkware)
crates/blockifier/src/fee/gas_usage_test.rs
line 50 at r3 (raw file):
Previously, dorimedini-starkware wrote…
I think you can add a short description of the resources - "a little bit of every resource"?
or, rename the fixture to something more indicative
ok
crates/blockifier/src/fee/gas_usage_test.rs
line 325 at r4 (raw file):
Previously, dorimedini-starkware wrote…
isn't it possible that gas consumption will change when using 0.13.1 constants, and not with 0.13.2.1?
will this test be hard to parametrize for all released versions?
hmm it is possible.
it will take some work, added a TODO
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 1 of 1 files at r6, all commit messages.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @amosStarkware and @aner-starkware)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @amosStarkware)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @amosStarkware)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @amosStarkware)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @amosStarkware)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @amosStarkware)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @amosStarkware)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @amosStarkware)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @amosStarkware)
This change is