Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(validium): Assert for smallPubdata test #194

Draft
wants to merge 164 commits into
base: feat_validium_pubdata_abstraction
Choose a base branch
from

Conversation

ColoCarletti
Copy link
Member

No description provided.

Artemka374 and others added 30 commits February 19, 2024 15:03
## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.
## What ❔

- Removes `*_from_env` functions from `vlog` crate.
- Introduces `ObservabilityConfig` for core and EN with env variable
based constructors that match the old behavior.
- Changes the binaries to use new approach.

## Why ❔

- `vlog` is a generic library, so it should have no assumptions on the
configuration system. It should be for users of `vlog` to decide where
they get the configuration from.
## What ❔

- Creates `en_syncTokens` RPC method that returns all tokens optionally
filtered by the miniblock at which the token was deployed.
- Uses this method during snapshot recovery to recover tokens.

## Why ❔

It's necessary to recover tokens in order for the VM sandbox to work
properly.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
- [x] Spellcheck has been run via `zk spellcheck`.
- [x] Linkcheck has been run via `zk linkcheck`.
…atter-labs#1098)

## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.

---------

Signed-off-by: Danil <deniallugo@gmail.com>
…matter-labs#1012)

## What ❔

This PR aims to reduce the diff between main and #298 to simplify the
future review.

The changes do not affect current server behavior.
Only changes related to the zkSync server went into this PR.

What didn't go into this PR:

- various variable renamings
- integration tests
- changes that depend on the future version of era-contracts

Where applicable, stubs and dummy (for now) config variables were
introduced, which can be replaced/populated later.

## Why ❔

Breaking up a PR into smaller chunks should be easier to review.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.

---------

Co-authored-by: Bence Haromi <56651250+benceharomi@users.noreply.github.com>
Co-authored-by: Bence Haromi <bence.haromi@gmail.com>
## What ❔

Take into account nonce from TxProxy, now if tx was submitted but
miniblock was not synced yet. en will return correct nonce

## Why ❔

[EN doesn't consider
](https://github.com/matter-labs/zksync-era/blob/main/core/lib/zksync_core/src/api_server/web3/namespaces/eth.rs#L303)transactions
currently residing in the TxProxy when calculating pending nonce, which
results in returned nonces being incorrect t times
## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.

---------

Signed-off-by: Danil <deniallugo@gmail.com>
Co-authored-by: Alex Ostrovski <aov@matterlabs.dev>
…#1133)

## What ❔

Documents the invariants after snapshot recovery, e.g. in the DAL crate
readme.

## Why ❔

Better DevEx (primarily internal for now).

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
- [x] Spellcheck has been run via `zk spellcheck`.
- [x] Linkcheck has been run via `zk linkcheck`.
## What ❔

* Added support for 4844 circuit to verification key generator

## Why ❔

* 4844 is the new circuit that will be used from the upcoming release.
## What ❔

EN runs tree in light mode.

## Why ❔

Full tree mode is no longer required for EN as commitment generation is
moved to a new component.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.
…labs#1143)

## What ❔

* Refactored the code, to keep passing ProverServiceDataKey as a circuit
identifier
* Added support for 4844 setup key generation

## Why ❔

* 4844 circuit was recently added.
## What ❔

- Implements shared thread-safe container for health checks,
`AppHealthCheck`.
- Refactors EN initialization to start the healthcheck server early into
the node lifecycle.
- Adds a healthcheck for snapshot recovery.
- Uses healthchecks in the snapshot recovery integration test.

## Why ❔

This increases EN observability.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
- [x] Spellcheck has been run via `zk spellcheck`.
- [x] Linkcheck has been run via `zk linkcheck`.
## What ❔

* Adding first part of support for 4844 circuit in prover.
* Added it to prover enums (like proof wrapper etc).
* But it is not connected to witness generators yet.

## Why ❔

* This is a part of the larger effort to handle 4844 blob support for
era.

---------

Co-authored-by: EmilLuta <EmilLuta@users.noreply.github.com>
Co-authored-by: Emil <evl@matterlabs.dev>
## What ❔

Return back token config for sepolia

## Why ❔

It was accidentally removed

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.
## What ❔

Removes ordering by `tx_index_in_block`

## Why ❔

Ordering by `tx_index_in_block` is not needed, (`miniblock_number `,
`event_index_in_block`) is enough. We currently have an index on
(`miniblock_number `, `tx_index_in_block`, `event_index_in_block`) and
it can be removed after this PR is applied.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.
## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.

---------

Signed-off-by: Danil <deniallugo@gmail.com>
## What ❔

Contract verifier allows any fields in standard json input settings and
passes them to zksolc.

## Why ❔

New fields are added to settings from time to time, contract verifier
should just accept them and pass to zksolc.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.
## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.

Signed-off-by: Danil <deniallugo@gmail.com>
…atter-labs#1151)

## What ❔

* Added recompute-if-missing option to key generator
* Updated setup.sh to use this option.

## Why ❔

* This allows to quickly recompute the setup keys that were missing,
making it easier to run local provers.
## What ❔

* Removing data files that are no longer needed

## Why ❔

* verification_leaf_1 - is now called 'verification_scheduler'
* verification_leaf_2 - is now called "verification_node'
* witness_arttifacts.json was used to generate a set of basic circuits,
but now we depend on the test_harness to get this list
* scheduler.bin was used in the past to generate snark wrapper
verification key - but since then we moved to verification key
generation that doesn't need a real proof anymore.
*
## What ❔

- adds force_evmla flag

## Why ❔

- it should be processed explicitly by verifier to pass `--force-evmla`
to zksolc

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [ ] Linkcheck has been run via `zk linkcheck`.
…#1145)

## What ❔
- Small change in the `EthWatch` component. Now `Eth Watch` 's `pool`
field is used instead of `ConnectionPool` passed in the Eth Watch's
`run()` method.
- Adds `Eth Watch` implementation, i. e. `EthWatchLayer` and
`EthWatchTask`, in the same way it was done for `StateKeeper `
[here](https://github.com/matter-labs/zksync-era/pull/1043/files#diff-8ff8babf7b83c79dbf96f4998cf71d888beaeb265c7ce33192b0bb0c808f662b).

Current "external" point of view for the framework(the only thing
changed is that we add ` .add_eth_watch_layer()?` after
`.add_state_keeper_layer()?`):

```rust
fn main() -> anyhow::Result<()> {
    #[allow(deprecated)] // TODO (QIT-21): Use centralized configuration approach.
    let log_format = vlog::log_format_from_env();
    let _guard = vlog::ObservabilityBuilder::new()
        .with_log_format(log_format)
        .build();

    MainNodeBuilder::new()
        .add_pools_layer()?
        .add_fee_input_layer()?
        .add_object_store_layer()?
        .add_metadata_calculator_layer()?
        .add_state_keeper_layer()?
        .add_eth_watch_layer()?
        .add_healthcheck_layer()?
        .build()
        .run()?;

    Ok(())
}
```
## What ❔

https://opensource.org/license/mit/ ->
https://opensource.org/blog/license/mit/

## Why ❔

CI is broken, seems like https://opensource.org changed the base URL for
the license pages

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
- [ ] Spellcheck has been run via `zk spellcheck`.
- [x] Linkcheck has been run via `zk linkcheck`.
## What ❔

Added a config flag to disable all filter-related methods. If turned on,
they report that the method is not implemented.

## Why ❔

If a node is behind a load balancer then there is no way for client to
reliably hit the same node where he/she created the filter. Hence
supporting this is a very awkward UX. Making this a flag since external
nodes might want to turn this on if they only have a single node mapped
to a static address.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
- [x] Spellcheck has been run via `zk spellcheck`.
- [x] Linkcheck has been run via `zk linkcheck`.
## What ❔

* Added verification key and finalization hint file for 4844 circuit
* Actually start saving finalization hint when generating verification
keys
* Fixed witness vector generator for 4844 circuit - used by GPU prover

## Why ❔

* Necessary steps to start creating 4844 proofs
…labs#1080)

## What ❔

- Propagates I/O errors in the state keeper using `anyhow::Result`,
adding context to calls where appropriate.
- Propagates DB errors in `FactoryDepsDal`.

## Why ❔

Propagating errors makes it easier to attach additional context to them;
thus, it improves DevEx and code maintainability.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
- [x] Spellcheck has been run via `zk spellcheck`.
- [x] Linkcheck has been run via `zk linkcheck`.
…#1205)

## What ❔

* Start using the new shivini function for  4844 circuit

## Why ❔

* Needed for GPU provers
ilitteri and others added 29 commits March 4, 2024 12:38
- First splits the functionality of
  `L1CommitBatchesDataGenerator::l1_commit_data` into two separate
  functions, one for a single batch and one for the whole array that
  includes the previous block.
- Then uses that for publish criteria and consistency checking, as they
  work on a per-batch basis.
…gen_mode_cross_check_en

feat(Validium): Commit data generation mode cross-check in EN
…ion' into feat_validium_data_commit_abstraction_v2
…ta_commit_abstraction_v2

feat(Validium): refactor batch commit data generation
…-labs/zksync-era into feat_modify_integration_test_for_validium_mode
toni-calvin pushed a commit that referenced this pull request Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.