cleanup: use test-utils for benching #112

jonathanpwang · 2023-08-17T06:07:53Z

Reduce code bloat in benchmarks by using gen_proof and check_proof helper functions
Add initial version of bench_builder helper function. Future benches should use this to save even more code.

* fix: change all `1` to `1u64` to prevent unexpected overflow (#72) * [Fix] Panic when dealing with identity point (#71) * More ecdsa tests * Update mod.rs * Update tests.rs * Update ecdsa.rs * Update ecdsa.rs * Update ecdsa.rs * msm tests * Update mod.rs * Update msm_sum_infinity.rs * fix: ec_sub_strict was panicing when output is identity * affects the MSM functions: right now if the answer is identity, there will be a panic due to divide by 0 instead of just returning 0 * there could be a more optimal solution, but due to the traits for EccChip, we just generate a random point solely to avoid divide by 0 in the case of identity point * Fix/fb msm zero (#77) * fix: fixed_base scalar multiply for [-1]P * feat: use `multi_scalar_multiply` instead of `scalar_multiply` * to reduce code maintanence / redundancy * fix: add back scalar_multiply using any_point * feat: remove flag from variable base `scalar_multiply` * feat: add scalar multiply tests for secp256k1 * fix: variable scalar_multiply last select * Fix/msm tests output identity (#75) * fixed base msm tests for output infinity * fixed base msm tests for output infinity --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> * feat: add tests and update CI --------- Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> Co-authored-by: yulliakot <yulliakotel@gmail.com> --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> * fix: redundant check in `ec_sub_unequal` * Add SafeType (#26) * Add SafeType * Refactor & add testing * Add doc comment * Refactor SafeChip * Move gen_proof/check_proof to utils * Fix merge issues * feat(CI): switch to larger runner * fix(builder): handle empty ctx with only equality constraints * feat: add SafeAddress and SafeUint160 (#85) * feat: add SafeAddress and SafeUint160 * fix incorrect byte size * Release 0.3.0 (#86) * feat: upgrade overall `halo2-base` API to support future multi-threaded assignments using our basic gate * WIP: currently `gates::flex_gate` is updated and passes basic test * BUG: `GateInstructions::idx_to_indicator` was missing a constraint to check that the indicator witness was equal to 1 when non-zero. * Previously the constraint ensured that `ind[i] = 0` when `idx != i` however `ind[idx]` could be anything!!! * update: working benches for `mul` and `inner_product` * feat: add `test_multithread_gates` * BUG: `get_last_bit` did not do an `assert_bit` check on the answer * this function was not used anywhere * fix: `builder::assign_*` was not handling cases where two gates overlap and there is a break point in that overlap * we need to copy a cell between columns to fix * feat: update `gates::range` to working tests and new API * In keygen mode, the `CircuitBuilder` will clone the `ThreadBuilder` instead of `take`ing it because the same circuit is used for both vk gen and pk gen. This could lead to more memory usage for pk gen. * fix: change `AssignedValue` type to `KeccakAssignedValue` for compatibility after halo2-base update * Initial version 0.3.0 of halo2-ecc (#12) * add multi-thread witness assignment support for `variable_base_msm` and `fixed_base_msm` * batch size 100 MSM witness generation went from 500ms -> 100ms * Sync with updates in `halo2_proofs_axiom` * `assign_advice` not longer returns `Result` so no more `unwrap` * Fix: assert uses of size hint in release mode (#13) * remove `size_hint` in `inner_product_simple` * change other uses of `size_hint` to follow with `assert_eq!` instead of `debug_assert_eq!` * Fix: bit decomposition edge cases (#14) * fix: change `debug_assert` in `decompose_u64_digits_limbs` to restrict `bit_len < 64` and `decompose_biguint` to `64 <= bit_len < 128` * add more comprehensive tests for above two functions * Initial checkpoint for halo2-ecc v0.3.0 (#15) * chore: clippy --fix * Feat/add readme (#4) * feat: add README * feat: re-enable `secp256k1` module with updated tests * chore: fix result println * chore: update Cargo halo2_proofs_axiom to axiom/dev branch * compatibility update with `halo2_proofs_axiom` Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> * Fix: make `GateThreadBuilder` compatible with external usage (#16) * chore: expose gate_builder.unknown * feat: `GateThreadBuilder::assign_all` takes assigned_{advices,constants} as input instead of new hashmap, in case we want to constrain equalities for cells not belonging to this builder * chore: update halo2-pse tag * fix: `GateThreadBuilder::assign_all` now returns `HashMap`s of assigned cells for external equality constraints (e.g., instance cells, `AssignedCells` from chips not using halo2-lib). * fix: `assign_all` was not assigning constants as desired: it was assigning a new constant per context. This leads to confusion and possible undesired consequences down the line. * Fix: under-constrained `idx_to_indicator` (#17) *fix(BUG): `GateChip::idx_to_indicator` still had soundness bug where at index `idx` the value could be 0 or 1 (instead of only 1) * feat: add some function documentation * test(idx_to_indicator): add comprehensive tests * both positive and negative tests * Fix: soundness error in `FpChip::assert_eq` due to typo (#18) * chore: update halo2-ecc version to 0.3.0 * fix(BUG): `FpChip::assert_equal` had `a` instead of `b` typo * feat: add tests for `FpChip::assert_eq` * positive and negative tests * Remove redundant code and prevent race conditions (#19) * feat: move `GateCircuitBuilder::synthesize` to `sub_synthesize` function which also returns the assigned advices. * reduces code duplication between `GateCircuitBuilder::synthesize` and `RangeCircuitBuilder::synthesize` and also makes it easier to assign public instances elsewhere (e.g., snark-verifier) * feat: remove `Mutex` to prevent non-deterministism * In variable and fixed base `msm_par` functions, remove use of `Mutex<GateThreadBuilder>` because even the `Mutex` is not thread- safe in the sense that: if you let `Mutex` decide order that `GateThreadBuilder` is unlocked, you may still add Contexts to the builder in a non-deterministic order. * fix: `fixed_base::msm_par` loading new zeros * In `msm_par` each parallelized context was loading a new zero via `ctx.load_zero()` * This led to using more cells than the non-parallelized version * In `fixed_base_msm_in`, the if statement depending on `rayon::current_number_threads` leads to inconsistent proving keys between different machines. This has been removed and now uses a fixed number `25`. * chore: use `info!` instead of `println` for params * Allow `assign_all` also if `witness_gen_only = true` * Fix: `inner_product_left_last` size hint (#25) * Add documentation for halo2-base (#27) * adds draft documentation for range.rs * draft docs for lib.rs, utiils.rs, builder.rs * fix: add suggested doc edits for range.rs * docs: add draft documentation for flex_gate.rs * fix: range.rs doc capitalization error * fix: suggested edits for utils.rs docs * fix: resolve comments for range.rs docs * fix: resolve comments on flex_gate.rs docs * fix: resolve comments for lib.rs, util.rs docs * fix: resolve comments for builder.rs docs * chore: use `info!` instead of `println` for params * Allow `assign_all` also if `witness_gen_only = true` * Fix: `inner_product_left_last` size hint (#25) * docs: minor fixes --------- Co-authored-by: PatStiles <pdstiles78@gmail.com> * Smart Range Builder (#29) * feat: smart `RangeCircuitBuilder` Allow `RangeCircuitBuilder` to not create lookup table if it detects that there's nothing to look up. * feat: add `RangeWithInstanceCircuitBuilder` * Moved from `snark-verifier-sdk` * Also made this circuit builder smart so it doesn't load lookup table if not necessary * In particular this can also be used as a `GateWithInstanceCircuitBuilder` * chore: derive Eq for CircuitBuilderStage * fix: RangeConfig should not unwrap LOOKUP_BITS * fix: `div_mod_var` when `a_num_bits <= b_num_bits` (#31) * Feat: extend halo2 base test coverage (#35) * feat: add flex_gate_test.rs and pos add() test * feat: add pos sub() test * feat: add pos neg() test * feat: add pos mul() test * feat: add pos mul_add() test * feat: add pos mul_not() test * feat: add pos assert_bit * feat: add pos div_unsafe() test * feat: add pos assert_is_const test * feat: add pos inner_product() test * feat: add pos inner_product_left_last() test * feat: add pos inner_product_with_sums test * feat: add pos sum_products_with_coeff_and_var test * feat: add pos and() test * feat: add pos not() test * feat: add pos select() test * feat: add pos or_and() test * feat: add pos bits_to_indicator() test * feat: add pos idx_to_indicator() test * feat: add pos select_by_indicator() test * feat: add pos select_from_idx() test * feat: add pos is_zero() test * feat: add pos is_equal() test * feat: add pos num_to_bits() test * feat: add pos lagrange_eval() test * feat: add pos get_field_element() test * feat: add pos range_check() tests * feat: add pos check_less_than() test * feat: add pos check_less_than_safe() test * feat: add pos check_big_less_than_safe() test * feat: add pos is_less_than() test * feat: add pos is_less_than_safe() test * feat: add pos is_big_less_than_safe() test * feat: add pos div_mod() test * feat: add pos get_last_bit() test * feat: add pos div_mod_var() test * fix: pass slices into test functions not arrays * feat: Add pos property tests for flex_gate * feat: Add positive property tests for flex_gate * feat: add pos property tests for range_check.rs * feat: add neg pranking test for idx_to_indicator * fix: change div_mod_var test values * feat(refactor): refactor property tests * fix: fix neg test, assert_const, assert_bit * fix: failing prop tests * feat: expand negative testing is_less_than_failing * fix: Circuit overflow errors on neg tests * fix: prop_test_mul_not * fix: everything but get_last_bit & lagrange * fix: clippy * fix: set LOOKUP_BITS in range tests, make range check neg test more robust * fix: neg_prop_tests cannot prank inputs Inputs have many copy constraints; pranking initial input will cause all copy constraints to fail * fix: test_is_big_less_than_safe, 240 bits max * Didn't want to change current `is_less_than` implementation, which in order to optimize lookups for smaller bits, only works when inputs have at most `(F::CAPACITY // lookup_bits - 1) * lookup_bits` bits * fix: inline doc for lagrange_and_eval * Remove proptest for lagrange_and_eval and leave as todo * tests: add readme about serial execution --------- Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> * fix(ecdsa): allow u1*G == u2*PK case (#36) NOTE: current ecdsa requires `r, s` to be given as proper CRT integers TODO: newtypes to guard this assumption * fix: `log2_ceil(0)` should return `0` (#37) * Guard `ScalarField` byte representations to always be little-endian (#38) fix: guard `ScalarField` to be little-endian * fix: get_last_bit two errors (#39) 2 embarassing errors: * Witness gen for last bit was wrong (used xor instead of &) * `ctx.get` was called after `range_check` so it was getting the wrong cell * Add documentation for all debug_asserts (#40) feat: add documentation for all debug_asserts * fix: `FieldChip::divide` renamed `divide_unsafe` (#41) Add `divide` that checks denomintor is nonzero. Add documentation in cases where `divide_unsafe` is used. * Use new types to validate input assumptions (#43) * feat: add new types `ProperUint` and `ProperCrtUint` To guard around assumptions about big integer representations * fix: remove unused `FixedAssignedCRTInteger` * feat: use new types for bigint and field chips New types now guard for different assumptions on non-native bigint arithmetic. Distinguish between: - Overflow CRT integers - Proper BigUint with native part derived from limbs - Field elements where inequality < modulus is checked Also add type to help guard for inequality check in ec_add_unequal_strict Rust traits did not play so nicely with references, so I had to switch many functions to move inputs instead of borrow by reference. However to avoid writing `clone` everywhere, we allow conversion `From` reference to the new type via cloning. * feat: use `ProperUint` for `big_less_than` * feat(ecc): add fns for assign private witness points that constrain point to lie on curve * fix: unnecessary lifetimes * chore: remove clones * Better handling of EC point at infinity (#44) * feat: allow `msm_par` to return identity point * feat: handle point at infinity `multi_scalar_multiply` and `multi_exp_par` now handle point at infinity completely Add docs for `ec_add_unequal, ec_sub_unequal, ec_double_and_add_unequal` to specify point at infinity leads to undefined behavior * feat: use strict ec ops more often (#45) * `msm` implementations now always use `ec_{add,sub}_unequal` in strict mode for safety * Add docs to `scalar_multiply` and a flag to specify when it's safe to turn off some strict assumptions * feat: add `parallelize_in` helper function (#46) Multi-threading of witness generation is tricky because one has to ensure the circuit column assignment order stays deterministic. To ensure good developer experience / avoiding pitfalls, we provide a new helper function for this. Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> * fix: minor code quality fixes (#47) * feat: `fixed_base::msm_par` handles identity point (#48) We still require fixed base points to be non-identity, but now handle the case when scalars may be zero or the final MSM value is identity point. * chore: add assert for query_cell_at_pos (#50) * feat: add Github CI running tests (#51) * fix: ignore code block for doctest (#52) * feat: add docs and assert with non-empty array checks (#53) * Release 0.3.0 ecdsa tests (#54) * More ecdsa tests * Update mod.rs * Update tests.rs * Update ecdsa.rs * Update ecdsa.rs * Update ecdsa.rs * chore: sync with release-0.3.0 and update CI Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> * chore: fix CI cannot multi-thread tests involving lookups due to environment variables * fix: `prop_test_is_less_than_safe` (#58) This test doesn't run any prover so the input must satisfy range check assumption. More serious coverage is provided by `prop_test_neg_is_less_than_safe` * Add halo2-base readme (#66) * feat: add halo2-base readme * fix: readme formatting * fix: readme edits * fix: grammer * fix: use relative links and formatting * fix: formatting * feat: add RangeCircuitBuilder description * feat: rewording and small edits --------- Co-authored-by: PatStiles <pdstiles78@gmail.com> * fix: change all `1` to `1u64` to prevent unexpected overflow (#72) * [Fix] Panic when dealing with identity point (#71) * More ecdsa tests * Update mod.rs * Update tests.rs * Update ecdsa.rs * Update ecdsa.rs * Update ecdsa.rs * msm tests * Update mod.rs * Update msm_sum_infinity.rs * fix: ec_sub_strict was panicing when output is identity * affects the MSM functions: right now if the answer is identity, there will be a panic due to divide by 0 instead of just returning 0 * there could be a more optimal solution, but due to the traits for EccChip, we just generate a random point solely to avoid divide by 0 in the case of identity point * Fix/fb msm zero (#77) * fix: fixed_base scalar multiply for [-1]P * feat: use `multi_scalar_multiply` instead of `scalar_multiply` * to reduce code maintanence / redundancy * fix: add back scalar_multiply using any_point * feat: remove flag from variable base `scalar_multiply` * feat: add scalar multiply tests for secp256k1 * fix: variable scalar_multiply last select * Fix/msm tests output identity (#75) * fixed base msm tests for output infinity * fixed base msm tests for output infinity --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> * feat: add tests and update CI --------- Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> Co-authored-by: yulliakot <yulliakotel@gmail.com> --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> * [Fix] scalar multiply completeness (#82) * fix: replace `scalar_multiply` with passthrough to MSM for now * feat(msm): use strict mode always * Previously did not use strict because we make assumptions about the curve `C`. Since this was not documented and is easy to miss, we use strict mode always. * docs: add assumptions to ec_sub_strict (#84) * fix: readme from previous merge * chore: cleanup CI for merge into main * chore: fix readme --------- Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> Co-authored-by: PatStiles <pdstiles78@gmail.com> Co-authored-by: PatStiles <33334338+PatStiles@users.noreply.github.com> Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> * chore: fix halo2_proofs_axiom SHA commit * Merge release v0.3.0 into develop (#90) * feat: `FpChip::range_check` now works with `max_bits < n * (k-1)` (#91) * feat(base): range_check 0 bits by asserting is zero * chore: add range_check 0 bits test * feat(ecc): `FpChip::range_check` now works with `max_bits < n * (k-1)` * fix(test): zkevm-keccak test should have `first_pass = SKIP_FIRST_PASS` (#96) Currently with `first_pass = true`, it skips the first pass, but when feature "halo2-axiom" is used, there is only one pass of `synthesize` so the whole thing gets skipped. Mea culpa! * Feat: test suite (#92) * feat: stop using env var to pass around FLEX_GATE_CONFIG_PARAMS and LOOKUP_BITS. Bad for testing (multi-threaded issues). Now we use thread_local to have a global static for these config params that can be passed around. * chore: make utils folder and move some files * Fix halo2 base tests naming (#76) * feat: `BaseConfig` to switch between `FlexGateConfig` and `RangeConfig` - `RangeCircuitBuilder` now uses `BaseConfig` to auto-decide whether to create lookup table or not. - In the future this should be renamed `BaseCircuitBuilder` or just `CircuitBuilder`, but for backwards compatibility we leave the name for now. - `GateCircuitBuilder` no longer implements `Circuit` because we want to switch to having everyone just use `RangeCircuitBuilder`. - Tests won't compile now because we still need to refactor * feat: refactored halo2-base tests to use new test suite * feat: remove use of env var in halo2-ecc CI now can just run `cargo test` * feat: remove use of env var from zkevm-keccak * Add zkevm-keccak test to CI * chore: fix CI * chore: add lint to CI * chore: make Baseconfig fns public * fix(test): zkevm-keccak test should have `first_pass = SKIP_FIRST_PASS` Currently with `first_pass = true`, it skips the first pass, but when feature "halo2-axiom" is used, there is only one pass of `synthesize` so the whole thing gets skipped. Mea culpa! --------- Co-authored-by: Xinding Wei <weixinding@gmail.com> * chore: make `bit_length` const function * feat: add debugging functions (#99) * feat: add debugging functions Functions only available for testing: * `ctx.debug_assert_false` for debug break point to search for other constrain failures in mock prover * `assigned_value.debug_prank(prank_value)` to prank witness values for negative tests * chore: code pretty * chore: remove cfg(test) for debug functions * feat(halo2-base): add `GateChip::pow_var` (#103) * Use halo2curves v0.4.0 and ff v0.13 (#107) * wip: change import to ff v0.13 * feat: remove `GateInstructions::get_field_element` halo2curves now has `bn256-table` which creates table of small field elements at compile time, so we should just use `F::from` always. This also improves readability. * chore: fix syntax and imports after update * chore: add asm feature * chore: workspace.resolver = 2 * chore: update ethers-core * chore: add jemallocator feature to zkevm-keccak crate * test: add bigger test case to keccak prover * feat: use `configure_with_params` remove `thread_local!` usage * chore: bump zkevm-keccak version to 0.1.1 * feat: add `GateThreadBuilder::from_stage` for convenience * chore: fixes * fix: removed `lookup_bits` from `GateThreadBuilder::config` * fix: debug_assert_false should load witness for debugging * chore: use unreachable to document that Circuit::configure is never used * chore: fix comment * feat(keccak): use configure_with_params * chore: fix halo2-pse errors * chore: change halo2_proofs to main * Add `sub_mul` to GateInstructions (#102) * Add `sub_mul` to GateInstructions * Add `sub_mul` prop test * fix(test): `select_from_idx` wasn't calling the right function (#105) * chore: add back RangeCircuitBuilder::config (#111) * fix: `RangeCircuitBuilder::config` remember `lookup_bits` * [Feat] Add Poseidon Hasher Chip (#110) * Add Poseidon chip * chore: minor fixes * test(poseidon): add compatbility tests Cherry-picked from #98 Co-authored-by: Antonio Mejías Gil <anmegi.95@gmail.com> * chore: minor refactor to more closely match snark-verifier https://github.com/axiom-crypto/snark-verifier/blob/main/snark-verifier/src/util/hash/poseidon.rs --------- Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> Co-authored-by: Antonio Mejías Gil <anmegi.95@gmail.com> * feat: add VariableByteArray (#88) * feat: add VariableByteArray * fix: correct type in panic msg * feat: make MAX_VAR_LEN const generic * feat: add `SafeBool` and `SafeByte` types These are very common so we have separate wrapper to avoid the extra length 1 vector heap allocation. * wip: add VarLenBytes * Refactor VarLenBytes Add VarLenBytesVec and FixLenBytes Fix tests * Add unsafe methods for bytes Address NITs --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> * chore: reduce CI real prover load * Rename PoseidonHasherChip to PoseidonHasher (#113) Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> * chore(safe_types): add conversion `SafeType` to/from `FixLenBytes` * chore(safe_type): add `unsafe_to_safe_type` unsafe conversion fn * feat: add `select_array_by_indicator` to `GateInstructions` (#115) feat(base): add `select_array_by_indicator` to `GateInstructions` * cleanup: use test-utils for benching (#112) * cleanup: use test-utils for benching * feat: add `{gen,check}_proof_with_instances` * feat: add initial `bench_builder` cmd to `BaseTester` * fix: cargo fmt * fix(safe_types): `VarLenBytes` should allow `len == MAX_LEN` (#117) * [feat] Add Poseidon Chip (#114) * Add Poseidon hasher * Fix test/lint * Fix nits * Fix lint * Fix nits & add comments * Add prover test * Fix CI * [chore] Reorg Folder Structure of hashes/zkevm (#118) * chore: rename crate zkevm-keccak to zkevm-hashes * fix: add `input_len` back to `KeccakTable` * chore: move keccak specific constants to `keccak_packed_multi/util` * Fix test --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> * [fix] CI for zkevm hashes (#119) Fix CI for zkevm hashes * [chore] Split keccak implementation into multiple files (#120) Split keccak implementation into multiple files * feat: keccak constant visibility changes (#121) feat: constant visibility changes * [feat] Keccak Raw Output (#122) * Replace raw keccak output instead of RLCOC * Fix lint * Add comments & improve expression performance * Virtual region managers and dynamic lookup support (#123) * feat: add `VirtualRegionManager` trait Create `CopyConstraintManager` to manage global copy constraints. * wip: separate `SinglePhaseGateManager` and `CopyConstraintManager` `GateThreadBuilder` was very messy before Todo: - Split out lookup functionality * wip: add `LookupAnyManager` * wip: `RangeChip` with `LookupAnyManager` reorg: - previous `builder/threads` moved to `flex_gate/threads` because it is all part of `FlexGateConfig` advice assignment logic - `builder` moved to `range/circuit/builder.rs` as it is part of the assignment logic of `RangeCircuitBuilder` * feat: working `BaseCircuitBuilder` backwards compatible - `GateThreadBuilder` has become `MultiPhaseCoreManager` - Some of the functionality has been moved into `BaseCircuitBuilder`, which is a generalization of `RangeCircuitBuilder` - Some fixes on virtual managers because keygen calls `synthesize` twice (once for vk, once for pk) so can't drop * fix: update halo2-ecc and sort `constant_equalities` Sort `constant_equalities` to ensure deterministism. Update `halo2-ecc` (mostly the tests) with new circuit builder format. * fix: `LookupAnyManager` drop check `Arc` strong_count * feat: add back single column lookup with selector Special case: if only single advice column that you need to lookup, you can create a selector and enable lookup on that column. This means you add 1 selector column, instead of 1 advice column. Only using this for `RangeConfig` and not generalizing it for now. * feat: add example of dynamic lookup memory table * Bump versions to 0.4.0 * chore: re-enable poseidon and safe_types `Drop` for managers no longer panics because rust `should_panic` test cannot handle non-unwinding panics. * chore: remove `row_offset` from `assigned_advices` This PR was merged: privacy-scaling-explorations/halo2#192 * chore: move `range::circuit` to `gates::circuit` * nits: address review comments * feat: add `num_instance_columns` to `BaseCircuitParams` No longer a const generic * chore(CI): use larger runner * Use `raw_assign_{advice,fixed}` in keccak (#125) * chore: switch `halo2_proofs` branch to `main` * chore: use `raw_assign_{advice,fixed}` in keccak * [feat] PoseidonHasher supports multiple inputs in compact format (#127) * PoseidonHasher supports multiple inputs in compact format * Add comments * Remove unnecessary uses * [feat] Expose Keccack Raw Inputs in Bytes instead of Input RLCs (#124) * Expose Keccack raw inputs in bytes instead of input RLCs * Fix column name in comments * Add comments * Compress 8 bytes of inputs into a single witness * chore: add some comments * Rewrite gates * Fix comments & typos * Fix naming * Add comments * Selector improvement * Remove unused --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> * Bump `zkevm-hashes` to v0.1.4 * chore: clippy fix * Generic vertical gate assignment (#129) * feat: make `single_phase::assign_with_constraints` generic Use const generic for max rotations accessed by the vertical gate. This way we can re-use the code for RLC gate. * chore: make single_phase pub * feat: add safety check for overlapping gates * Add `deep_clone` to `BaseCircuitBuilder` (#131) * chore: add convenience function to `BaseConfig` * feat: add `deep_clone` to `BaseCircuitBuilder` We sometimes want to clone `BaseCircuitBuilder` completely (for example to re-run witness generation). The derived clone only clones the shared references, instead of the underlying objects. * fix: `SingleCorePhaseManager` should not create thread in constructor Because the thread will default to phase 0. * chore: make `new_context` public * Convenience functions and fixes for multi-phase (#133) * feat: add `clear` function to circuit builder and managers * feat: add `BaseConfig::initialize` * fix: break points for multiphase * fix: clear should not change phase * chore: remove dbg * chore: add `BaseCircuitBuilder::set_k` fn * fix: `CopyConstraintManager::clear` was dropping * feat: impl `From<SafeByte>` for `AssignedValue` * chore(poseidon): add `derive` statements * fix(copy_constraints): backend permutation argument depends on order Backend implementation of `constrain_equal` depends on the order in which you add equality constraints, so it is not thread-safe... * feat: add `left_pad` functions for var length arrays (#137) * chore: use `PrimeField` for `OptimizedPoseidonSpec` (#139) * chore: add getter functions to Poseidon spec (#140) * feat: use `(TypeId, usize)` instead of `usize` for lookup tag (#142) * chore: add `ContextTag` type alias * feat(base): add `GateInstructions::inner_product_left` function (#143) * chore: fix comments * feat(base): add `GateInstructions::inner_product_left` function * [feat] Keccak Coprocessor Leaf Circuit (#130) * WIP * chore: make `KeccakAssignedRow` fields public * Refactor Keccak coprocessor circuit * Optimize Keccak circuit MAX_DEGREE * Fix comments * Fix bug & typos * Add testing & refactor folder structure * Debugging * Fix bugs * Fix typo & bug * feat(test): real prover tests use dummy input for keygen * chore: make `LoadedKeccakF` public * Also made `encoded_inputs_from_keccak_fs` public * Both are useful for external use to make lookup tables in app circuits * fix(keccak_leaf): review comments and optimization * chore: use `gate` when `range` not necessary * Move calculate base ciruit params out & Fix naming/comments * Make Field-related parameter functions const * feat: change `is_final` to `SafeBool` * nit * Fix typo --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> * [feat] App Circuit Utils for Keccak Coprocessor (#141) * Add keccak coprocessor encoding for VarLenBytesVec/FixLenBytesVec * Fix naming/nits * Fix nit * [chore] Fix fmt (#144) Fix fmt * chore: add misc utility functions (#146) * chore(keccak_leaf): make `generate_circuit_final_outputs` public * chore: add misc utility functions * feat(keccak): add `ingestion` module for Rust native input formatting (#147) * chore(keccak): use `snark-verifier` native Poseidon for encoding (#148) Currently only used for testing * feat: optimize leaf poseidon with `hash_compact_chunk_inputs` (#149) * [chore] cleanup code (#150) chore: cleanup code * chore: get halo2-pse working again * chore: fix fmt * chore: fix fmt * [Doc] Keccak Doc (#145) * Keccak docs * Fix typos * Add examples * Fix comments/docs * chore: pin snark-verifier branch * Revert "chore: pin snark-verifier branch" This reverts commit 1bf36b1. * [fix] max_rows in BaseCircuitBuilder in Keccak Leaf Circuit (#152) Fix max_rows in BaseCircuitBuilder inside Keccak Vanilla * fix: readme after merge * Fix bls test (#157) * fix bls test * rm * [fix] max_rows in BaseCircuitBuilder in Keccak Leaf Circuit (#152) Fix max_rows in BaseCircuitBuilder inside Keccak Vanilla * [chore] Remove Unnecessary Lookup in Keccak Coprocessor Leaf Circuit (#153) * chore: fix fmt * [fix] max_rows in BaseCircuitBuilder in Keccak Leaf Circuit (#152) Fix max_rows in BaseCircuitBuilder inside Keccak Vanilla * Remove lookup in Keccak Leaf circuit * Revert "chore: pin snark-verifier branch" This reverts commit 1bf36b1. * [rename] (coprocessor, leaf) -> (component, shard) (#161) * Fix schnorr test (#158) * fix schnorr test * fix infinite loop due to rnd not change, fmt * fix rng clone err * [fix] Multiple Phase Lookup (#162) Fix multiple phase lookup * chore: clippy fix --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> Co-authored-by: Xinding Wei <weixinding@gmail.com> Co-authored-by: PatStiles <33334338+PatStiles@users.noreply.github.com> Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> Co-authored-by: PatStiles <pdstiles78@gmail.com> Co-authored-by: mmagician <marcin.gorny.94@protonmail.com> Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> Co-authored-by: Antonio Mejías Gil <anmegi.95@gmail.com> Co-authored-by: MonkeyKing-1 <67293785+MonkeyKing-1@users.noreply.github.com> Co-authored-by: doxxed account - may discard soon, i love u since there's some nice work i've done with u but unfortunately i may have to switch again <104515688+justcode740@users.noreply.github.com>

* fix: change all `1` to `1u64` to prevent unexpected overflow (#72) * [Fix] Panic when dealing with identity point (#71) * More ecdsa tests * Update mod.rs * Update tests.rs * Update ecdsa.rs * Update ecdsa.rs * Update ecdsa.rs * msm tests * Update mod.rs * Update msm_sum_infinity.rs * fix: ec_sub_strict was panicing when output is identity * affects the MSM functions: right now if the answer is identity, there will be a panic due to divide by 0 instead of just returning 0 * there could be a more optimal solution, but due to the traits for EccChip, we just generate a random point solely to avoid divide by 0 in the case of identity point * Fix/fb msm zero (#77) * fix: fixed_base scalar multiply for [-1]P * feat: use `multi_scalar_multiply` instead of `scalar_multiply` * to reduce code maintanence / redundancy * fix: add back scalar_multiply using any_point * feat: remove flag from variable base `scalar_multiply` * feat: add scalar multiply tests for secp256k1 * fix: variable scalar_multiply last select * Fix/msm tests output identity (#75) * fixed base msm tests for output infinity * fixed base msm tests for output infinity --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> * feat: add tests and update CI --------- Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> Co-authored-by: yulliakot <yulliakotel@gmail.com> --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> * fix: redundant check in `ec_sub_unequal` * Add SafeType (#26) * Add SafeType * Refactor & add testing * Add doc comment * Refactor SafeChip * Move gen_proof/check_proof to utils * Fix merge issues * feat(CI): switch to larger runner * fix(builder): handle empty ctx with only equality constraints * feat: add SafeAddress and SafeUint160 (#85) * feat: add SafeAddress and SafeUint160 * fix incorrect byte size * Release 0.3.0 (#86) * feat: upgrade overall `halo2-base` API to support future multi-threaded assignments using our basic gate * WIP: currently `gates::flex_gate` is updated and passes basic test * BUG: `GateInstructions::idx_to_indicator` was missing a constraint to check that the indicator witness was equal to 1 when non-zero. * Previously the constraint ensured that `ind[i] = 0` when `idx != i` however `ind[idx]` could be anything!!! * update: working benches for `mul` and `inner_product` * feat: add `test_multithread_gates` * BUG: `get_last_bit` did not do an `assert_bit` check on the answer * this function was not used anywhere * fix: `builder::assign_*` was not handling cases where two gates overlap and there is a break point in that overlap * we need to copy a cell between columns to fix * feat: update `gates::range` to working tests and new API * In keygen mode, the `CircuitBuilder` will clone the `ThreadBuilder` instead of `take`ing it because the same circuit is used for both vk gen and pk gen. This could lead to more memory usage for pk gen. * fix: change `AssignedValue` type to `KeccakAssignedValue` for compatibility after halo2-base update * Initial version 0.3.0 of halo2-ecc (#12) * add multi-thread witness assignment support for `variable_base_msm` and `fixed_base_msm` * batch size 100 MSM witness generation went from 500ms -> 100ms * Sync with updates in `halo2_proofs_axiom` * `assign_advice` not longer returns `Result` so no more `unwrap` * Fix: assert uses of size hint in release mode (#13) * remove `size_hint` in `inner_product_simple` * change other uses of `size_hint` to follow with `assert_eq!` instead of `debug_assert_eq!` * Fix: bit decomposition edge cases (#14) * fix: change `debug_assert` in `decompose_u64_digits_limbs` to restrict `bit_len < 64` and `decompose_biguint` to `64 <= bit_len < 128` * add more comprehensive tests for above two functions * Initial checkpoint for halo2-ecc v0.3.0 (#15) * chore: clippy --fix * Feat/add readme (#4) * feat: add README * feat: re-enable `secp256k1` module with updated tests * chore: fix result println * chore: update Cargo halo2_proofs_axiom to axiom/dev branch * compatibility update with `halo2_proofs_axiom` Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> * Fix: make `GateThreadBuilder` compatible with external usage (#16) * chore: expose gate_builder.unknown * feat: `GateThreadBuilder::assign_all` takes assigned_{advices,constants} as input instead of new hashmap, in case we want to constrain equalities for cells not belonging to this builder * chore: update halo2-pse tag * fix: `GateThreadBuilder::assign_all` now returns `HashMap`s of assigned cells for external equality constraints (e.g., instance cells, `AssignedCells` from chips not using halo2-lib). * fix: `assign_all` was not assigning constants as desired: it was assigning a new constant per context. This leads to confusion and possible undesired consequences down the line. * Fix: under-constrained `idx_to_indicator` (#17) *fix(BUG): `GateChip::idx_to_indicator` still had soundness bug where at index `idx` the value could be 0 or 1 (instead of only 1) * feat: add some function documentation * test(idx_to_indicator): add comprehensive tests * both positive and negative tests * Fix: soundness error in `FpChip::assert_eq` due to typo (#18) * chore: update halo2-ecc version to 0.3.0 * fix(BUG): `FpChip::assert_equal` had `a` instead of `b` typo * feat: add tests for `FpChip::assert_eq` * positive and negative tests * Remove redundant code and prevent race conditions (#19) * feat: move `GateCircuitBuilder::synthesize` to `sub_synthesize` function which also returns the assigned advices. * reduces code duplication between `GateCircuitBuilder::synthesize` and `RangeCircuitBuilder::synthesize` and also makes it easier to assign public instances elsewhere (e.g., snark-verifier) * feat: remove `Mutex` to prevent non-deterministism * In variable and fixed base `msm_par` functions, remove use of `Mutex<GateThreadBuilder>` because even the `Mutex` is not thread- safe in the sense that: if you let `Mutex` decide order that `GateThreadBuilder` is unlocked, you may still add Contexts to the builder in a non-deterministic order. * fix: `fixed_base::msm_par` loading new zeros * In `msm_par` each parallelized context was loading a new zero via `ctx.load_zero()` * This led to using more cells than the non-parallelized version * In `fixed_base_msm_in`, the if statement depending on `rayon::current_number_threads` leads to inconsistent proving keys between different machines. This has been removed and now uses a fixed number `25`. * chore: use `info!` instead of `println` for params * Allow `assign_all` also if `witness_gen_only = true` * Fix: `inner_product_left_last` size hint (#25) * Add documentation for halo2-base (#27) * adds draft documentation for range.rs * draft docs for lib.rs, utiils.rs, builder.rs * fix: add suggested doc edits for range.rs * docs: add draft documentation for flex_gate.rs * fix: range.rs doc capitalization error * fix: suggested edits for utils.rs docs * fix: resolve comments for range.rs docs * fix: resolve comments on flex_gate.rs docs * fix: resolve comments for lib.rs, util.rs docs * fix: resolve comments for builder.rs docs * chore: use `info!` instead of `println` for params * Allow `assign_all` also if `witness_gen_only = true` * Fix: `inner_product_left_last` size hint (#25) * docs: minor fixes --------- Co-authored-by: PatStiles <pdstiles78@gmail.com> * Smart Range Builder (#29) * feat: smart `RangeCircuitBuilder` Allow `RangeCircuitBuilder` to not create lookup table if it detects that there's nothing to look up. * feat: add `RangeWithInstanceCircuitBuilder` * Moved from `snark-verifier-sdk` * Also made this circuit builder smart so it doesn't load lookup table if not necessary * In particular this can also be used as a `GateWithInstanceCircuitBuilder` * chore: derive Eq for CircuitBuilderStage * fix: RangeConfig should not unwrap LOOKUP_BITS * fix: `div_mod_var` when `a_num_bits <= b_num_bits` (#31) * Feat: extend halo2 base test coverage (#35) * feat: add flex_gate_test.rs and pos add() test * feat: add pos sub() test * feat: add pos neg() test * feat: add pos mul() test * feat: add pos mul_add() test * feat: add pos mul_not() test * feat: add pos assert_bit * feat: add pos div_unsafe() test * feat: add pos assert_is_const test * feat: add pos inner_product() test * feat: add pos inner_product_left_last() test * feat: add pos inner_product_with_sums test * feat: add pos sum_products_with_coeff_and_var test * feat: add pos and() test * feat: add pos not() test * feat: add pos select() test * feat: add pos or_and() test * feat: add pos bits_to_indicator() test * feat: add pos idx_to_indicator() test * feat: add pos select_by_indicator() test * feat: add pos select_from_idx() test * feat: add pos is_zero() test * feat: add pos is_equal() test * feat: add pos num_to_bits() test * feat: add pos lagrange_eval() test * feat: add pos get_field_element() test * feat: add pos range_check() tests * feat: add pos check_less_than() test * feat: add pos check_less_than_safe() test * feat: add pos check_big_less_than_safe() test * feat: add pos is_less_than() test * feat: add pos is_less_than_safe() test * feat: add pos is_big_less_than_safe() test * feat: add pos div_mod() test * feat: add pos get_last_bit() test * feat: add pos div_mod_var() test * fix: pass slices into test functions not arrays * feat: Add pos property tests for flex_gate * feat: Add positive property tests for flex_gate * feat: add pos property tests for range_check.rs * feat: add neg pranking test for idx_to_indicator * fix: change div_mod_var test values * feat(refactor): refactor property tests * fix: fix neg test, assert_const, assert_bit * fix: failing prop tests * feat: expand negative testing is_less_than_failing * fix: Circuit overflow errors on neg tests * fix: prop_test_mul_not * fix: everything but get_last_bit & lagrange * fix: clippy * fix: set LOOKUP_BITS in range tests, make range check neg test more robust * fix: neg_prop_tests cannot prank inputs Inputs have many copy constraints; pranking initial input will cause all copy constraints to fail * fix: test_is_big_less_than_safe, 240 bits max * Didn't want to change current `is_less_than` implementation, which in order to optimize lookups for smaller bits, only works when inputs have at most `(F::CAPACITY // lookup_bits - 1) * lookup_bits` bits * fix: inline doc for lagrange_and_eval * Remove proptest for lagrange_and_eval and leave as todo * tests: add readme about serial execution --------- Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> * fix(ecdsa): allow u1*G == u2*PK case (#36) NOTE: current ecdsa requires `r, s` to be given as proper CRT integers TODO: newtypes to guard this assumption * fix: `log2_ceil(0)` should return `0` (#37) * Guard `ScalarField` byte representations to always be little-endian (#38) fix: guard `ScalarField` to be little-endian * fix: get_last_bit two errors (#39) 2 embarassing errors: * Witness gen for last bit was wrong (used xor instead of &) * `ctx.get` was called after `range_check` so it was getting the wrong cell * Add documentation for all debug_asserts (#40) feat: add documentation for all debug_asserts * fix: `FieldChip::divide` renamed `divide_unsafe` (#41) Add `divide` that checks denomintor is nonzero. Add documentation in cases where `divide_unsafe` is used. * Use new types to validate input assumptions (#43) * feat: add new types `ProperUint` and `ProperCrtUint` To guard around assumptions about big integer representations * fix: remove unused `FixedAssignedCRTInteger` * feat: use new types for bigint and field chips New types now guard for different assumptions on non-native bigint arithmetic. Distinguish between: - Overflow CRT integers - Proper BigUint with native part derived from limbs - Field elements where inequality < modulus is checked Also add type to help guard for inequality check in ec_add_unequal_strict Rust traits did not play so nicely with references, so I had to switch many functions to move inputs instead of borrow by reference. However to avoid writing `clone` everywhere, we allow conversion `From` reference to the new type via cloning. * feat: use `ProperUint` for `big_less_than` * feat(ecc): add fns for assign private witness points that constrain point to lie on curve * fix: unnecessary lifetimes * chore: remove clones * Better handling of EC point at infinity (#44) * feat: allow `msm_par` to return identity point * feat: handle point at infinity `multi_scalar_multiply` and `multi_exp_par` now handle point at infinity completely Add docs for `ec_add_unequal, ec_sub_unequal, ec_double_and_add_unequal` to specify point at infinity leads to undefined behavior * feat: use strict ec ops more often (#45) * `msm` implementations now always use `ec_{add,sub}_unequal` in strict mode for safety * Add docs to `scalar_multiply` and a flag to specify when it's safe to turn off some strict assumptions * feat: add `parallelize_in` helper function (#46) Multi-threading of witness generation is tricky because one has to ensure the circuit column assignment order stays deterministic. To ensure good developer experience / avoiding pitfalls, we provide a new helper function for this. Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> * fix: minor code quality fixes (#47) * feat: `fixed_base::msm_par` handles identity point (#48) We still require fixed base points to be non-identity, but now handle the case when scalars may be zero or the final MSM value is identity point. * chore: add assert for query_cell_at_pos (#50) * feat: add Github CI running tests (#51) * fix: ignore code block for doctest (#52) * feat: add docs and assert with non-empty array checks (#53) * Release 0.3.0 ecdsa tests (#54) * More ecdsa tests * Update mod.rs * Update tests.rs * Update ecdsa.rs * Update ecdsa.rs * Update ecdsa.rs * chore: sync with release-0.3.0 and update CI Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> * chore: fix CI cannot multi-thread tests involving lookups due to environment variables * fix: `prop_test_is_less_than_safe` (#58) This test doesn't run any prover so the input must satisfy range check assumption. More serious coverage is provided by `prop_test_neg_is_less_than_safe` * Add halo2-base readme (#66) * feat: add halo2-base readme * fix: readme formatting * fix: readme edits * fix: grammer * fix: use relative links and formatting * fix: formatting * feat: add RangeCircuitBuilder description * feat: rewording and small edits --------- Co-authored-by: PatStiles <pdstiles78@gmail.com> * fix: change all `1` to `1u64` to prevent unexpected overflow (#72) * [Fix] Panic when dealing with identity point (#71) * More ecdsa tests * Update mod.rs * Update tests.rs * Update ecdsa.rs * Update ecdsa.rs * Update ecdsa.rs * msm tests * Update mod.rs * Update msm_sum_infinity.rs * fix: ec_sub_strict was panicing when output is identity * affects the MSM functions: right now if the answer is identity, there will be a panic due to divide by 0 instead of just returning 0 * there could be a more optimal solution, but due to the traits for EccChip, we just generate a random point solely to avoid divide by 0 in the case of identity point * Fix/fb msm zero (#77) * fix: fixed_base scalar multiply for [-1]P * feat: use `multi_scalar_multiply` instead of `scalar_multiply` * to reduce code maintanence / redundancy * fix: add back scalar_multiply using any_point * feat: remove flag from variable base `scalar_multiply` * feat: add scalar multiply tests for secp256k1 * fix: variable scalar_multiply last select * Fix/msm tests output identity (#75) * fixed base msm tests for output infinity * fixed base msm tests for output infinity --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> * feat: add tests and update CI --------- Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> Co-authored-by: yulliakot <yulliakotel@gmail.com> --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> * [Fix] scalar multiply completeness (#82) * fix: replace `scalar_multiply` with passthrough to MSM for now * feat(msm): use strict mode always * Previously did not use strict because we make assumptions about the curve `C`. Since this was not documented and is easy to miss, we use strict mode always. * docs: add assumptions to ec_sub_strict (#84) * fix: readme from previous merge * chore: cleanup CI for merge into main * chore: fix readme --------- Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> Co-authored-by: PatStiles <pdstiles78@gmail.com> Co-authored-by: PatStiles <33334338+PatStiles@users.noreply.github.com> Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> * chore: fix halo2_proofs_axiom SHA commit * Merge release v0.3.0 into develop (#90) * feat: `FpChip::range_check` now works with `max_bits < n * (k-1)` (#91) * feat(base): range_check 0 bits by asserting is zero * chore: add range_check 0 bits test * feat(ecc): `FpChip::range_check` now works with `max_bits < n * (k-1)` * fix(test): zkevm-keccak test should have `first_pass = SKIP_FIRST_PASS` (#96) Currently with `first_pass = true`, it skips the first pass, but when feature "halo2-axiom" is used, there is only one pass of `synthesize` so the whole thing gets skipped. Mea culpa! * Feat: test suite (#92) * feat: stop using env var to pass around FLEX_GATE_CONFIG_PARAMS and LOOKUP_BITS. Bad for testing (multi-threaded issues). Now we use thread_local to have a global static for these config params that can be passed around. * chore: make utils folder and move some files * Fix halo2 base tests naming (#76) * feat: `BaseConfig` to switch between `FlexGateConfig` and `RangeConfig` - `RangeCircuitBuilder` now uses `BaseConfig` to auto-decide whether to create lookup table or not. - In the future this should be renamed `BaseCircuitBuilder` or just `CircuitBuilder`, but for backwards compatibility we leave the name for now. - `GateCircuitBuilder` no longer implements `Circuit` because we want to switch to having everyone just use `RangeCircuitBuilder`. - Tests won't compile now because we still need to refactor * feat: refactored halo2-base tests to use new test suite * feat: remove use of env var in halo2-ecc CI now can just run `cargo test` * feat: remove use of env var from zkevm-keccak * Add zkevm-keccak test to CI * chore: fix CI * chore: add lint to CI * chore: make Baseconfig fns public * fix(test): zkevm-keccak test should have `first_pass = SKIP_FIRST_PASS` Currently with `first_pass = true`, it skips the first pass, but when feature "halo2-axiom" is used, there is only one pass of `synthesize` so the whole thing gets skipped. Mea culpa! --------- Co-authored-by: Xinding Wei <weixinding@gmail.com> * chore: make `bit_length` const function * feat: add debugging functions (#99) * feat: add debugging functions Functions only available for testing: * `ctx.debug_assert_false` for debug break point to search for other constrain failures in mock prover * `assigned_value.debug_prank(prank_value)` to prank witness values for negative tests * chore: code pretty * chore: remove cfg(test) for debug functions * feat(halo2-base): add `GateChip::pow_var` (#103) * Use halo2curves v0.4.0 and ff v0.13 (#107) * wip: change import to ff v0.13 * feat: remove `GateInstructions::get_field_element` halo2curves now has `bn256-table` which creates table of small field elements at compile time, so we should just use `F::from` always. This also improves readability. * chore: fix syntax and imports after update * chore: add asm feature * chore: workspace.resolver = 2 * chore: update ethers-core * chore: add jemallocator feature to zkevm-keccak crate * test: add bigger test case to keccak prover * feat: use `configure_with_params` remove `thread_local!` usage * chore: bump zkevm-keccak version to 0.1.1 * feat: add `GateThreadBuilder::from_stage` for convenience * chore: fixes * fix: removed `lookup_bits` from `GateThreadBuilder::config` * fix: debug_assert_false should load witness for debugging * chore: use unreachable to document that Circuit::configure is never used * chore: fix comment * feat(keccak): use configure_with_params * chore: fix halo2-pse errors * chore: change halo2_proofs to main * Add `sub_mul` to GateInstructions (#102) * Add `sub_mul` to GateInstructions * Add `sub_mul` prop test * fix(test): `select_from_idx` wasn't calling the right function (#105) * chore: add back RangeCircuitBuilder::config (#111) * fix: `RangeCircuitBuilder::config` remember `lookup_bits` * [Feat] Add Poseidon Hasher Chip (#110) * Add Poseidon chip * chore: minor fixes * test(poseidon): add compatbility tests Cherry-picked from #98 Co-authored-by: Antonio Mejías Gil <anmegi.95@gmail.com> * chore: minor refactor to more closely match snark-verifier https://github.com/axiom-crypto/snark-verifier/blob/main/snark-verifier/src/util/hash/poseidon.rs --------- Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> Co-authored-by: Antonio Mejías Gil <anmegi.95@gmail.com> * feat: add VariableByteArray (#88) * feat: add VariableByteArray * fix: correct type in panic msg * feat: make MAX_VAR_LEN const generic * feat: add `SafeBool` and `SafeByte` types These are very common so we have separate wrapper to avoid the extra length 1 vector heap allocation. * wip: add VarLenBytes * Refactor VarLenBytes Add VarLenBytesVec and FixLenBytes Fix tests * Add unsafe methods for bytes Address NITs --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> * chore: reduce CI real prover load * Rename PoseidonHasherChip to PoseidonHasher (#113) Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> * chore(safe_types): add conversion `SafeType` to/from `FixLenBytes` * chore(safe_type): add `unsafe_to_safe_type` unsafe conversion fn * feat: add `select_array_by_indicator` to `GateInstructions` (#115) feat(base): add `select_array_by_indicator` to `GateInstructions` * cleanup: use test-utils for benching (#112) * cleanup: use test-utils for benching * feat: add `{gen,check}_proof_with_instances` * feat: add initial `bench_builder` cmd to `BaseTester` * fix: cargo fmt * fix(safe_types): `VarLenBytes` should allow `len == MAX_LEN` (#117) * [feat] Add Poseidon Chip (#114) * Add Poseidon hasher * Fix test/lint * Fix nits * Fix lint * Fix nits & add comments * Add prover test * Fix CI * [chore] Reorg Folder Structure of hashes/zkevm (#118) * chore: rename crate zkevm-keccak to zkevm-hashes * fix: add `input_len` back to `KeccakTable` * chore: move keccak specific constants to `keccak_packed_multi/util` * Fix test --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> * [fix] CI for zkevm hashes (#119) Fix CI for zkevm hashes * [chore] Split keccak implementation into multiple files (#120) Split keccak implementation into multiple files * feat: keccak constant visibility changes (#121) feat: constant visibility changes * [feat] Keccak Raw Output (#122) * Replace raw keccak output instead of RLCOC * Fix lint * Add comments & improve expression performance * Virtual region managers and dynamic lookup support (#123) * feat: add `VirtualRegionManager` trait Create `CopyConstraintManager` to manage global copy constraints. * wip: separate `SinglePhaseGateManager` and `CopyConstraintManager` `GateThreadBuilder` was very messy before Todo: - Split out lookup functionality * wip: add `LookupAnyManager` * wip: `RangeChip` with `LookupAnyManager` reorg: - previous `builder/threads` moved to `flex_gate/threads` because it is all part of `FlexGateConfig` advice assignment logic - `builder` moved to `range/circuit/builder.rs` as it is part of the assignment logic of `RangeCircuitBuilder` * feat: working `BaseCircuitBuilder` backwards compatible - `GateThreadBuilder` has become `MultiPhaseCoreManager` - Some of the functionality has been moved into `BaseCircuitBuilder`, which is a generalization of `RangeCircuitBuilder` - Some fixes on virtual managers because keygen calls `synthesize` twice (once for vk, once for pk) so can't drop * fix: update halo2-ecc and sort `constant_equalities` Sort `constant_equalities` to ensure deterministism. Update `halo2-ecc` (mostly the tests) with new circuit builder format. * fix: `LookupAnyManager` drop check `Arc` strong_count * feat: add back single column lookup with selector Special case: if only single advice column that you need to lookup, you can create a selector and enable lookup on that column. This means you add 1 selector column, instead of 1 advice column. Only using this for `RangeConfig` and not generalizing it for now. * feat: add example of dynamic lookup memory table * Bump versions to 0.4.0 * chore: re-enable poseidon and safe_types `Drop` for managers no longer panics because rust `should_panic` test cannot handle non-unwinding panics. * chore: remove `row_offset` from `assigned_advices` This PR was merged: privacy-scaling-explorations/halo2#192 * chore: move `range::circuit` to `gates::circuit` * nits: address review comments * feat: add `num_instance_columns` to `BaseCircuitParams` No longer a const generic * chore(CI): use larger runner * Use `raw_assign_{advice,fixed}` in keccak (#125) * chore: switch `halo2_proofs` branch to `main` * chore: use `raw_assign_{advice,fixed}` in keccak * [feat] PoseidonHasher supports multiple inputs in compact format (#127) * PoseidonHasher supports multiple inputs in compact format * Add comments * Remove unnecessary uses * [feat] Expose Keccack Raw Inputs in Bytes instead of Input RLCs (#124) * Expose Keccack raw inputs in bytes instead of input RLCs * Fix column name in comments * Add comments * Compress 8 bytes of inputs into a single witness * chore: add some comments * Rewrite gates * Fix comments & typos * Fix naming * Add comments * Selector improvement * Remove unused --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> * Bump `zkevm-hashes` to v0.1.4 * chore: clippy fix * Generic vertical gate assignment (#129) * feat: make `single_phase::assign_with_constraints` generic Use const generic for max rotations accessed by the vertical gate. This way we can re-use the code for RLC gate. * chore: make single_phase pub * feat: add safety check for overlapping gates * Add `deep_clone` to `BaseCircuitBuilder` (#131) * chore: add convenience function to `BaseConfig` * feat: add `deep_clone` to `BaseCircuitBuilder` We sometimes want to clone `BaseCircuitBuilder` completely (for example to re-run witness generation). The derived clone only clones the shared references, instead of the underlying objects. * fix: `SingleCorePhaseManager` should not create thread in constructor Because the thread will default to phase 0. * chore: make `new_context` public * Convenience functions and fixes for multi-phase (#133) * feat: add `clear` function to circuit builder and managers * feat: add `BaseConfig::initialize` * fix: break points for multiphase * fix: clear should not change phase * chore: remove dbg * chore: add `BaseCircuitBuilder::set_k` fn * fix: `CopyConstraintManager::clear` was dropping * feat: impl `From<SafeByte>` for `AssignedValue` * chore(poseidon): add `derive` statements * fix(copy_constraints): backend permutation argument depends on order Backend implementation of `constrain_equal` depends on the order in which you add equality constraints, so it is not thread-safe... * feat: add `left_pad` functions for var length arrays (#137) * chore: use `PrimeField` for `OptimizedPoseidonSpec` (#139) * chore: add getter functions to Poseidon spec (#140) * feat: use `(TypeId, usize)` instead of `usize` for lookup tag (#142) * chore: add `ContextTag` type alias * feat(base): add `GateInstructions::inner_product_left` function (#143) * chore: fix comments * feat(base): add `GateInstructions::inner_product_left` function * [feat] Keccak Coprocessor Leaf Circuit (#130) * WIP * chore: make `KeccakAssignedRow` fields public * Refactor Keccak coprocessor circuit * Optimize Keccak circuit MAX_DEGREE * Fix comments * Fix bug & typos * Add testing & refactor folder structure * Debugging * Fix bugs * Fix typo & bug * feat(test): real prover tests use dummy input for keygen * chore: make `LoadedKeccakF` public * Also made `encoded_inputs_from_keccak_fs` public * Both are useful for external use to make lookup tables in app circuits * fix(keccak_leaf): review comments and optimization * chore: use `gate` when `range` not necessary * Move calculate base ciruit params out & Fix naming/comments * Make Field-related parameter functions const * feat: change `is_final` to `SafeBool` * nit * Fix typo --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> * [feat] App Circuit Utils for Keccak Coprocessor (#141) * Add keccak coprocessor encoding for VarLenBytesVec/FixLenBytesVec * Fix naming/nits * Fix nit * [chore] Fix fmt (#144) Fix fmt * chore: add misc utility functions (#146) * chore(keccak_leaf): make `generate_circuit_final_outputs` public * chore: add misc utility functions * feat(keccak): add `ingestion` module for Rust native input formatting (#147) * chore(keccak): use `snark-verifier` native Poseidon for encoding (#148) Currently only used for testing * feat: optimize leaf poseidon with `hash_compact_chunk_inputs` (#149) * [chore] cleanup code (#150) chore: cleanup code * chore: get halo2-pse working again * chore: fix fmt * [Doc] Keccak Doc (#145) * Keccak docs * Fix typos * Add examples * Fix comments/docs * chore: pin snark-verifier branch * [fix] max_rows in BaseCircuitBuilder in Keccak Leaf Circuit (#152) Fix max_rows in BaseCircuitBuilder inside Keccak Vanilla * [chore] Remove Unnecessary Lookup in Keccak Coprocessor Leaf Circuit (#153) * chore: fix fmt * [fix] max_rows in BaseCircuitBuilder in Keccak Leaf Circuit (#152) Fix max_rows in BaseCircuitBuilder inside Keccak Vanilla * Remove lookup in Keccak Leaf circuit * Revert "chore: pin snark-verifier branch" This reverts commit 1bf36b1. * [rename] (coprocessor, leaf) -> (component, shard) (#161) * [rename] (coprocessor, leaf) -> (component, shard) (#161) * [fix] Multiple Phase Lookup (#162) Fix multiple phase lookup * [fix] Multiple Phase Lookup (#162) Fix multiple phase lookup * [chore] add conversion `SafePrimitive` to `QuantumCell::Existing` (#169) chore: add conversion `SafePrimitive` to `QuantumCell::Existing` * fix: bad import on halo2-pse * [chore] add conversion `SafePrimitive` to `QuantumCell::Existing` (#169) chore: add conversion `SafePrimitive` to `QuantumCell::Existing` * fix: bad import on halo2-pse * chore: make `{Fixed,Var}LenBytes*` constructor public It's convenient to be able to construct the structs from vectors of safe bytes externally. Only unsafe-ness is `len <= max_len` is not checked. * chore: make `{Fixed,Var}LenBytes*` constructor public It's convenient to be able to construct the structs from vectors of safe bytes externally. Only unsafe-ness is `len <= max_len` is not checked. * chore(keccak): `format_requests` always returns true capacity (#171) * [chore] derive `Hash` for `BaseCircuitParams` (#172) chore: derive `Hash` for `BaseCircuitParams` * [chore] impl `AsRef, AsMut` for `BaseCircuitBuilder` to self (#173) chore: impl `AsRef, AsMut` for `BaseCircuitBuilder` to self * [chore] impl `AsRef, AsMut` for `BaseConfig` to self (#174) chore: impl `AsRef, AsMut` for `BaseConfig` to self * [chore] Add getters to `KeccakComponentShardCircuit` (#178) * chore: add getters to `KeccakComponentShardCircuit` For example, it's useful to access `BaseCircuitBuilder` to read public instances. * chore: `inputs` getter for `KeccakComponentShardCircuit` * feat: remove getter for `BaseCircuitBuilder` `BaseCircuitBuilder` is built during `synthesize` after raw vanilla circuit synthesis, so it should not be accessed externally. * [chore] Expose Keccak Packing (#180) Expose Keccak packing * [chore] Expose Keccak Format (#181) Expose Keccak format_input * [feat] basic dynamic lookup table gadget (#182) * feat: basic dynamic lookup table gadget * chore: fix imports * [chore] expose `spec` in `PoseidonHasher` (#183) chore: expose `spec` in `PoseidonHasher` * [chore] fix halo2-pse build error (#184) chore: fix halo2-pse build error * [feat] expose keccak table loading/packing functions for external crate usage (#195) * feat: expose `load_keccak_assigned_rows` for external crates to use * feat: split `encode_inputs_from_keccak_fs` into `pack_inputs_from_keccak_fs` and poseidon hashing part. The packing part can be used separately from the Poseidon-specific part. * chore: rename function * [chore] add getters to `PoseidonCompactChunkInput` (#196) chore: add getters to `PoseidonCompactChunkInput` * [chore] fix deref after using CopyGetters (#197) * chore: fix deref from get_copy * chore: add missing docs from getters * [feat] implement `CircuitExt` for `KeccakComponentShardCircuit` (#198) * chore: import `snark-verifier-sdk` * feat: implement `CircuitExt` for `KeccakComponentShardCircuit` so it can be aggregated by `snark-verifier-sdk` * chore: derive `Serialize` for keccak circuit params * chore: fix `snark-verifier-sdk` version * [chore] add `cargo audit` to CI (#207) chore: add `cargo audit` to CI Upgrade criterion version * [fix] `FieldChip::range_check` should take `FieldPoint` instead of `UnsafeFieldPoint` (#209) * fix: `FieldChip::range_check` should take `FieldPoint` instead of `UnsafeFieldPoint` * chore: fix audit-check CI * chore: toggle CI on release branches * [feat] update docs (#211) * feat: update doc comments with function assumptions * feat: update readme * chore: fix CI * [chore] fix dev graph tests (#212) * chore: CI uses clippy all-targets * fix: dev-graph tests (only works for halo2-pse) Didn't bother refactoring halo2-axiom to support dev-graph * [fix] `BasicDynLookupConfig` needs selector on advice table to prevent lookup poison (#206) * fix: add fixed column to `BasicDynLookupConfig` To prevent looking up into poisoned rows of `table`. * feat: change `memory` example to use `BasicDynLookupConfig` for testing * feat: change `BasicDynLookupConfig` to support zero key * chore: move helper functions to `utils::halo2` * [feat] add keccak circuit tests against Known Answer Test vectors (#213) feat: add keccak circuit tests against Known Answer Test vectors * [chore] fix documentation (#215) * chore: fix keccak comment * chore: remove redundant * chore: fix test case description * chore: fix documentation * chore: add comment * [chore] fix doc comment (#216) chore: fix doc comment * chore: fix halo2-pse compile * fix: `TypeId` in `ContextTag` not stable across builds (#217) * fix: use &str instead of TypeId in ContextTag * chore: add warning to readme * chore: fix comment * chore: fix RAM test to use `&str` type id * [chore] add crate prefix to `type_id`s (#218) * chore: add crate prefix to `type_id`s * fix: module_path! url * chore: add type_id warning to `Context::new` and `ContextCell::new` * chore: use halo2-axiom from crates.io * chore: use poseidon-primitives from crates.io * chore: Bump halo2-axiom to v0.4 * chore: add `get_mut` for keccak circuit params * chore: bring back readme --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> Co-authored-by: Xinding Wei <weixinding@gmail.com> Co-authored-by: PatStiles <33334338+PatStiles@users.noreply.github.com> Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> Co-authored-by: PatStiles <pdstiles78@gmail.com> Co-authored-by: mmagician <marcin.gorny.94@protonmail.com> Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> Co-authored-by: Antonio Mejías Gil <anmegi.95@gmail.com> Co-authored-by: MonkeyKing-1 <67293785+MonkeyKing-1@users.noreply.github.com>

* feat: upgrade overall `halo2-base` API to support future multi-threaded assignments using our basic gate * WIP: currently `gates::flex_gate` is updated and passes basic test * BUG: `GateInstructions::idx_to_indicator` was missing a constraint to check that the indicator witness was equal to 1 when non-zero. * Previously the constraint ensured that `ind[i] = 0` when `idx != i` however `ind[idx]` could be anything!!! * update: working benches for `mul` and `inner_product` * feat: add `test_multithread_gates` * BUG: `get_last_bit` did not do an `assert_bit` check on the answer * this function was not used anywhere * fix: `builder::assign_*` was not handling cases where two gates overlap and there is a break point in that overlap * we need to copy a cell between columns to fix * feat: update `gates::range` to working tests and new API * In keygen mode, the `CircuitBuilder` will clone the `ThreadBuilder` instead of `take`ing it because the same circuit is used for both vk gen and pk gen. This could lead to more memory usage for pk gen. * fix: change `AssignedValue` type to `KeccakAssignedValue` for compatibility after halo2-base update * Initial version 0.3.0 of halo2-ecc (#12) * add multi-thread witness assignment support for `variable_base_msm` and `fixed_base_msm` * batch size 100 MSM witness generation went from 500ms -> 100ms * Sync with updates in `halo2_proofs_axiom` * `assign_advice` not longer returns `Result` so no more `unwrap` * Fix: assert uses of size hint in release mode (#13) * remove `size_hint` in `inner_product_simple` * change other uses of `size_hint` to follow with `assert_eq!` instead of `debug_assert_eq!` * Fix: bit decomposition edge cases (#14) * fix: change `debug_assert` in `decompose_u64_digits_limbs` to restrict `bit_len < 64` and `decompose_biguint` to `64 <= bit_len < 128` * add more comprehensive tests for above two functions * Initial checkpoint for halo2-ecc v0.3.0 (#15) * chore: clippy --fix * Feat/add readme (#4) * feat: add README * feat: re-enable `secp256k1` module with updated tests * chore: fix result println * chore: update Cargo halo2_proofs_axiom to axiom/dev branch * compatibility update with `halo2_proofs_axiom` Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> * Fix: make `GateThreadBuilder` compatible with external usage (#16) * chore: expose gate_builder.unknown * feat: `GateThreadBuilder::assign_all` takes assigned_{advices,constants} as input instead of new hashmap, in case we want to constrain equalities for cells not belonging to this builder * chore: update halo2-pse tag * fix: `GateThreadBuilder::assign_all` now returns `HashMap`s of assigned cells for external equality constraints (e.g., instance cells, `AssignedCells` from chips not using halo2-lib). * fix: `assign_all` was not assigning constants as desired: it was assigning a new constant per context. This leads to confusion and possible undesired consequences down the line. * Fix: under-constrained `idx_to_indicator` (#17) *fix(BUG): `GateChip::idx_to_indicator` still had soundness bug where at index `idx` the value could be 0 or 1 (instead of only 1) * feat: add some function documentation * test(idx_to_indicator): add comprehensive tests * both positive and negative tests * Fix: soundness error in `FpChip::assert_eq` due to typo (#18) * chore: update halo2-ecc version to 0.3.0 * fix(BUG): `FpChip::assert_equal` had `a` instead of `b` typo * feat: add tests for `FpChip::assert_eq` * positive and negative tests * Remove redundant code and prevent race conditions (#19) * feat: move `GateCircuitBuilder::synthesize` to `sub_synthesize` function which also returns the assigned advices. * reduces code duplication between `GateCircuitBuilder::synthesize` and `RangeCircuitBuilder::synthesize` and also makes it easier to assign public instances elsewhere (e.g., snark-verifier) * feat: remove `Mutex` to prevent non-deterministism * In variable and fixed base `msm_par` functions, remove use of `Mutex<GateThreadBuilder>` because even the `Mutex` is not thread- safe in the sense that: if you let `Mutex` decide order that `GateThreadBuilder` is unlocked, you may still add Contexts to the builder in a non-deterministic order. * fix: `fixed_base::msm_par` loading new zeros * In `msm_par` each parallelized context was loading a new zero via `ctx.load_zero()` * This led to using more cells than the non-parallelized version * In `fixed_base_msm_in`, the if statement depending on `rayon::current_number_threads` leads to inconsistent proving keys between different machines. This has been removed and now uses a fixed number `25`. * chore: use `info!` instead of `println` for params * Allow `assign_all` also if `witness_gen_only = true` * Fix: `inner_product_left_last` size hint (#25) * Add documentation for halo2-base (#27) * adds draft documentation for range.rs * draft docs for lib.rs, utiils.rs, builder.rs * fix: add suggested doc edits for range.rs * docs: add draft documentation for flex_gate.rs * fix: range.rs doc capitalization error * fix: suggested edits for utils.rs docs * fix: resolve comments for range.rs docs * fix: resolve comments on flex_gate.rs docs * fix: resolve comments for lib.rs, util.rs docs * fix: resolve comments for builder.rs docs * chore: use `info!` instead of `println` for params * Allow `assign_all` also if `witness_gen_only = true` * Fix: `inner_product_left_last` size hint (#25) * docs: minor fixes --------- Co-authored-by: PatStiles <pdstiles78@gmail.com> * Smart Range Builder (#29) * feat: smart `RangeCircuitBuilder` Allow `RangeCircuitBuilder` to not create lookup table if it detects that there's nothing to look up. * feat: add `RangeWithInstanceCircuitBuilder` * Moved from `snark-verifier-sdk` * Also made this circuit builder smart so it doesn't load lookup table if not necessary * In particular this can also be used as a `GateWithInstanceCircuitBuilder` * chore: derive Eq for CircuitBuilderStage * fix: RangeConfig should not unwrap LOOKUP_BITS * fix: `div_mod_var` when `a_num_bits <= b_num_bits` (#31) * Feat: extend halo2 base test coverage (#35) * feat: add flex_gate_test.rs and pos add() test * feat: add pos sub() test * feat: add pos neg() test * feat: add pos mul() test * feat: add pos mul_add() test * feat: add pos mul_not() test * feat: add pos assert_bit * feat: add pos div_unsafe() test * feat: add pos assert_is_const test * feat: add pos inner_product() test * feat: add pos inner_product_left_last() test * feat: add pos inner_product_with_sums test * feat: add pos sum_products_with_coeff_and_var test * feat: add pos and() test * feat: add pos not() test * feat: add pos select() test * feat: add pos or_and() test * feat: add pos bits_to_indicator() test * feat: add pos idx_to_indicator() test * feat: add pos select_by_indicator() test * feat: add pos select_from_idx() test * feat: add pos is_zero() test * feat: add pos is_equal() test * feat: add pos num_to_bits() test * feat: add pos lagrange_eval() test * feat: add pos get_field_element() test * feat: add pos range_check() tests * feat: add pos check_less_than() test * feat: add pos check_less_than_safe() test * feat: add pos check_big_less_than_safe() test * feat: add pos is_less_than() test * feat: add pos is_less_than_safe() test * feat: add pos is_big_less_than_safe() test * feat: add pos div_mod() test * feat: add pos get_last_bit() test * feat: add pos div_mod_var() test * fix: pass slices into test functions not arrays * feat: Add pos property tests for flex_gate * feat: Add positive property tests for flex_gate * feat: add pos property tests for range_check.rs * feat: add neg pranking test for idx_to_indicator * fix: change div_mod_var test values * feat(refactor): refactor property tests * fix: fix neg test, assert_const, assert_bit * fix: failing prop tests * feat: expand negative testing is_less_than_failing * fix: Circuit overflow errors on neg tests * fix: prop_test_mul_not * fix: everything but get_last_bit & lagrange * fix: clippy * fix: set LOOKUP_BITS in range tests, make range check neg test more robust * fix: neg_prop_tests cannot prank inputs Inputs have many copy constraints; pranking initial input will cause all copy constraints to fail * fix: test_is_big_less_than_safe, 240 bits max * Didn't want to change current `is_less_than` implementation, which in order to optimize lookups for smaller bits, only works when inputs have at most `(F::CAPACITY // lookup_bits - 1) * lookup_bits` bits * fix: inline doc for lagrange_and_eval * Remove proptest for lagrange_and_eval and leave as todo * tests: add readme about serial execution --------- Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> * fix(ecdsa): allow u1*G == u2*PK case (#36) NOTE: current ecdsa requires `r, s` to be given as proper CRT integers TODO: newtypes to guard this assumption * fix: `log2_ceil(0)` should return `0` (#37) * Guard `ScalarField` byte representations to always be little-endian (#38) fix: guard `ScalarField` to be little-endian * fix: get_last_bit two errors (#39) 2 embarassing errors: * Witness gen for last bit was wrong (used xor instead of &) * `ctx.get` was called after `range_check` so it was getting the wrong cell * Add documentation for all debug_asserts (#40) feat: add documentation for all debug_asserts * fix: `FieldChip::divide` renamed `divide_unsafe` (#41) Add `divide` that checks denomintor is nonzero. Add documentation in cases where `divide_unsafe` is used. * Use new types to validate input assumptions (#43) * feat: add new types `ProperUint` and `ProperCrtUint` To guard around assumptions about big integer representations * fix: remove unused `FixedAssignedCRTInteger` * feat: use new types for bigint and field chips New types now guard for different assumptions on non-native bigint arithmetic. Distinguish between: - Overflow CRT integers - Proper BigUint with native part derived from limbs - Field elements where inequality < modulus is checked Also add type to help guard for inequality check in ec_add_unequal_strict Rust traits did not play so nicely with references, so I had to switch many functions to move inputs instead of borrow by reference. However to avoid writing `clone` everywhere, we allow conversion `From` reference to the new type via cloning. * feat: use `ProperUint` for `big_less_than` * feat(ecc): add fns for assign private witness points that constrain point to lie on curve * fix: unnecessary lifetimes * chore: remove clones * Better handling of EC point at infinity (#44) * feat: allow `msm_par` to return identity point * feat: handle point at infinity `multi_scalar_multiply` and `multi_exp_par` now handle point at infinity completely Add docs for `ec_add_unequal, ec_sub_unequal, ec_double_and_add_unequal` to specify point at infinity leads to undefined behavior * feat: use strict ec ops more often (#45) * `msm` implementations now always use `ec_{add,sub}_unequal` in strict mode for safety * Add docs to `scalar_multiply` and a flag to specify when it's safe to turn off some strict assumptions * feat: add `parallelize_in` helper function (#46) Multi-threading of witness generation is tricky because one has to ensure the circuit column assignment order stays deterministic. To ensure good developer experience / avoiding pitfalls, we provide a new helper function for this. Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> * fix: minor code quality fixes (#47) * feat: `fixed_base::msm_par` handles identity point (#48) We still require fixed base points to be non-identity, but now handle the case when scalars may be zero or the final MSM value is identity point. * chore: add assert for query_cell_at_pos (#50) * feat: add Github CI running tests (#51) * fix: ignore code block for doctest (#52) * feat: add docs and assert with non-empty array checks (#53) * Release 0.3.0 ecdsa tests (#54) * More ecdsa tests * Update mod.rs * Update tests.rs * Update ecdsa.rs * Update ecdsa.rs * Update ecdsa.rs * chore: sync with release-0.3.0 and update CI Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> * chore: fix CI cannot multi-thread tests involving lookups due to environment variables * fix: `prop_test_is_less_than_safe` (#58) This test doesn't run any prover so the input must satisfy range check assumption. More serious coverage is provided by `prop_test_neg_is_less_than_safe` * Add halo2-base readme (#66) * feat: add halo2-base readme * fix: readme formatting * fix: readme edits * fix: grammer * fix: use relative links and formatting * fix: formatting * feat: add RangeCircuitBuilder description * feat: rewording and small edits --------- Co-authored-by: PatStiles <pdstiles78@gmail.com> * fix: change all `1` to `1u64` to prevent unexpected overflow (#72) * [Fix] Panic when dealing with identity point (#71) * More ecdsa tests * Update mod.rs * Update tests.rs * Update ecdsa.rs * Update ecdsa.rs * Update ecdsa.rs * msm tests * Update mod.rs * Update msm_sum_infinity.rs * fix: ec_sub_strict was panicing when output is identity * affects the MSM functions: right now if the answer is identity, there will be a panic due to divide by 0 instead of just returning 0 * there could be a more optimal solution, but due to the traits for EccChip, we just generate a random point solely to avoid divide by 0 in the case of identity point * Fix/fb msm zero (#77) * fix: fixed_base scalar multiply for [-1]P * feat: use `multi_scalar_multiply` instead of `scalar_multiply` * to reduce code maintanence / redundancy * fix: add back scalar_multiply using any_point * feat: remove flag from variable base `scalar_multiply` * feat: add scalar multiply tests for secp256k1 * fix: variable scalar_multiply last select * Fix/msm tests output identity (#75) * fixed base msm tests for output infinity * fixed base msm tests for output infinity --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> * feat: add tests and update CI --------- Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> Co-authored-by: yulliakot <yulliakotel@gmail.com> --------- Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> * fix: redundant check in `ec_sub_unequal` * Add SafeType (#26) * Add SafeType * Refactor & add testing * Add doc comment * Refactor SafeChip * Move gen_proof/check_proof to utils * Fix merge issues * feat(CI): switch to larger runner * fix(builder): handle empty ctx with only equality constraints * feat: add SafeAddress and SafeUint160 (#85) * feat: add SafeAddress and SafeUint160 * fix incorrect byte size * Merge release v0.3.0 into develop (#90) * feat: `FpChip::range_check` now works with `max_bits < n * (k-1)` (#91) * feat(base): range_check 0 bits by asserting is zero * chore: add range_check 0 bits test * feat(ecc): `FpChip::range_check` now works with `max_bits < n * (k-1)` * Feat: test suite (#92) * feat: stop using env var to pass around FLEX_GATE_CONFIG_PARAMS and LOOKUP_BITS. Bad for testing (multi-threaded issues). Now we use thread_local to have a global static for these config params that can be passed around. * chore: make utils folder and move some files * Fix halo2 base tests naming (#76) * feat: `BaseConfig` to switch between `FlexGateConfig` and `RangeConfig` - `RangeCircuitBuilder` now uses `BaseConfig` to auto-decide whether to create lookup table or not. - In the future this should be renamed `BaseCircuitBuilder` or just `CircuitBuilder`, but for backwards compatibility we leave the name for now. - `GateCircuitBuilder` no longer implements `Circuit` because we want to switch to having everyone just use `RangeCircuitBuilder`. - Tests won't compile now because we still need to refactor * feat: refactored halo2-base tests to use new test suite * feat: remove use of env var in halo2-ecc CI now can just run `cargo test` * feat: remove use of env var from zkevm-keccak * Add zkevm-keccak test to CI * chore: fix CI * chore: add lint to CI * chore: make Baseconfig fns public * fix(test): zkevm-keccak test should have `first_pass = SKIP_FIRST_PASS` Currently with `first_pass = true`, it skips the first pass, but when feature "halo2-axiom" is used, there is only one pass of `synthesize` so the whole thing gets skipped. Mea culpa! --------- Co-authored-by: Xinding Wei <weixinding@gmail.com> * chore: make `bit_length` const function * feat: add debugging functions (#99) * feat: add debugging functions Functions only available for testing: * `ctx.debug_assert_false` for debug break point to search for other constrain failures in mock prover * `assigned_value.debug_prank(prank_value)` to prank witness values for negative tests * chore: code pretty * chore: remove cfg(test) for debug functions * feat(halo2-base): add `GateChip::pow_var` (#103) * Use halo2curves v0.4.0 and ff v0.13 (#107) * wip: change import to ff v0.13 * feat: remove `GateInstructions::get_field_element` halo2curves now has `bn256-table` which creates table of small field elements at compile time, so we should just use `F::from` always. This also improves readability. * chore: fix syntax and imports after update * chore: add asm feature * chore: workspace.resolver = 2 * chore: update ethers-core * chore: add jemallocator feature to zkevm-keccak crate * test: add bigger test case to keccak prover * feat: use `configure_with_params` remove `thread_local!` usage * chore: bump zkevm-keccak version to 0.1.1 * feat: add `GateThreadBuilder::from_stage` for convenience * chore: fixes * fix: removed `lookup_bits` from `GateThreadBuilder::config` * fix: debug_assert_false should load witness for debugging * chore: use unreachable to document that Circuit::configure is never used * chore: fix comment * feat(keccak): use configure_with_params * chore: fix halo2-pse errors * chore: change halo2_proofs to main * Add `sub_mul` to GateInstructions (#102) * Add `sub_mul` to GateInstructions * Add `sub_mul` prop test * fix(test): `select_from_idx` wasn't calling the right function (#105) * chore: add back RangeCircuitBuilder::config (#111) * fix: `RangeCircuitBuilder::config` remember `lookup_bits` * [Feat] Add Poseidon Hasher Chip (#110) * Add Poseidon chip * chore: minor fixes * test(poseidon): add compatbility tests Cherry-picked from #98 Co-authored-by: Antonio Mejías Gil <anmegi.95@gmail.com> * chore: minor refactor to more closely match snark-verifier https://github.com/axiom-crypto/snark-verifier/blob/main/snark-verifier/src/util/hash/poseidon.rs --------- Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> Co-authored-by: Antonio Mejías Gil <anmegi.95@gmail.com> * feat: add VariableByteArray (#88) * feat: add VariableByteArray * fix: correct type in panic msg * feat: make MAX_VAR_LEN const generic * feat: add `SafeBool` and `SafeByte` types These are very common so we have separate wrapper to avoid the extra length 1 vector heap allocation. * wip: add VarLenBytes * Refactor VarLenBytes Add VarLenBytesVec and FixLenBytes Fix tests * Add unsafe methods for bytes Address NITs --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> * chore: reduce CI real prover load * Rename PoseidonHasherChip to PoseidonHasher (#113) Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> * chore(safe_types): add conversion `SafeType` to/from `FixLenBytes` * chore(safe_type): add `unsafe_to_safe_type` unsafe conversion fn * feat: add `select_array_by_indicator` to `GateInstructions` (#115) feat(base): add `select_array_by_indicator` to `GateInstructions` * cleanup: use test-utils for benching (#112) * cleanup: use test-utils for benching * feat: add `{gen,check}_proof_with_instances` * feat: add initial `bench_builder` cmd to `BaseTester` * fix: cargo fmt * fix(safe_types): `VarLenBytes` should allow `len == MAX_LEN` (#117) * [feat] Add Poseidon Chip (#114) * Add Poseidon hasher * Fix test/lint * Fix nits * Fix lint * Fix nits & add comments * Add prover test * Fix CI * [chore] Reorg Folder Structure of hashes/zkevm (#118) * chore: rename crate zkevm-keccak to zkevm-hashes * fix: add `input_len` back to `KeccakTable` * chore: move keccak specific constants to `keccak_packed_multi/util` * Fix test --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> * [fix] CI for zkevm hashes (#119) Fix CI for zkevm hashes * [chore] Split keccak implementation into multiple files (#120) Split keccak implementation into multiple files * feat: keccak constant visibility changes (#121) feat: constant visibility changes * [feat] Keccak Raw Output (#122) * Replace raw keccak output instead of RLCOC * Fix lint * Add comments & improve expression performance * Virtual region managers and dynamic lookup support (#123) * feat: add `VirtualRegionManager` trait Create `CopyConstraintManager` to manage global copy constraints. * wip: separate `SinglePhaseGateManager` and `CopyConstraintManager` `GateThreadBuilder` was very messy before Todo: - Split out lookup functionality * wip: add `LookupAnyManager` * wip: `RangeChip` with `LookupAnyManager` reorg: - previous `builder/threads` moved to `flex_gate/threads` because it is all part of `FlexGateConfig` advice assignment logic - `builder` moved to `range/circuit/builder.rs` as it is part of the assignment logic of `RangeCircuitBuilder` * feat: working `BaseCircuitBuilder` backwards compatible - `GateThreadBuilder` has become `MultiPhaseCoreManager` - Some of the functionality has been moved into `BaseCircuitBuilder`, which is a generalization of `RangeCircuitBuilder` - Some fixes on virtual managers because keygen calls `synthesize` twice (once for vk, once for pk) so can't drop * fix: update halo2-ecc and sort `constant_equalities` Sort `constant_equalities` to ensure deterministism. Update `halo2-ecc` (mostly the tests) with new circuit builder format. * fix: `LookupAnyManager` drop check `Arc` strong_count * feat: add back single column lookup with selector Special case: if only single advice column that you need to lookup, you can create a selector and enable lookup on that column. This means you add 1 selector column, instead of 1 advice column. Only using this for `RangeConfig` and not generalizing it for now. * feat: add example of dynamic lookup memory table * Bump versions to 0.4.0 * chore: re-enable poseidon and safe_types `Drop` for managers no longer panics because rust `should_panic` test cannot handle non-unwinding panics. * chore: remove `row_offset` from `assigned_advices` This PR was merged: privacy-scaling-explorations/halo2#192 * chore: move `range::circuit` to `gates::circuit` * nits: address review comments * feat: add `num_instance_columns` to `BaseCircuitParams` No longer a const generic * chore(CI): use larger runner * Use `raw_assign_{advice,fixed}` in keccak (#125) * chore: switch `halo2_proofs` branch to `main` * chore: use `raw_assign_{advice,fixed}` in keccak * [feat] PoseidonHasher supports multiple inputs in compact format (#127) * PoseidonHasher supports multiple inputs in compact format * Add comments * Remove unnecessary uses * [feat] Expose Keccack Raw Inputs in Bytes instead of Input RLCs (#124) * Expose Keccack raw inputs in bytes instead of input RLCs * Fix column name in comments * Add comments * Compress 8 bytes of inputs into a single witness * chore: add some comments * Rewrite gates * Fix comments & typos * Fix naming * Add comments * Selector improvement * Remove unused --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> * Bump `zkevm-hashes` to v0.1.4 * chore: clippy fix * Generic vertical gate assignment (#129) * feat: make `single_phase::assign_with_constraints` generic Use const generic for max rotations accessed by the vertical gate. This way we can re-use the code for RLC gate. * chore: make single_phase pub * feat: add safety check for overlapping gates * Add `deep_clone` to `BaseCircuitBuilder` (#131) * chore: add convenience function to `BaseConfig` * feat: add `deep_clone` to `BaseCircuitBuilder` We sometimes want to clone `BaseCircuitBuilder` completely (for example to re-run witness generation). The derived clone only clones the shared references, instead of the underlying objects. * fix: `SingleCorePhaseManager` should not create thread in constructor Because the thread will default to phase 0. * chore: make `new_context` public * Convenience functions and fixes for multi-phase (#133) * feat: add `clear` function to circuit builder and managers * feat: add `BaseConfig::initialize` * fix: break points for multiphase * fix: clear should not change phase * chore: remove dbg * chore: add `BaseCircuitBuilder::set_k` fn * fix: `CopyConstraintManager::clear` was dropping * feat: impl `From<SafeByte>` for `AssignedValue` * chore(poseidon): add `derive` statements * fix(copy_constraints): backend permutation argument depends on order Backend implementation of `constrain_equal` depends on the order in which you add equality constraints, so it is not thread-safe... * feat: add `left_pad` functions for var length arrays (#137) * chore: use `PrimeField` for `OptimizedPoseidonSpec` (#139) * chore: add getter functions to Poseidon spec (#140) * feat: use `(TypeId, usize)` instead of `usize` for lookup tag (#142) * chore: add `ContextTag` type alias * feat(base): add `GateInstructions::inner_product_left` function (#143) * chore: fix comments * feat(base): add `GateInstructions::inner_product_left` function * [feat] Keccak Coprocessor Leaf Circuit (#130) * WIP * chore: make `KeccakAssignedRow` fields public * Refactor Keccak coprocessor circuit * Optimize Keccak circuit MAX_DEGREE * Fix comments * Fix bug & typos * Add testing & refactor folder structure * Debugging * Fix bugs * Fix typo & bug * feat(test): real prover tests use dummy input for keygen * chore: make `LoadedKeccakF` public * Also made `encoded_inputs_from_keccak_fs` public * Both are useful for external use to make lookup tables in app circuits * fix(keccak_leaf): review comments and optimization * chore: use `gate` when `range` not necessary * Move calculate base ciruit params out & Fix naming/comments * Make Field-related parameter functions const * feat: change `is_final` to `SafeBool` * nit * Fix typo --------- Co-authored-by: Jonathan Wang <31040440+jonathanpwang@users.noreply.github.com> * [feat] App Circuit Utils for Keccak Coprocessor (#141) * Add keccak coprocessor encoding for VarLenBytesVec/FixLenBytesVec * Fix naming/nits * Fix nit * [chore] Fix fmt (#144) Fix fmt * chore: add misc utility functions (#146) * chore(keccak_leaf): make `generate_circuit_final_outputs` public * chore: add misc utility functions * feat(keccak): add `ingestion` module for Rust native input formatting (#147) * chore(keccak): use `snark-verifier` native Poseidon for encoding (#148) Currently only used for testing * feat: optimize leaf poseidon with `hash_compact_chunk_inputs` (#149) * [chore] cleanup code (#150) chore: cleanup code * chore: get halo2-pse working again * chore: fix fmt * [Doc] Keccak Doc (#145) * Keccak docs * Fix typos * Add examples * Fix comments/docs * chore: pin snark-verifier branch * [fix] max_rows in BaseCircuitBuilder in Keccak Leaf Circuit (#152) Fix max_rows in BaseCircuitBuilder inside Keccak Vanilla * [chore] Remove Unnecessary Lookup in Keccak Coprocessor Leaf Circuit (#153) * chore: fix fmt * [fix] max_rows in BaseCircuitBuilder in Keccak Leaf Circuit (#152) Fix max_rows in BaseCircuitBuilder inside Keccak Vanilla * Remove lookup in Keccak Leaf circuit * [rename] (coprocessor, leaf) -> (component, shard) (#161) * [fix] Multiple Phase Lookup (#162) Fix multiple phase lookup * [chore] add conversion `SafePrimitive` to `QuantumCell::Existing` (#169) chore: add conversion `SafePrimitive` to `QuantumCell::Existing` * fix: bad import on halo2-pse * chore: make `{Fixed,Var}LenBytes*` constructor public It's convenient to be able to construct the structs from vectors of safe bytes externally. Only unsafe-ness is `len <= max_len` is not checked. * [chore] add `cargo audit` to CI (#207) chore: add `cargo audit` to CI Upgrade criterion version * [fix] `FieldChip::range_check` should take `FieldPoint` instead of `UnsafeFieldPoint` (#209) * fix: `FieldChip::range_check` should take `FieldPoint` instead of `UnsafeFieldPoint` * chore: fix audit-check CI * chore: toggle CI on release branches * [feat] update docs (#211) * feat: update doc comments with function assumptions * feat: update readme * chore: fix CI * [chore] fix dev graph tests (#212) * chore: CI uses clippy all-targets * fix: dev-graph tests (only works for halo2-pse) Didn't bother refactoring halo2-axiom to support dev-graph * [feat] add keccak circuit tests against Known Answer Test vectors (#213) feat: add keccak circuit tests against Known Answer Test vectors * [chore] fix documentation (#215) * chore: fix keccak comment * chore: remove redundant * chore: fix test case description * chore: fix documentation * chore: add comment * [chore] fix doc comment (#216) chore: fix doc comment * fix: `TypeId` in `ContextTag` not stable across builds (#217) * fix: use &str instead of TypeId in ContextTag * chore: add warning to readme * chore: fix comment * [chore] add crate prefix to `type_id`s (#218) * chore: add crate prefix to `type_id`s * fix: module_path! url * chore: add type_id warning to `Context::new` and `ContextCell::new` * chore: use halo2-axiom from crates.io * chore: use poseidon-primitives from crates.io --------- Co-authored-by: Jonathan Wang <jonathanpwang@users.noreply.github.com> Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de> Co-authored-by: PatStiles <pdstiles78@gmail.com> Co-authored-by: PatStiles <33334338+PatStiles@users.noreply.github.com> Co-authored-by: yulliakot <yulliakotel@gmail.com> Co-authored-by: yuliakot <93175658+yuliakot@users.noreply.github.com> Co-authored-by: Xinding Wei <weixinding@gmail.com> Co-authored-by: mmagician <marcin.gorny.94@protonmail.com> Co-authored-by: Xinding Wei <xinding@intrinsictech.xyz> Co-authored-by: Antonio Mejías Gil <anmegi.95@gmail.com> Co-authored-by: MonkeyKing-1 <67293785+MonkeyKing-1@users.noreply.github.com>

cleanup: use test-utils for benching

a22f450

jonathanpwang force-pushed the feat/test-suite-2 branch from aae800d to a22f450 Compare August 22, 2023 00:50

jonathanpwang added 3 commits August 21, 2023 17:59

feat: add {gen,check}_proof_with_instances

1b41b14

feat: add initial bench_builder cmd to BaseTester

35ef1db

fix: cargo fmt

2de05b4

jonathanpwang merged commit 1c33fbc into develop Aug 22, 2023
1 of 2 checks passed

jonathanpwang deleted the feat/test-suite-2 branch August 22, 2023 01:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cleanup: use test-utils for benching #112

cleanup: use test-utils for benching #112

jonathanpwang commented Aug 17, 2023 •

edited

Loading

cleanup: use test-utils for benching #112

cleanup: use test-utils for benching #112

Conversation

jonathanpwang commented Aug 17, 2023 • edited Loading

jonathanpwang commented Aug 17, 2023 •

edited

Loading