Design proposal for testing framework for Engine compliance with Ethereum Test Suite/Bundle and the Ethereum Yellow Paper

[andrcmdr]

Design proposal for testing framework for Engine compliance with Ethereum Test Suite/Bundle and the Ethereum Yellow Paper

Summary:

Objectives and motivation:

Making of a framework for testing of Aurora Engine to be fully compliant with Ethereum Test Suite and thus Ethereum Yellow Paper, as part of a product Quality Assurance process.

Related to and covers the issue #170

Design:

Abstract:

Implementation of a testing framework should include following:

• a standard Ethereum's tests bundle (made for testing of clients and nodes to ensure compatibility and compliance with Ethereum)
• a tests parser, which would transform a stream of definitions and data from json files to data types usable from Rust code - this can be done by usage of ethjson parser and json_tests helper (formerly this was part of ethjson, but now this helpers is part of ethcore), this is reasonable, 'cause corresponded to DRY and its code an actively developed and supported by openethereum project
• a glue layer for mapping parsed data types (unwrapping) into internal engine types and abstractions (initialization and filling of internal types) for running engine with tests execution
• implement standard testing suite/bundle for the Engine, as usual test sections and assertions in a test files (as showed here), using glue layer - this is for first class support of cargo test and it's usage in CI pipeline and current makefile approach for building process
• CLI app for convenient usage of testing framework as an end-user tool in a CI pipeline and in a local development environment - this would be good to have, but it's optional

Architecture:

• tests base layer - Ethereum tests bundle in json/yaml format
• representation layer - json/yaml data parser, using ethjson json spec parser and serde
• helpers - vm wrapper (as more high level VM interface for testing purposes), json_tests from ethcore as spec parsing helper
• glue layer, mapper/filler - mapping of parsed data to Aurora Engine internal types and abstractions, can be formed as unwrapping functions (to unwrap data from tests to corresponding internal data types and fill data structures)
• standard testing suite/bundle for the Engine, as usual test files, for first class support of cargo test
• clap or abscissa based CLI app, for convenient usage of testing framework as an end-user tool (test utility), for manually run tests for various Engine subsystems, which can be useful in local environment in development process of various subsystems, to run and for test framework usage, and for CI pipelines improvement
  https://crates.io/crates/clap
  https://crates.io/crates/abscissa
  https://github.com/iqlusioninc/abscissa

Implementation steps:

First of all - making a proof of concept, walking through the following steps, to feel out any caveats and pitfalls, and improve these steps, to implement testing framework:

0. base layer: including dependencies, using cargo abilities or as git submodules
1. representation layer: json data parser, corresponding to json specs - could be done with usage of ethjson, following DRY, but also be implemented as own parser (takes more efforts). On this step we would have a data from json tests represented as convenient Rust data types.
2. glue layer: data types mapper/filler, to unwrap parsed data types and map/fill them into internal types and abstractions of the Engine.
3. form Ethereum compliance testing suite/bundle for the Engine: implementation through standard test files, to provide first class support of cargo test, as we have here
4. CLI App for convenient testing and usage in CI pipeline (optional, not a high priority)

Tests from Ethereum tests bundle to be used:

• Blockchain tests
• Transaction tests
• State transition tests (many of tests already done as part of VM tests)
• VM tests - already implemented, [1], [2], and implementation would be included in a testing framework
• Contract tests
• EIP tests
• Network hard-forks compatibility tests

Tests from Ethereum tests bundle, that won't be used, 'cause not applied for Engine (as a WASM contract for NEAR) or for Standalone Engine conceptions:

• Trie tests (world state trie, transaction trie, transaction receipts trie)
• RLP tests (for accounts and account storage contents trie)
• Encoding tests
• Cryptographic tests
• ABI tests
• Genesis tests (for genesis block)
• Difficulty tests (for blocks producing in a PoW consensus based networks)

Tests format:

Mainly tests are represented as json key/value definitions of data, the key is represent of a meta-information about data in value field, and value field is a data payload itself, and this payload data can be represented in a format of a block, transaction, transaction receipt, contract account, account storage contents trie, etc.

The particular formats are described in documentation and json schemas:

• tests structure in documentation:
  https://ethereum-tests.readthedocs.io/en/latest/test_filler/blockchain_filler.html#test-structure
  https://ethereum-tests.readthedocs.io/en/latest/test_types/blockchain_tests.html#test-structure
  https://ethereum-tests.readthedocs.io/en/latest/test_filler/state_filler.html#test-structure
  https://ethereum-tests.readthedocs.io/en/latest/test_types/gstate_tests.html#test-structure
  https://ethereum-tests.readthedocs.io/en/latest/test_types/transaction_tests.html#test-structure

• JSON schemas for data types and definitions, used in tests (block-chain, state transition and vm tests):
  https://github.com/ethereum/tests/tree/develop/JSONSchema

Repository structure:

Repository will include git submodule for tests, and maybe a git submodule for usage openethereum's ethjson and json_tests as a libraries, for usage as testing helpers, if these crates cannot be included in other way with using declaration of dependency in a cargo toml file.

Dependencies:

• ethjson json spec parser
• json_tests helpers

License compatibility:

• Tests - licensed under MIT
• Parser and helpers - licensed under GPLv3 as part of openethereum node, crates isn't provided through crates.io and due to licensing under GPL can be used only as external dependency, as external library, through provided API, as git submodule or using cargo capabilities of including crates from subdirectories.

[joshuajbouw]

• Blockchain tests

Are you sure we can use the blockchain tests?

CLI App for convenient testing and usage in CI pipeline (optional, not a high priority)

We shouldn't go down this route. Just simply running tests in the CI.

Network hard-forks compatibility tests

From my experience, I'm unsure if this is needed since each ethjson test has each hardfork tests in them.

Encoding tests should be included, RLP too. Sure, we are using an external library, but redundancy sakes, it's a good idea.

The architecture is sound, except the CLI part.

Thanks for the license compatibility check. It is clear that the tests library would need to be GPLv3.

Do note that we need to get started on this as soon as possible. I know that you are caught up with NATS, and we can discuss the strategy for next week. I suspect that I will get the initial framework out there though, then we can build upon that.

[andrcmdr]

@birchmd Some of these things we're discussing previously.

[joshuajbouw]

Ok so I did a bit of digging this morning into other ethjson alternatives... There are two paths ultimately that we could take. I will say that I am highly in favour of option 1. I want to use GPLv3 less, as much as possible. We can also skip quite a bit of code and maintenance. On top of that, we can include it right into our tests and don't require yet-another-crate.

Option 1

We simply skip ETH Json parsing, and instead manually add it all as tests. This is the approach that evmodin had taken, and I generally like it.

Pros

• Don't need to import a GPLv3 test.
• Skips writing a lot of glue entirely.
• Don't need to bring in a JSON parser
• More control
• Don't need yet-another-crate

Cons

• Need to ensure that we are still on par with ETH tests.
• Could potentially make a mistake when translating the tests, however unlikely that we would even get a correct result.
• Most likely will actually take more time to implement

Option 2

Write the ETH json parser as described in this document.

Pros

• Using official ETH Json tests
• We can ensure we are not missing anything
• Adding new tests would be almost a breeze

Cons

• License
• Lots of extra code needs to be written and interpreted to our EVM types

[birchmd]

Writing hundreds of tests by hand and committing to writing all the new ones that come out in the future sounds like much more work to me than writing one parser to consume the existing JSON. Our aurora-engine-tests crate is already GPLv3 because it depends on NEAR, so we could use the ETH tests in that crate no problem.

[andrcmdr]

Both approaches are good. I've already think about this while working on this doc. I like Option 1 more, 'cause this gives us more flexibility, versatility and independence/autonomy from external deps, 'cause the tests format already described well in docs (pointed in a doc above) and won't be changed a lot with introduction of new tests (as we saw this already happen at the first quarter of this year - tests repo layout changed, Eth authors started to migrate all tests to a new repo, new tests appeared as well, but the formats remained the same, and that's why for now we actually have tests repo, and legacy tests repo and only Constantinople hard-fork tests remained there), and most of the clients (Geth and other projects) already had their own representation layer with just parsing JSON tests according to formats, described in documentation and JSON schemas.
We can follow both approaches gradually - first for correctness and quick implementation follow Option 2, and then improve framework with our own data representation layer, just using serde and eliminate the need of ethjson and all openethereum project dependencies at all.

[joshuajbouw]

Yeah forgot to mention a couple of things post me writing the above.

• There is a LOT of tests which would be a nightmare to write out by hand.
• Yes, test crate already is GPLv3. For whatever reason, must've looked at the wrong crate.
• Option 2 by far, is faster