Consistent e2e testing

[mzabaluev]

We need a consistent suite of e2e tests that are run by the CI.

Workspace organization

Expectation: The e2e tests should be cleanly separated from other tests (unit or "light/minimal integration" for library crates) and have a prominent location in the workspace.

Current state: The e2e tests that have been merged to main are found as module and binary grafts on the (possibly misnamed) suzuka-client crate.

If the crate currently serves no other purpose (the only other code there is a re-export of aptos-sdk), rename it to suzuka-e2e-tests.

Organization of the test suite

The tests should be organized under a uniform structure, preferably plugging into the cargo test harness: either integration-style test binaries under ./tests, or modules under ./src/. The test suite should not be a hodgepodge of non-enumerable binaries in ./bin, ./src/bin, etc. and unit tests.

The load and soak test cases should be grouped so as to be easily invoked as the respective group. Alternatives:

• Top-level modules in the library crate, giving load::* and soak::*.
• Dedicated test binaries: ./tests/load-tests and ./tests/soak-tests.

Run time requirements

The e2e tests for Movement are expected to be run in an environment that has access to DA and Eth testnets, and has the services built from the same workspace running and available on the virtual network.

The Rust crate hosting the tests is not responsible for setting up the test environment. By the time cargo test -p suzuka-e2e-tests is run, all services should be provisioned into containers and deployed by docker-compose.

Technical details of the test suite

Decide how we want to run the e2e tests:

1. Use Rust built-in harness for #[test] cases.

• Infrastructure setup is either left entirely to outside tools such as docker-compose, or copied as boilerplate.
• Third-party utility macros could be used to facilitate custom setup.

2. Provide a custom test harness.

• Can use custom setup.
• Still get to organize tests as #[test] fns.
• Requires nightly, which is anathema for Nix?

4. A binary, with DIY test organization.

• Can do in stable Rust.
• Any amount of customization we want.
• No #[test] support, tedious.

Test scenarios

• Happy path: submit transactions, block received from DA, finalized by settlement.
• Rejection of invalid transactions.
  • Which cases of invalid to simulate in e2e?
• Settlement rejection.

[l-monninger]

A binary, with DIY test organization.
Can do in stable Rust.
Any amount of customization we want.
No #[test] support, tedious.

Let's go for this. You can always define a method and then trivial wrap it in something with a #[test] macro elsewhere.

[l-monninger]

@mzabaluev Can you list the tests we need to split out here?

[mzabaluev]

Looking at the contents of suzuka-client in the main branch, I can see:

• The standalone binaries in ./bin.
• The simple_interaction binary, placed into ./src/bin/e2e as if only to offer some variety.
• The load_soak_testing module with a DIY test running framework.
• The tests module and its submodules. Some cases are implemented as Scenario trait impls that load-and-soak can hook into. Others are just #[test] cases.

This looks like our start at an e2e test suite, but we need to do some grooming. The main principle, as I see it now, is that you can run most of the tests as cargo test (i.e. using the default harness). The standalone binaries should be reworked under the general ./src/tests umbrella. The simple_interaction binary looks simple enough to be moved to ./tests, so it's automatically built and run by cargo test as well.

[mzabaluev]

Upon some consideration, I'd go for approach 1. Any setup can be managed by setup crates/scripts, like @musitdev managed to do for the load-and-soak tests. I have updated the note with the run time requirements.

We already have the full node entry points exposed in lib.rs; we can move even more code into the library from main.rs to be able to run the service from test functions with negligible difference to testability. This will allow rebuilding tests with normal dependency management in the workspace, without reliance on the binary being present.

[musitdev]

I agree that suzuka-client to host every test is not clear. Most e2e test are testing a protocol and not how a specific node or entity is working. So they can be put in the same place less dependent from node implementation with a better spelling.

Effectively, e2e test, need to have a pre started environment. Whereas the actual way to do, this environment that is started using a setup, depends on the e2e test or protocol to test. So this setup should be defined with the test and not as it's currently done with the subcomponent (suzuka node, mcr, executor). The e2e crate should provide some way to reuse this setup routine.
I think it's the same for load and soak test. Some part of the test should be share.

I agree with @l-monninger, the 4th option, use a binary is the most open way to define e2e test.

e2e test workflow is run build, run the e2e test setup, start the environment to test, run the e2e test.

[mzabaluev]

use a binary is the most open way to define e2e test.

Reliance on the bootstrapped test environment means there is no need to perform infrastructure setup for each test case.

Are there still benefits in not using the standard test harness? Surely, internally you have the Scenario framework, but you call into that from #[test] entry points. I think this is a workable approach.