Testing Methodology

[mzabaluev]

Discuss our approach to testing.

Short term: essential coverage

Tests described in this section must be implemented to consider our implementation initially workable.

End-to-end testing

Test scenarios

• Happy path: submit transactions, block received from DA, finalized by settlement.
• Rejection of invalid transactions.
  • Which kinds of invalid to simulate in e2e?
• Settlement rejection.

Load and soak testing

Define the methodology used here.

• Definitions of load and soak, target performance.
• What scenarios to test, besides a load of valid transactions: a mix of valid/invalid?
• Simulation of working in a network?
• Simulate realistic load on interrogational endpoints, state sync etc., concurrent with transactions.
  • A known likely pain point: load on submit_transaction vs. get_transaction_by_hash.

Unit testing

Written to support development, mainly in protocol units and utility crates.
May include integration-style tests, requiring local/remote setup to run, but mocking should be preferred.

• No goal to reach comprehensive coverage, but essential use cases need to be tested.
• Regression testing normally goes here.
• Property testing is more practical on level of individual crates.

Medium-term plans

Test coverage

Coverage can be used as KPI.

Decide what we want to collect coverage on: e2e only or including the unit tests? There are tradeoffs.

• Coverage of E2E only:

  • ➕ Shows the code you can really reach in realistic scenarios.
  • ➖ Simulation of some conditions could be difficult.

• Coverage of unit tests:

  • ➖ Potential for testing dead code and irrelevant scenarios, considered good by the coverage metric.
  • ➕ Easier to reach unexpected conditions.

Long shot

Fuzzing

What kinds of fuzzing?

• Trivial fuzzing of payloads is easier to implement, but most of the invalidation cases are covered by dependencies: gRPC, serialization formats, etc.
• Data domain fuzzing (transaction fields, etc.) requires more intelligent generation of messages that pass validation on protocol and encoding levels.

[l-monninger]

Load and soak testing

I think the main thing that needs to happen with load and soak testing is its application to individual units as opposed to just the e2e which currently include several services.

[mzabaluev]

I consider e2e in a wider sense, as the test setup in which the full node runs under test and interfaces with real-world decentralized services such as Celestia and ETH testnets.

Since load and soak is a performance test, the minimal object under test should be the integrated node. Or do you think there is value in L&S testing unit libraries in partially mocked setups as well?

[l-monninger]

Define the methodology used here.

Are there starting points for a methodology you want to suggest? @musitdev 's framework already encodes a fairly opinionated one. Are we considering implementing tests that explore a test space--i.e., evolutionary testing or randomized testing?

[mzabaluev]

@musitdev 's framework already encodes a fairly opinionated one.

1. Is it fully present in main?
2. Is there a document describing the current approach to load and soak?

[mzabaluev]

I have checked what's available in main. I think some items to start with will be:

• Reorganize the tests from a collection of module and binary grafts on the suzuka-client crate (which is, besides that, just pub use aptos_sdk::*;)
• Put out a README describing the load and soak approach.

[l-monninger]

No goal to reach comprehensive coverage, but essential use cases need to be tested.

I think a good small task we could split out immediately is reporting some coverage. Let's open up another discussion for this.

[mzabaluev]

Coverage should be covered by #396.

[l-monninger]

May include integration-style tests, requiring local/remote setup to run, but mocking should be preferred.

I would say we throw anything that requires setup (beyond the development environment provided by the nix shell), should be throw behind an e2e flag. The distinction between integration and unit should mainly be conceptual.

The separation of e2e and unit tests in code is another more discrete task we can immediately split out. It's also something we've put off or half-baked several times. I say we open up another discussion and try to get parts of this assigned and down to issues ASAP (EOD).

[mzabaluev]

Discussion on e2e testing is continued under #395.

[l-monninger]

Data domain fuzzing (transaction fields, etc.) requires more intelligent generation of messages that pass validation on protocol and encoding levels.

I do think we should lead with data domain fuzzing. I've worked on fuzzing setups before, so I can help here. But, this is a long term project and definitely another discussion.

[mzabaluev]

Taken out into #397.