[Stress Testing] - Runners

[noklam]

Description

The goal for this is test different runners,

In Kedro, there are 3 runners:

• SequentialRunner
• ThreadRunner (I/O bound, usually used for Spark pipeline)
• ParallelRunner (CPU bound)

The test should measure:

• Time
• Memory (Memory Profiling)

The test should be target for the runner, for example we need to test I/O heavy pipeline for ThreadRunner

Context

#3957 (comment)

Component stress test:

• The main goal for this is to benchmark performance of individual component, this will inform if refactoring work has positive/negative impact. Currently we only check if test pass, so we have no idea if a change may slow down performance. We have done this in the past but usually ad-hoc basis, we should run this regularly (or at least per release).

The direction of this is simple, we want to make measure the change of time against # number of entries. We would start with Datasets and Catalog, as this fits in the DataCatalog2.0 work and will be immediately useful.

This can address:

• Potentially inefficient memory usage #4111

Should work with the setup decided in #4128