You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There are several features across the Kedro organisation that could benefit from manual testing on large projects to evaluate performance. The proposal is to create several kedro projects of varying size that can be used to test and experiment with.
The example projects don't have to be "legit" as in tackling real data science problems. It can be toy examples. The main point here is creating projects with scale.
The expensive operation before starting the viz server is loading the data from the Kedro session (possibly related to Lazy Loading of Catalog Items kedro#2829 ?)
Most of the time taken to load the data is from catalog and pipelines_dict resolution, which worsens as the pipeline count increases
merelcht
changed the title
Assess Kedro performance for complex pipelines
Create example projects to assess Kedro performance for complex pipelines
Jun 17, 2024
noklam
changed the title
Create example projects to assess Kedro performance for complex pipelines
[Stress Testing] - Create example projects to assess Kedro performance for complex pipelines
Aug 29, 2024
Since this is in sprint already, I update the title so this will be the ticket for pipeline stress testing. I will create separate tickets for the component part.
Description
#3957 needs to be done first as pre-work.
There are several features across the Kedro organisation that could benefit from manual testing on large projects to evaluate performance. The proposal is to create several kedro projects of varying size that can be used to test and experiment with.
The example projects don't have to be "legit" as in tackling real data science problems. It can be toy examples. The main point here is creating projects with scale.
This could be particularly useful for testing Viz features, CC @rashidakanchwala, @NeroOkwa
Context
The Kedro-Viz team carried out a performance analysis using an internal QB pipeline, with preliminary results shown here kedro-org/kedro-viz#1064
(from kedro-org/kedro-viz#1064 (comment), summary of internal report).
There is preliminary evidence that the Kedro Framework CLI is a bottleneck for Kedro Viz.
This is on top of the already existing evidence that Kedro takes a lot of time to load even for trivial commands or almost empty projects #1476
We noted that there are several factors that make a pipeline "complex":
In I expanded on @AhdraMeraliQB's original proposal and suggested that we create a family of pipelines, comprising
Comes from #3790
Originally posted by AhdraMeraliQB January 6, 2024
The text was updated successfully, but these errors were encountered: