Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancing Kedro-Viz Performance with Lazy Loading #1806

Open
1 task done
rashidakanchwala opened this issue Mar 15, 2024 · 0 comments
Open
1 task done

Enhancing Kedro-Viz Performance with Lazy Loading #1806

rashidakanchwala opened this issue Mar 15, 2024 · 0 comments
Labels
Issue: Feature Request Python Pull requests that update Python code

Comments

@rashidakanchwala
Copy link
Contributor

rashidakanchwala commented Mar 15, 2024

Description

The current loading strategy for Kedro-Viz can lead to significant delays. It requires all data, including session information like the catalog and session_store for experiment tracking, to be fully loaded before the visualization is rendered. This process can slow down the display of the Kedro-Viz flowchart, which only needs the quicker-to-load pipeline_data for initial rendering.

To improve the performance, we need to -

  1. Refactor Kedro-Viz CLI - Refactor CLI with lazy subcommands and deferring imports #1920
  2. Refactor Kedro-Viz FAST API app creation - Refactor Kedro-Viz FAST API app creation #2061
  3. Asynchronously load pipeline data - I tested this while profiling work and created a test branch with some changes in server.py - https://github.com/kedro-org/kedro-viz/blob/test/large-pipeline-profiling/package/kedro_viz/server.py . I did not see a huge improvement in load time, but we can take this up as a minor improvement. I will create a ticket if needed. cc: @rashidakanchwala

Testing the performance improvements -

  1. Test against a massive pipeline once created - Create massive pipeline to test with flowchart on Kedro-viz  #1064
  2. Test against a complex pipeline once created - [Stress Testing] - Create example projects to assess Kedro performance for complex pipelines kedro#3866

Checklist

  • Include labels so that we can categorise your feature request
@rashidakanchwala rashidakanchwala added Issue: Feature Request Python Pull requests that update Python code labels Mar 15, 2024
@rashidakanchwala rashidakanchwala changed the title Implementing Lazy Loading in Kedro-Viz for Enhanced Performance Enhancing Kedro-Viz Performance with Lazy Loading Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue: Feature Request Python Pull requests that update Python code
Projects
Status: Todo
Development

No branches or pull requests

1 participant