You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The new find_pipelines feature is really great and works very well in simple cases.
I would like to propose to extend the functionality so that it also works for nested structure
Context
In most of our kedro projects, we resort to creating subfolders of the src/pipelines folder, for ex
This is incompatible with find_pipelines which means we have to register the pipelines manually. It works but I feel like this is extra work that could be avoided.
Possible Implementation
Fortunately I think it is easy to implement. We just need to replace the following lines
forpipeline_dirinpipelines_package.glob("**/"): # navigates dirs and subdirsifnotpipeline_dir.is_dir():
continue# new check, we want to ignore folders that don't include a pipeline.py fileifnot (pipeline_dir/"pipeline.py").is_file(): # maybe it is better to check for __init__.pycontinue
...
# keep pycache check
...
pipeline_relative_path=pipeline_dir.relative_to(pipelines_package) # path/to/dirfull_pipeline_name=os.path.normpath(pipeline_relative_path).replace(os.path.sep, '.') #path.to.dir, also works with windows pathpipeline_module_name=f"{PACKAGE_NAME}.pipelines.{full_pipeline_name}"# same as before
...
If you are interested in this change, I can create a pull request.
If you think find_pipelines should not support nested folder structure, I am also interested in your arguments.
Thanks a lot.
The text was updated successfully, but these errors were encountered:
Thanks for this idea @Gabriel2409 ! I think this makes a ton of sense. We are focusing on getting the 0.19 release out of the door so it's unlikely we look at this in the next month or so, but we'll make sure we consider it.
Hi @astrojuanlu thank you for your reply.
I understand that this is not an urgent change.
Still I added a draft PR as a support for the discussion. Do not hesitate to add comments if and when you have the time.
Description
The new
find_pipelines
feature is really great and works very well in simple cases.I would like to propose to extend the functionality so that it also works for nested structure
Context
In most of our kedro projects, we resort to creating subfolders of the src/pipelines folder, for ex
This is incompatible with find_pipelines which means we have to register the pipelines manually. It works but I feel like this is extra work that could be avoided.
Possible Implementation
Fortunately I think it is easy to implement. We just need to replace the following lines
https://github.com/kedro-org/kedro/blob/aa3e2106f1491e83004a19a87f1d4e839476a33a/kedro/framework/project/__init__.py#L347C5-L355C75
with something like:
If you are interested in this change, I can create a pull request.
If you think find_pipelines should not support nested folder structure, I am also interested in your arguments.
Thanks a lot.
The text was updated successfully, but these errors were encountered: