Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pyspark to add-ons options #3073

Closed
merelcht opened this issue Sep 25, 2023 · 1 comment · Fixed by #3169
Closed

Add pyspark to add-ons options #3073

merelcht opened this issue Sep 25, 2023 · 1 comment · Fixed by #3169
Assignees
Labels
Component: CLI Issue/PR that addresses the CLI for Kedro

Comments

@merelcht
Copy link
Member

Description

pyspark should be added to the add-ons options, allowing users to get setup for pyspark (see #2506 (comment)). This option should make use of the new spaceflights-pyspark starter: #2984

When a user select pyspark they should get the config needed to run Kedro project with pyspark. This shouldn't include the example pipelines. This will happen in a follow up ticket.

Possible Implementation

This solution is likely going to be the same as for the other add-ons work, leveraging the post-project hooks. Since we'd use the spaceflights-pyspark starter and not the regular template, we probably need to copy some of the logic from the post-project hook in the regular template to the starter.

Possible Alternatives

N/A

@SajidAlamQB
Copy link
Contributor

Completed in: #3169

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: CLI Issue/PR that addresses the CLI for Kedro
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants