Add PySpark to the add-ons flow

[SajidAlamQB]

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

PySpark should be added to the add-ons options, allowing users to get setup for pyspark (see #2506 (comment)). This option should make use of the new spaceflights-pyspark starter: #2984

When a user select pyspark they should get the config needed to run Kedro project with pyspark. This shouldn't include the example pipelines. This will happen in a follow up ticket.

Development notes

Blocked by:
kedro-org/kedro-starters#170

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[merelcht]

The general approach looks good, but instead of adding the spark specific files in the basic template, these should be fetched from the new starter: https://github.com/kedro-org/kedro-starters/tree/main/spaceflights-pyspark. You might have to also create a post_gen_project file there..

[noklam]

I see all tests are passing but linting, I take the liberty to fix it.

[SajidAlamQB]

I see all tests are passing but linting, I take the liberty to fix it.

lol we did it at the same time.

[merelcht]

If you combine pyspark with any of the other add-ons you potentially get duplicate entries in your pyproject.toml, because the pyproject.toml in the starter already contains all settings for e.g. docs, linting etc. I guess the pyproject.toml will need to be stripped.

Not related to just pyspark: I also noticed that docs requirements are already present in the base pyproject.toml and will also end up being duplicated even if pyspark isn't selected.

[merelcht]

Yeah part of the stuff was stripped from the pyproject.toml in the base template in https://github.com/kedro-org/kedro/pull/2987/files, but the starters were merged first so I didn't realise this had to happen.. 😅

[SajidAlamQB]

If you combine pyspark with any of the other add-ons you potentially get duplicate entries in your pyproject.toml, because the pyproject.toml in the starter already contains all settings for e.g. docs, linting etc. I guess the pyproject.toml will need to be stripped.

We could just add checks to make sure that the entries we're adding are not already present.

[merelcht]

If you combine pyspark with any of the other add-ons you potentially get duplicate entries in your pyproject.toml, because the pyproject.toml in the starter already contains all settings for e.g. docs, linting etc. I guess the pyproject.toml will need to be stripped.

We could just add checks to make sure that the entries we're adding are not already present.

But these optional requirements should only be added if the user wants them. If they haven't selected lint they shouldn't get any requirements/settings in their pyproject.toml related to ruff.

[noklam]

Our tests should have caught this - I will make a separate PR to fix the test. Essentially I think we need to parse the pyproject.toml properly instead of reading it as text file.

        with open(pyproject_file_path) as pyproject_file:
            requirements = pyproject_file.read()

[noklam]

I created the test in #3230, it catches an invalid combination that we need to fix.

[merelcht]

I've left some minor comments but I think this is almost ready to be merged. I suggest fixing the other issues I found in a separate PR, because they aren't caused by the changes here and that shouldn't block the work for adding the viz option.

[SajidAlamQB]

I've left some minor comments but I think this is almost ready to be merged. I suggest fixing the other issues I found in a separate PR, because they aren't caused by the changes here and that shouldn't block the work for adding the viz option.

Oh, I've already added some changes that attempt to fix this but I can revert and open it up on new PR? @merelcht

[merelcht]

Oh, I've already added some changes that attempt to fix this but I can revert and open it up on new PR? @merelcht

Ah no, that's great!! I'll have a look 😄 I just didn't want to block this PR by my earlier comments.

[merelcht]

Left one more comment about the pyspark check. add_ons is a list there so == will always fail. But otherwise this looks good to be merged! 👍