Use cookiecutter hooks to strip-out not selected "add-ons" from base template

[merelcht]

Description

Follow up on #2758

Context

Depending on the utilities/add-ons a user selects from the following:

1. testing
2. linting
3. logging
4. data structure
5. documentation

We need to add in certain directories/files in a new project when the user executes kedro new without starter.

Proposed Implementation

Use cookiecutter hooks to remove and update files/directories depending on the add-ons selection: https://cookiecutter.readthedocs.io/en/stable/advanced/hooks.html#example-conditional-files-directories

• Remove /tests and update requirements.txt and pyproject.toml if testing is not selected
• Remove setup.cfg (might already be gone by the time this ticket is taken up) and update requirements.txt and pyproject.toml if linting is not selected
• Remove logging.yml if logging is not selected
• Remove /data if data structure is not selected
• Remove /docs and update setup.py if documentation is not selected

[SajidAlamQB]

I had some thoughts on some approaches we could take for this issue which is being done in #2987. Specifically to handle the dependencies in requirements.txt and pyproject.toml in our template. I would love to get any feedback/opinions on them:

Approach A (Reactive):

Start with a requirements.txt and pyproject.toml that includes dependencies for all add-ons by default as we do now.
The post-generation hook script will then remove the dependencies for the add-ons that the user didn't select.

Adv:

• Easier to manage if we only occasionally add or remove add-ons.

Cons:

• More likely have errors as if the files change somehow in the future, the script might fail or introduce weird errors.

Approach B (Proactive):

Start with a barebones requirements.txt and pyproject.toml that only contain the essential dependencies.
The post-generation hook script will append necessary dependencies based on the add-ons the user selects.

Adv:

• Less error-prone since we're adding known dependencies rather than trying to remove potentially unknown ones.
• Keeps the base files clean and simple.

Cons:

• If we frequently add/change dependencies the script might also have to be updated frequently (More burden?).

Which approach do you think is more better for our needs? I am inclined to Approach B but also open to hear everyone's thoughts or opinions. @merelcht @lrcouto @noklam @astrojuanlu @amandakys

PS, maybe an even more radical idea we have separate dependency files for each add-on: e.g., requirements-linting.txt, requirements-testing.txt, etc and we use the hook script to concatenate the selected add-on dependency files into the main requirements.txt post-generation.

PPS, This opens up to a larger question what should the default add-ons case be, in #2987, I've set it to none, no add-ons included, but it this the right way?

[merelcht]

I'm also in favour of approach B as it's the most adaptable if we want to allow users to add plugins later in their project journey and not only in the project creation flow.

PPS, This opens up to a larger question what should the default add-ons case be, in #2987, I've set it to none, no add-ons included, but it this the right way?

IMO, that is indeed the right way. Since we're calling this the "add-ons" flow to me it makes the most sense that you start with nothing and add on what you need, so that means a default of none.

[AhdraMeraliQB]

Closed by #2987