Proof of concept: Rework Windows CI

[astrojuanlu]

Description

Windows tests are noticeably slow, but most of the time is spent recreating the conda cache and installing the dependencies (up to 20 minutes!)

In addition, lots of system dependencies are installed manually with choco or other systems, while they could be easily installed with conda in one stroke. This includes make, pyspark, and others.

The problem with this approach is that pip install .[test] will ignore the packages that are in the conda environment and reinstall them again. This creates problems not only with pyspark, but also with numpy, since many packages are compiled against it and reinstalling it should never be done. This lack of interoperability between pip and conda is unfortunately a known issue.

The only way I see to proceed is generating an environment.yml that conda understands that sort of duplicates setup.py and contains the dependencies in a way that can be installed with conda, and then leaving only the pip instal . to install the development version of kedro. This would introduce some duplication, but maybe it would help us keep the conda recipe up to date (see conda-forge/kedro-feedstock#42).

For now, to do an experiment I duplicated the table of extra requirements in setup.py, but I don't think it looks great to be honest.

What do folks think?

Development notes

This uses some newer capabilities of conda, such as the libmamba solver which is much faster https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes

[astrojuanlu]

Closing this in favor of #2843, which doesn't even use conda!