pip install kedro-datasets[<optional>] doesn't work with pip 24.0

[noklam]

Description

pip has been an ongoing issue with CI, but now it's also affecting users. For an user who start with an environment with pip==24.0, after running kedro-datasets[pandas.ParquetDataest], it does not fail but rather silently skip depdencies such as pyarrow.

pip related hot fixes recently:

• Pin pip<23.2 and skip one micropkg test to unblock CI from breaking changes kedro#2816
• Fix ci - Use --resolver=backtracking with kedro build-reqs in e2e tests kedro#3010
• Unpin pip version and bump actions versions kedro#3598
• Pin pip < 23.2 and Port unit tests, lint and e2e tests to Github Actions unblock CI kedro#2843
• fix(datasets): Correct pyproject.toml syntax for optional dependencies #302

Context

Steps to Reproduce

Expected Result

Actual Result

-- If you received an error, place it here.

-- Separate them if you have more than one.

Your Environment

• Kedro version used (pip show kedro or kedro -V):
• Python version used (python -V):
• Operating system and version:

[astrojuanlu]

Confirmed. With pip 23.3 onwards:

WARNING: kedro-datasets 2.0.0 does not provide the extra 'pandas-parquetdataset'

With pip < 23.3 it proceeds normally.

From the changelog: https://pip.pypa.io/en/stable/news/#v23-3

Normalize extras according to PEP 685 from package metadata in the resolver for comparison. This ensures extras are correctly compared and merged as long as the package providing the extra(s) is built with values normalized according to the standard. Note, however, that this does not solve cases where the package itself contains unnormalized extra values in the metadata. (#11649)

Closing this as duplicate of #313 and raising the priority of that one.