Improve Windows CI setup

[merelcht]

Description

Our Windows CI has always been problematic. We've got several recurring failures, some related to install steps, other related to flaky tests.

Is there anything we can do to improve the setup? This issue is an investigation into what we can possibly do and come up with a plan if there are any improvements we can make. Time box this to 2-3 days max. Perhaps talking to other teams will give inspiration.

Considering we're very likely moving our setup to Github actions, we shouldn't spend more than a sprint on this.

[astrojuanlu]

Windows flakiness has been getting worse and worse: #2570, #2732

Maybe that part of the CI could be moved to GHA (not necessary to move everything just yet)?

[astrojuanlu]

Opened a proof of concept here #2841 (still on Circle CI but streamlining the conda setup)

[merelcht]

Closing this now we've moved to Github Actions.