Transition of VTK to scikit-build

[jcfr]

This discussion is intended to provide a collaborative ground to better understand what would be needed to build the official VTK wheels published on PyPI by leveraging the new scikit-build-core back-end.

Indeed, considering that the use of setuptools as it is currently done in VTK can be considered legacy ¹² and will likely break in the near future, the idea would be to integrate scikit-build-core and remove the brittle and hard to maintain approach implemented in CMake/setup.py.in.

As described in vtk#18750³ by @mathstuf , the current understanding is that system want to know what modules will exist before the build starts. And that based on this assumption, the VTK buld-system may not be able to leverage scikit-build-core.

Footnotes

1. https://setuptools.pypa.io/en/latest/build_meta.html ↩

2. https://setuptools.pypa.io/en/latest/deprecated/index.html ↩

3. https://gitlab.kitware.com/vtk/vtk/-/issues/18750#note_1291216 ↩

[mathstuf]

So the problem is that scikit-build-core must be an optional thing in building VTK. It cannot be required to do anything, so pretty much all of the SK_BUILD_* variables are of no use to it. What it can do is detect that scikit-build-core is running and go into the "I need to set up rpaths and resource directories such that the wheel will work" (basically what vtkWheelPreparation.cmake does now). The thing is that if scikit-build-core cannot do:

• pass CMake a bundle of configure flags
• run the build
• package whatever comes out of the build (vtkmodules.* and vtk.py)
• there may be special cases like "add this data file if this module is enabled" that need detected
• the Python dependencies of a given VTK build are not known until after CMake has configured the build

I don't see how it will be any more maintainable than the existing setup.py (and if Python wants to go the way of "we only support things that start from pyproject.toml", then VTK may well be a Conda-only package unless someone steps up to maintain whatever rat's nest of duplicated logic that will (likely) end up being.

Note that I expect these to only work on one-shot builds. That is, once configure flags change after a build, it needs to be cleaned because a module that was on and is now off may still be laying around and should not be packaged.

Trying to dictate that VTK will use a specific Python should be done through whatever FindPython3 expects. Trying to tell VTK "you must use the limited API" should go through whatever CMake variables it exposes for that.

Looking at the README for scikit-build-core, I see these problems:

• The minimum supported Python is 3.7
  • VTK has yet to deprecate 3.6, so we cannot drop the old stuff until at least post-9.3, but we still have customers using 3.5 (though I don't think they care about wheels)
• Wheels are not fully reproducible yet
  • What non-determinism is scikit-build-core introducing itself? This is not something that can be dictated from outside; every project must be careful to not break the property at every level.