feat: Python wheels workflow and build backend

[zachlewis]

Description

Summary

This PR is the spiritual successor to #4011. It implements a scikit-build-core-based python build-backend, making it possible to use pip install . to build from source; and it adds a Github workflow for building with cibuildwheel and publishing to pypi.org binary distributions (bdists) of the Python module / extensions / CLI tools for cpython 3.8-3.12, across major operating systems and architectures.

When you pip install OpenImageIO, pip attempts to retrieve an OpenImageIO bdist from pypi.org for the host's platform / architecture / Python interpreter. If it can't find something appropriate, pip will attempt to build locally from the OpenImageIO source distribution (sdist), downloading and temporarily installing cmake and ninja if necessary.

PEP-Compliant Packaging: pyproject.toml

The pyproject.toml file is organized in three parts:

1. Package metadata: standard attributes identifying and describing the Python package, its run-time and build-time requirements, entry-points to executable scripts, and so forth.
2. scikit-build-core options: governs how pip install ... interacts with cmake.
3. cibuildwheel options: additional steps and considerations for building, packaging, "repairing", and testing relocatable wheel build artifacts in isolated environments.

Additions to __ init __.py

Previously, we were using a custom OpenImageIO/__ init__.py file to help Python-3.8+ on Windows load the shared libraries linked by the Python module (i.e., the .dll files that live alongside oiiotool.exe under $PATH).

This PR adds an additional method for loading the DLL path, necessitated by differences between pip-based and traditional CMake-based installs.

It also adds a mechanism for invoking binary executables found in the .../site-packages/OpenImageIO/bin directory. This provides a means for exposing Python script "shims" for each CLI tool, installed to platform-specific locations under $PATH, while keeping the actual binaries in a static location relative to the dynamic libraries. Upshot is, in pyproject.toml,
each item under [project.scripts] is turned into a Python script upon installation that behaves indistinguishably to the end user to the CLI binary executable of the same name.

Relocatable Binary Distributions with cibuildwheel + repairwheel

cibuildwheel is a widely-used tool for drastically streamlining the process of building, repairing, and testing Python wheels across platforms, architectures, interpreters, and interpreter versions.

Importantly, cibuildwheel allows for an arbitrary "repair-wheel-command" step, intended as a means for collecting and bundling any external shared libraries linked by the module (or the CLI tools) living outside the package, and subsequently patching RPATH entries to point to relative paths to these libraries inside the wheel. This ensures that compiled extensions have no runtime dependencies outside the pip toolchain.

In order to keep wheel file sizes to an absolute minimum, our repair-wheel-command is a multi-step process, implemented in a new tasks.py file living in the repo's root, invoked with invoke:

1. Remove everything except for the python module and the binary executables under /bin.
2. Use repairwheel to collect and bundle only and exactly the shared libraries needed -- i.e., missing dependencies built by OIIO's build system, as well as any other needed libraries found in the environment.

Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to "MinSizeRel" to optimize for size (instead of speed) -- this seems to shave ~1.5MB off each .whl's size.

Note: the steps taken in tasks.py remove everything except that which is absolutely necessary for end users to import the Python modules and run the CLI tools, partly in order to determine a baseline minimum file size for the wheels. However, part of the motivation for setting things up the way I have is to make it easier to add things back into the wheels in the future; e.g., should we decide to package the headers and/or the fonts, generate or modify pkgconfig .pcs and/or CMake modules and configs, etc. Also worth noting, there's pyproject.toml metadata we can add to help downstream scikit-build-core-based builds find OpenImageIO.

"Wheels" Github workflow

I straight-up copied .github/workflows/wheel.yml from OpenColorIO and made a few OIIO-specific modifications. When pushing a commit tagged v2.6* or v3*, the workflow will invoke a platform-agnostic "build sdist" (source distribution) task, followed by a series of tasks for building OIIO wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new libstdc++), and MacOS (x86_64 + arm64, min deployment 10.15) and persisting build artifacts; followed finally by a task for publishing the build artifacts to pypi.org

Note: For the sake of simplicity and troubleshooting, I've made as few changes to OpenColorIO's wheel.yml as I could get away with; but in the future, we can also build wheels for the PyPy interpreter, and possibly pyodide.

Note: THE PUBLISH TASK WILL FAIL until a "trusted publisher" is set up on pypi.org. See https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/

I've submitted separate PRs for these, since they fall outside the intended scope of this PR.

Note: One of the aims for this PR is to leverage OIIO's new dependency-self-building mechanisms as much as possible, so that users who pip install from source are guaranteed a nominally working installation. However, we can also use platform-specific package managers for the wheels uploaded to pypi.org (e.g., added as cibuildwheel "before-build" commands for each platform).

Other Changes

I made some minor adjustments to pythonutils.cmake and fancy_add_executable.cmake that only affect scikit-build-core-based installs -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property to point to the relative path to the dynamic libraries, for the Python module and CLI tools, respectively. This helps ensure that pip-based builds and installs from source (as opposed to installs from repaired, pre-built wheels) yield relocatable, importable packages, without needing to mess with $LD_LIBRARY_PATH etc.

Tests

The cibuildwheel "test-command" is oiiotool --help. If that command elicits code zero, it means the "oiiotool" Python script installed by the wheel is able to import OpenImageIO; that the actual binary executable oiiotool is properly packaged and exists in the expected location (e.g., at .../site-packages/OpenImageIO/bin); and that all runtime dependencies are found.

Inspiration, Credit, Prior Art

• @aclark4life's and @JeanChristopheMorinPerso's efforts + direction + discussion + advice. See #3249, and #4011, as well as JCM's python_wheels and python_wheels_windows branches. This PR is an attempt to leverage OIIO-2.6+ self-building-dependency features with # 4011's minimalist and modern approach to packaging.
• OpenColorIO -- I tried to copy as much as I could from @remia et al's fantastic work with all things wheels-related. The __init __.py modifications, the way we're wrapping the CLI tools, and the github Wheels workflow are lifted almost-verbatim from OCIO. Insert pun about reinventing the wheel here.
• @joaovbs96's help and patience with testing stuff on Windows.

[zachlewis]

Okay, I think I was able to prevent OIIO from linking x265. It seems the system TIFF was built against WebP... so now we're always building all missing dependencies (on MacOS), and LINKSTATIC is enabled, in order to force the build system to pick up our locally-built static dependencies, where possible. We're also forcibly uninstalling the runner image's Freetype, (installed with homebrew) in order to avoid linking some of its dependencies.

The x86_64 .dylibs directory looks like this:

.
├── libbrotlicommon.1.1.0.dylib
├── libbrotlidec.1.1.0.dylib
├── libbrotlienc.1.1.0.dylib
├── libhwy.1.2.0.dylib
├── libIex-3_2.31.3.2.4.dylib
├── libIlmThread-3_2.31.3.2.4.dylib
├── libImath-3_1.29.11.0.dylib
├── libjpeg.8.3.2.dylib
├── libjxl.0.11.0.dylib
├── libjxl_cms.0.11.0.dylib
├── libjxl_threads.0.11.0.dylib
├── liblcms2.2.dylib
├── libOpenColorIO_v_2_3_2_OIIO.2.3.2.dylib
├── libOpenEXR-3_2.31.3.2.4.dylib
├── libOpenEXRCore-3_2.31.3.2.4.dylib
├── libOpenImageIO.2.6.7.dylib
├── libOpenImageIO_Util.2.6.7.dylib
├── libpng16.16.dylib
├── libsharpyuv.0.1.0.dylib
├── libwebp.7.1.9.dylib
└── libwebpdemux.2.0.15.dylib

This is still larger than the ARM64 wheels, and some of these dependencies seem to be coming from the system level (libwebp, ...), and others from homebrew (libpng16, jpeg i think). Interestingly, OCIO is linking the shared lcms2 library, and probably the shared IMath library as well, both of which are provided by the runner image.

As long as there are no GPL-licensed dependencies, I'm cool with leaving this as-is.

I believe at this point, the only failing check is the CI / bleeding edge run.

[github-advanced-security]

Scorecard found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.