scikit-build-core build failing due to find_package in project's submodule

[ltsaprounis]

I'm trying to learn/use scikit-build-core in my project here -> https://github.com/ltsaprounis/python-cpp-experiments

The project is relatively simple but it has one submodule that is a git-submodule of the carma library. It all works fine with editable/non-editable installs if I remove the submodule and any code that uses it (check the no-carma branch in the repo). However with the carma submodule, scikit-build doesn't work. And to make things even more confusing, If I just build it using cmake and bypass scikit-build-core it all works fine.

Looking at the error logs I spotted that the problem lies in lines 157-166 in the carma project's CMakeLitsts.txt file:

  find_package(
    Python3
    COMPONENTS Development NumPy
    QUIET)
  if(NOT Python3_FOUND)
    find_package(
      Python3
      COMPONENTS Development.Module NumPy
      REQUIRED)
  endif()

When I use cmake the first call to find_package works and the code in the if statement is never called. When I use scikit-build-core the first find_package call fails and then the one in the if statement is called which fails too.

Below is the error logs I get from the failed installation:

Obtaining file:///Users/<user>/Local_Repos/python-cpp-experiments
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: pybind11[global]>=2.6.0 in ./.venv/lib/python3.9/site-packages (from pycpp-experiments==0.0.1) (2.10.4)
Requirement already satisfied: numpy>=1.14 in ./.venv/lib/python3.9/site-packages (from pycpp-experiments==0.0.1) (1.24.3)
Requirement already satisfied: pybind11-global==2.10.4 in ./.venv/lib/python3.9/site-packages (from pybind11[global]>=2.6.0->pycpp-experiments==0.0.1) (2.10.4)
Building wheels for collected packages: pycpp-experiments
  Building editable for pycpp-experiments (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building editable for pycpp-experiments (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [59 lines of output]
      2023-05-08 19:11:36,482 - scikit_build_core - INFO - CMake version: 3.25.2
      2023-05-08 19:11:36,568 - scikit_build_core - INFO - Build directory: /Users/<user>/Local_Repos/python-cpp-experiments/build
      2023-05-08 19:11:36,663 - scikit_build_core - WARNING - libdir/ldlibrary: /Users/<user>/opt/anaconda3/lib/libpython3.9.a is not a real file!
      2023-05-08 19:11:36,663 - scikit_build_core - WARNING - Can't find a Python library, got libdir=/Users/<user>/opt/anaconda3/lib, ldlibrary=libpython3.9.a, multiarch=darwin, masd=None
      loading initial cache file build/CMakeInit.txt
      -- The C compiler identification is AppleClang 14.0.0.14000029
      -- The CXX compiler identification is AppleClang 14.0.0.14000029
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- The project is built using scikit-build
      -- Found PythonInterp: /Users/<user>/Local_Repos/python-cpp-experiments/.venv/bin/python (found suitable version "3.9.13", minimum required is "3.6")
      -- Found PythonLibs: /Users/<user>/opt/anaconda3/lib/libpython3.9.dylib
      -- Performing Test HAS_FLTO
      -- Performing Test HAS_FLTO - Success
      -- Performing Test HAS_FLTO_THIN
      -- Performing Test HAS_FLTO_THIN - Success
      -- Found pybind11: /Users/<user>/Local_Repos/python-cpp-experiments/.venv/lib/python3.9/site-packages/pybind11/include (found version "2.10.4")
      CMake Warning (dev) at /usr/local/Cellar/cmake/3.25.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
        The package name passed to `find_package_handle_standard_args` (Armadillo)
        does not match the name of the calling package (armadillo).  This can lead
        to problems in calling code that expects `find_package` result variables
        (e.g., `_FOUND`) to follow a certain pattern.
      Call Stack (most recent call first):
        /usr/local/Cellar/cmake/3.25.2/share/cmake/Modules/Findarmadillo.cmake:117 (find_package_handle_standard_args)
        CMakeLists.txt:81 (find_package)
      This warning is for project developers.  Use -Wno-dev to suppress it.
      
      -- Found Armadillo: /usr/local/include (found version "12.0.1")
      -- carma: enable arma no debug: OFF
      -- carma: enable extra debug: OFF
      -- carma: enable soft steal: OFF
      -- carma: enable hard steal: OFF
      -- carma: don't require owndata: OFF
      -- carma: don't require f-contiguous: OFF
      CMake Error at /private/var/folders/<some_path>/T/pip-build-env-4njk82s8/overlay/lib/python3.9/site-packages/scikit_build_core/resources/find_python/FindPackageHandleStandardArgs.cmake:230 (message):
        Could NOT find Python3 (missing: Python3_NumPy_INCLUDE_DIRS NumPy) (found
        version "3.9.13")
      Call Stack (most recent call first):
        /private/var/folders/<some_path>/T/pip-build-env-4njk82s8/overlay/lib/python3.9/site-packages/scikit_build_core/resources/find_python/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
        /private/var/folders/<some_path>/T/pip-build-env-4njk82s8/overlay/lib/python3.9/site-packages/scikit_build_core/resources/find_python/FindPython/Support.cmake:3800 (find_package_handle_standard_args)
        /private/var/folders/<some_path>/T/pip-build-env-4njk82s8/overlay/lib/python3.9/site-packages/scikit_build_core/resources/find_python/FindPython3.cmake:551 (include)
        pycpp/cpp/carma/CMakeLists.txt:162 (find_package)
      
      
      -- Configuring incomplete, errors occurred!
      See also "/Users/<user>/Local_Repos/python-cpp-experiments/build/CMakeFiles/CMakeOutput.log".
      See also "/Users/<user>/Local_Repos/python-cpp-experiments/build/CMakeFiles/CMakeError.log".
      *** scikit-build-core 0.3.3 using CMake 3.25.2
      *** Configurating CMake...
      
      *** CMake configuration failed
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for pycpp-experiments
Failed to build pycpp-experiments
ERROR: Could not build wheels for pycpp-experiments, which is required to install pyproject.toml-based projects

I tried to set the variables for the python executable, include dir and library dir in the CMakeLists.txt file as suggested in #506 but didn't have much luck with that. I think it has to do with the python path that is passed along to carma's cmake from the scikit-build-core process, but I don't know how to debug this further.

Any help/tips will be very appreciated! 🙏

Additional information:

the project's pyproject.toml file:

[build-system]
requires = ["scikit-build-core>=0.3.3", "pybind11"]
build-backend = "scikit_build_core.build"

[project]
name = "pycpp_experiments"
version = "0.0.1"
description = "A project for experimenting with python C++ bindings"
readme = "README.md"
requires-python=">=3.8,<3.11"
dependencies=[
    "pybind11[global]>=2.6.0",
    "numpy>=1.14",
]

# [tool.setuptools.packages]
# find = {}

[project.optional-dependencies]
dev = [
    "flake8",
    "black",
    "pytest",
    "pydocstyle",
    "pre-commit",
    "ipykernel",
]

[tool.pytest.ini_options]
addopts= "--ignore=pycpp/cpp/carma"

[tool.scikit-build]
# wheel.expand-macos-universal-tags = true
build-dir = "build"
logging.level = "INFO"
ninja.make-fallback = true
# This activates verbose builds
cmake.verbose = true
ninja.minimum-version = "1.5"
cmake.minimum-version = "3.25"
# cmake.args = ["-DPYTHON_EXECUTABLE=$(python3 -c 'import sys; print(sys.executable)')", "-DCMAKE_INSTALL_PREFIX:PATH=.."]

[henryiii]

You are missing NumPy. You should probably add “numpy” (or, more likely “oldest-supported-numpy”) to your build-system.requires list.

(traveling currently, can explain more later)

[ltsaprounis]

That was fast 🚀
Thanks a lot!

I added "numpy" in build-system.requires and it worked like a charm! In hindsight, it seems obvious that the same way pybind11 is required in build-system.requires so should numpy be required 🤦‍♂️

If you add some more details when you have some more time, for the other people that stumble upon this discussion (and me of course) that would be great!