Handling a submodule within your repository that has compiled code

[adam2392]

Hi,

I am loving the new feature of editable installs on v0.13. I have a question regarding submodules. I am using Meson-Python itself to build my entire Python package, which has Cython and Python code and prolly soon a little bit of C++.

I am currently trying to vendor a fork of a popular package (scikit-learn) within my codebase. The current strategy I have is to create a maintained fork of scikit-learn and release it on pypi under an alias (e.g. scikit-learn-tree). However, this is not an ideal strategy as this prevents 3rd party packages that depend on scikit-learn from depending on my package. As a work-around it has been suggested that I vendor the forked repo as a submodule within my package instead.

However, this now raises a few questions as to how to build and develop on the package:

1. Is meson-python capable of building a submodule that doesn't use meson, but setuptools? E.g. something like ./spin build -j 4 (which also calls python setup.py install ./<my_pacakage>/_lib/scikit_learn_fork/)
2. Can this be done automatically and cleanly?
3. If not, is there another way of handling this situation using Meson?

[eli-schwartz]

You probably need to bite the bullet and port that submodule to meson too.

Question: why do you need to vendor at all, though? Vendoring just means you:

• build a private copy every time that cannot be reused between reinstalls of your own package
• the private copy is probably imported as mypackage.scikit_learn_tree instead of scikit_learn. But you could simply publish a public fork as "scikit_learn_tree" that doesn't conflict at all?

[dnicolodi]

The current strategy I have is to create a maintained fork of scikit-learn

I don't understand why you need a fork of scikit-learn. Isn't scikit-learn maintained? I see a release on March 9th, thus I would call it maintained. What is the reason why you need to fork or vendor scikit-learn?

[adam2392]

We want a fork because we make changes to a specific submodule inside scikit-learn that isn't in main yet. We therefore want the explicit fork so we don't get into this business of copying/pasting the code and then not being able to sync in changes from main. This just improves our maintainence in terms of refactoring to "be up-to-date".

Question: why do you need to vendor at all, though? Vendoring just means you:

• build a private copy every time that cannot be reused between reinstalls of your own package
• the private copy is probably imported as mypackage.scikit_learn_tree instead of scikit_learn. But you could simply publish a public fork as "scikit_learn_tree" that doesn't conflict at all?

This is an interesting strategy... I guess the issue is that rn scikit-learn-tree (the repo) gets imported with the sklearn namespace, which causes the issue. I suppose, I could just rename the entire module as sklearn_fork and stick w/ releasing the fork in this way then... I know this is not the original question anymore, but does that sound like a good fix?

scikit-learn unfortunately hasn't swapped to meson build yet, so would prefer not doing the first soln :p

[rgommers]

Releasing a fork like that seems reasonable.

scikit-learn unfortunately hasn't swapped to meson build yet, so would prefer not doing the first soln :p

I'm hopeful that they will, once they're sure everything is stable for other packages. There are nice benefits after all:)

[adam2392]

Hi @rgommers a quick follow up question. This approach unfortunately was not a fan by ppl because it involves releasing on pypi a whole new package.

Instead it was suggested that I just add a few meson.build files to the necessary modules within the forked sklearn git-submodule. That is I would add a meson.build file to sklearn/, sklearn/tree/, sklearn/ensemble/. I was wondering if there is some documentation on how to best handle this situation.

I basically have a meson build structure like:

sktree/
     _lib/
           scikit-learn/
                  sklearn/
                         tree/
                         ensemble/
                         ...
    stuff.py
    stuff.pxd
    stuff.pyx
    meson.build

At the top level outside of _lib/ I have Cython files that will need to depend on stuff inside the _lib/ folder, which houses the submodule. Can meson.build files build stuff outside their directory?

[eli-schwartz]

The cython files need to depend on, or build? Shouldn't you be building whatever sklearn files inside _lib/scikit-learn/sklearn/meson.build or something?

sktree/meson.build can definitely depend on files created in _lib/scikit-learn/sklearn/meson.build, just use subdir() on _lib/ etc. first.

[adam2392]

The cython files inside sktree/_lib/scikit_learn/sklearn/tree/ need to build and then the cython files inside sktree/stuff.pyx depend on those. I think I realized the subdir() works sequentially, but for some reason, it is impossible to "discover the files inside sktree/_lib/scikit_learn/sklearn/tree/ from sktree/stuff.pyx.

E.g. I do things like:

# inside the stuff.pyx file

# this basically gives me an error saying there is no such pxd file even tho there is one
from sktree._lib.scikit_learn.sklearn.tree._criterion cimport Criterion
...

Sorry this is turning into a whole mess beyond the question that was originally asked. I guess it's hard cuz this is not a common use-case. I tried also pinging Chat-GPT LOl but the solns given were bogus or not what we want.

[adam2392]

I ended up trying the following meson.build file inside sktree/_lib/

fs = import('fs')
if not fs.exists('sklearn/README.rst')
  error('Missing the `sklearn` fork submodule! Run `git submodule update --init` to fix this.')
endif

# install _lib/
python_sources = [
  '__init__.py',
  'test.py',
]
py3.install_sources(
  python_sources,
  subdir: 'sktree/_lib'
)

# install tree/ submodule
extensions = [
  '_splitter',
  '_tree',
  '_criterion',
  '_utils',
]

foreach ext: extensions
  py3.extension_module(ext,
    cython_gen_cpp.process('./sklearn/sklearn/tree/' + ext + '.pyx'),
    c_args: cython_c_args,
    include_directories: [incdir_numpy, ],
    install: true,
    subdir: './sklearn/sklearn/tree',
  )
endforeach

python_sources = [
  '_classes.py',
]
foreach py_source: python_sources
  py3.install_sources(
    './sklearn/sklearn/tree/' + py_source,
    subdir: './sklearn/sklearn/tree'
  )
endforeach

# install ensemble/ submodule
python_sources = [
  '_forest.py',
]
foreach py_source: python_sources
  py3.install_sources(
    './sklearn/sklearn/ensemble/' + py_source,
    subdir: './sklearn/sklearn/ensemble'
  )
endforeach

# subdir('sklearn/sklearn/tree/')
# subdir('sklearn/sklearn/ensemble/')

This ends up compiling and building, but as soon as I try to add Cython files inside sktree/tree/*, I get errors like this:

[1/15] Generating 'sktree/tree/_sklearn_splitter.cpython-39-darwin.so.p/_sklearn_splitter.cpp'.
FAILED: sktree/tree/_sklearn_splitter.cpython-39-darwin.so.p/_sklearn_splitter.cpp 
python /Users/adam2392/Documents/scikit-tree/sktree/_build_utils/cythoner.py ../sktree/tree/_sklearn_splitter.pyx sktree/tree/_sklearn_splitter.cpython-39-darwin.so.p/_sklearn_splitter.cpp --cplus

Error compiling Cython file:
------------------------------------------------------------
...
from .._lib.sklearn.sklearn.tree._tree cimport DOUBLE_t  # Type of y, sample_weight
^
------------------------------------------------------------

/Users/adam2392/Documents/scikit-tree/sktree/tree/_sklearn_splitter.pxd:1:0: 'sktree/_lib/sklearn/sklearn/tree/_tree.pxd' not found
inside cythoner: 
input file:  /Users/adam2392/Documents/scikit-tree/sktree/tree/_sklearn_splitter.pyx
output file:  /Users/adam2392/Documents/scikit-tree/build/sktree/tree/_sklearn_splitter.cpython-39-darwin.so.p/_sklearn_splitter.cpp

Any chance you understand why this is occurring. My intuition is saying that somehow these files inside _lib/ are not being discovered by the Python environment?