Best practice for project organization with meson-python?

[inklesspen]

I started poking around with meson-python the other day, and I've achieved a working project which builds a C extension using CFFI: https://github.com/inklesspen/meson-python-cffi-example

I am a little curious if I've made any bad decisions in the organization, though. For example, I've segregated the actual Python source from the CFFI module-generation source. (The _fontconfig.c.py file is executed at build time, producing a _fontconfig.c file, which is then compiled using the py.extension_module rule. It's very much a different sort of file from the __init__.py and fontconfig.py files.)

[rgommers]

Hi @inklesspen, thanks for sharing. The codegen part with CFFI using find_program and generator looks good to me, that is idiomatic use I'd say.

The split between pure Python code to be installed under src/ vs. other sources under cffi does seem a little unusual to me - although that may be from the perspective of a Python package author; it's more common to do it like this for a C/C++ library which also has Python bindings to put Python code in a separate top-level directory.
I suspect you did it like this in order to be able to use install_subdir on src/? What you did should certainly work, it's just more common I think to have a structure like:

src/
  pkgname/
    __init__.py
    fontconfig.py
    cffi/
      _fontconfig.c.py

and then install of install_subdir use py.install_sources['__init__.py, 'fontconfig.py'] (also avoids installing _fontconfig.c.py).

[inklesspen]

As a package gets larger, doesn't it get annoying to specify each file in the install_sources?

But the main reason I put it in another tree is because the Python code under cffi is executed at build time (to generate C source), while the Python code under src is shipped and executed at runtime. The cffi code can't import anything from the src code, or indeed from any package which isn't part of the build requirements. I thought this justified a separate code location. How do other projects handle build-time files of this nature?

[rgommers]

As a package gets larger, doesn't it get annoying to specify each file in the install_sources?

It may indeed be a little annoying, especially when a package is in heavy development. Once it's mature, it's fine in practice in my experience. We use install_sources everywhere in SciPy, and it isn't much of a problem while also having the upside of having granular control so it's easy to avoid installing .py files that are not meant to be installed (e.g., code generation scripts).

How do other projects handle build-time files of this nature?

CFFI is similar to Cython, and if you look in the source trees of numpy/scipy/scikit-learn/etc., you'll see that .pyx files are intermingled with .py files. This is partly historical I think, because that's what distutils/setuptools also did. The upside is that all files that will be installed in pkgname/subdir reside under that subdir, which may make the source tree more easy to navigate.

[inklesspen]

Ah, but that's just my point! _fontconfig.c.py won't be installed; it will be executed to produce c source, which will be compiled to produce a binary, which will be installed.

Perhaps that is too fiddly of a distinction, though.

[eli-schwartz]

There's a bunch of different types of project files:

• C files that are compiled and the resulting CPython extension is installed
• python files that are installed
• tests
• build scripts

I've seen projects that keep all of them in src/, projects that keep the first three there but not the fourth, projects that keep only the first two there, and even...

... projects that keep only the second one there, as well as projects that keep only the first one there.

Expecting any form of universal consensus feels highly unlikely to me.

My personal opinion is that any combination of the first two is reasonable but I wouldn't put tests or build scripts there at all.

[rgommers]

@inklesspen yes indeed. My point was that .pyx files are like .c.py files - or like C files as @eli-schwartz points out - all are not installed but only used to produce extension modules.

My personal opinion is that any combination of the first two is reasonable but I wouldn't put tests or build scripts there at all.

That sounds about right to me.