Excluding files from sdist archive

[paugier]

If I understand correctly, meson-python includes all the files in the sdist and it is not possible to exclude some directories or files.

In contrast, setuptools and Hatch support controlling which files are included in the sdist. For Hatch, there are few related options in the pyproject.toml file: https://hatch.pypa.io/latest/config/build/#file-selection

Would it be possible to at least be able to exclude some files with meson-python ?

[dnicolodi]

meson-python uses meson dist to create sdist. meson dist obeys the export-ignore git attribute. Just mark the files you want to exclude from the sdist with this attribute in a .gitattributes file.

[paugier]

Thanks for this partial solution!

Does meson-Python officially support only Git repositories? What about other cases? See mesonbuild/meson#12796

[dnicolodi]

This is not a partial solution: it is the only solution there is.

meson-python uses meson dist to create the sdist archive. meson dist supports only Git and Mercurial. Thus, meson-python supports only Git and Mercurial. If you need support for other revision control systems, it would need to be added to Meson, but it is easy to implement. Not supporting generating source distributions from sources that are not under revision control is a decision that it is extremely unlikely to be revised. Actually, meson-python currently allows to generate a source distribution from a "dirty" checkout. It is on my todo list to stop meson-python from picking up these local changes.

I think the equivalent of export-ignore in .gitattributes for hg is to add a .hgrc with something like

[defaults]
archive = -X /pattern/

in it.

[eli-schwartz]

Actually, meson-python currently allows to generate a source distribution from a "dirty" checkout. It is on my todo list to stop meson-python from picking up these local changes.

IIRC this just involves additional code to go back and copy more files into the sdist, right?

(I think I recall that this was implemented so that SciPy could modify pyproject.toml to select a backend and then copy that change into the sdist?)

[rgommers]

Not just SciPy, it was to allow applying patches in the normal way, without having to know about the underlying build system. While a project that uses meson-python could learn about dist scripts or another specific way, this would be very onerous on distros.

[dnicolodi]

@rgommers Which distributions repack the original sdist into a modified sdist? I don't know any doing so. Even then, how is this something specific to Python packages and not a problem for distribution packages of sofware using Meson?

There are two things that I find very confusing: building from a git clone and from an sdist producing different results, and having the sdist contain modification to version controlled files picked up, but not new files. Why do we need to enforce that patching existing files is a good thing, but adding files to the sdist is not?

It is much easier to explain that an sdist is equivalent to the archive created by meson dist plus metadata.

In the case of SciPy, I spent quite some time trying to find where in the build process that produces the sdist files are or were patched and I didn't find anything.

The only remotely valid explanation for the current behavior that I could find, is that build by default creates an sdist and from that it creates a wheel. Thus, if someone is testing a build of a locally modified package, without specifying the correct command line arguments, they would get a build not containing the uncommitted changes. However, picking up modifications to version controlled files but no new files is even more confusing.

[dnicolodi]

IIRC this just involves additional code to go back and copy more files into the sdist, right?

Yes, that's right, but not more files, only modifications to files that are already in the dist archive.

[rgommers]

Sorry for the slow reply - holiday week this week.

It's been almost 2 years, so I am likely misremembering. I just re-read gh-53, which has useful context.

Which distributions repack the original sdist into a modified sdist? I don't know any doing so

None, and that's not a valid thing to do even. I think I confused myself here; distros apply patches for building a wheel, which can be done from sdist or git tag/commit. As long as they can apply such patches without a change in behavior, there should not be a problem.

In the case of SciPy, I spent quite some time trying to find where in the build process that produces the sdist files are or were patched and I didn't find anything.

gh-53 has a lot of context. However, after investigating: numpy/scipy sdists are no longer patched. Currently, sdists aren't patched, wheels are. E.g., https://github.com/scipy/scipy/blob/c2806e4ec7651fbfa401829884476a9036988579/tools/wheels/cibw_before_build_linux.sh#L20. Or https://github.com/scipy/scipy/blob/c2806e4ec7651fbfa401829884476a9036988579/tools/openblas_support.py#L243.

It looks like I gave up on sdist patching in anticipation of PEP 621 and PEP 639 support for multiple license files. It's bad that that still hasn't materialized, so right now the licensing situation for numpy and scipy isn't great; there's just a separate LICENSES_bundled.txt file that contains all the license info other than BSD-3. What PyPI displays as the license is incomplete.

The only remotely valid explanation for the current behavior that I could find, is that build by default creates an sdist and from that it creates a wheel. Thus, if someone is testing a build of a locally modified package [...]

This was touched upon in gh-53 indeed. It will introduce a bigger deviation between the repo -> wheel and repo -> sdist -> wheel paths, but there are quite a few differences possible already, so this isn't critical.

tl;dr I think a change to the sdist-only behavior is now okay.

[eli-schwartz]

I'd just also like to note that hatch's approach to selecting files for inclusion/exclusion in an sdist / wheel uses the .gitignore file and therefore violates the inherent definition of the git project's gitignore functionality. It cannot be used correctly and it violates the actual use of gitignore.

This is why meson didn't select such an approach, and never will.