Question: Non-wheel audits

[jaimergp]

Hello! I read your blog post and was amazed by the amount of resources that could be saved in organizations like conda-forge if abi3 was supported by default in compatible cases.

I would like to ask some questions to assess how easy it would be run an audit like the one done with PyPI, but for conda-forge (we have an upper bound of ~2k projects that could be audited):

• In this issue, you mention that support for non-abi3 wheels is not available (yet?). One concern is that a no-errors report might give the impression that wheels can be retagged as abi3 with no further changes. However, I guess that rebuilding from source with abi3 mode on would indeed work, right?
• Following on that issue, is it possible to audit arbitrary collections of shared objects, even if they are not part of a wheel? This would enable us to run the tool on extracted conda packages.
• What's the story for Windows? Are DLLs supported? If not, would it be easy to implement? nvm, I see support for DLLs here, awesome!

Thank you so much for this work and the fascinating story in the blog post, really enjoyed that read!

[woodruffw]

Thank you for the kind words! I'm glad you found the blog post interesting and useful 🙂

• In this issue, you mention that support for non-abi3 wheels is not available (yet?). One concern is that a no-errors report might give the impression that wheels can be retagged as abi3 with no further changes. However, I guess that rebuilding from source with abi3 mode on would indeed work, right?

Yep -- the problem is that wheels that are not explicitly tagged as abi3 might appear to be abi3 compliant solely by virtue of the subset of CPython APIs they link against. But this doesn't actually make than abi3 compliant, since without the actual macros that control the API/ABI we have no way of determining that those symbols actually have implementations that are abi3 compatible.

Rebuilding from source with the abi3 flags passed in would indeed work -- if your packaging tooling is configured correctly, that should result in a wheel that has the abi3 tag and constituent shared objects that also have the .abi3. infix.

• Following on that issue, is it possible to audit arbitrary collections of shared objects, even if they are not part of a wheel? This would enable us to run the tool on extracted conda packages.

Yes! This is already supported: you should be able to run a command like this:

# audit foo.abi3.so, bar.abi3.so, baz.abi3.so
$ abi3audit {foo,bar,baz}.abi3.so

...and abi3audit will audit those individual shared objects, outside of any wheel. However, there are limitations with this mode: the shared object needs to explicitly signal that it's abi3 compatible with the .abi3. infix, and abi3audit isn't able to provide as reliable "baseline" information since the infix doesn't say which CPython abi3 version the object is actually compatible with. Consequently it assumes CPython 3.2 (the earliest abi3 version), and so you may end up with more false positives.

[jaimergp]

Awesome, thanks for the thorough answer!

I'd like to see if we can use abi3audit to run a scan on our a sample of our Python packages, hoping to find good abi3 candidates.

The idea would be something like:

for pkg in pkgs_sample:
  contents = extract(pkg)
  so_list = find_shared_objects(contents)
  for so in so_list:
    # so = add_abi3_infix(so)
    audit_shared_object(so, min_python="3.7")

Would it be possible to have a flag to indicate a different minimum Python version (instead of 3.2)? Ideally, if we can also get rid of the .abi3. infix requirement with an --advanced mode or something, it would save a few (but trivial) symlinks :D

[woodruffw]

I'd be okay with an --assume-minimum-abi3=37 or similar flag! That would preserve the current .abi3. infix expectation, but treat it as a minimum abi3 of 3.7 (or whatever is supplied) rather than 3.2.

I'm hesitant to get rid of the infix check even in an advanced mode, because it's really the only remaining signal CPython gives us to expect abi3 compatibility and not just a "normal" CPython extension. Maybe this is something we could do with Conda-specific functionality? If the expectation is that Conda always builds abi3 compatible packages, then maybe we could add .conda as a supported format at the top-level.

[woodruffw]

Opened #34 for Conda support more generally 🙂