How to build and publish platform-specific wheels to PyPI

[breznak]

Hello,

I am evaluatiing GH Actions and converted my multi platform build to it (runs smoothly on OSX, Linux, Windows).
Great job at providing pypi-publish action 👍 integrates very nicely to the flow.

Now I got error on OSX that the action is only supported on Linux, which is a pity

• Readme should mention only linux support
• could you please provide your action on all GH Actions' platforms?
  • it used to be possible, with twine

https://github.com/htm-community/htm.core/commit/babeb019b8ec3439122ea08d872ebcc265c3717c/checks

Cheers,

[webknjaz]

Wow, that's a good point. I didn't realize that GitHub doesn't run the docker daemon in macOS VMs. Maybe it's even a bug in GitHub workflows.
I think one of the ways to deal with it is to build dists in whatever jobs you want and then use https://github.com/actions/upload-artifact & https://github.com/actions/download-artifact to bypass artifacts (maybe built using different jobs) to the final job that does the upload itself.
WDYT?

[webknjaz]

As for improving the cross-platform support, we need to evaluate whether it's really needed.
Usually, you'd build a bunch of artifacts and then do the upload of all of them at once which seems to be possible with those actions for uploading/downloading artifacts.
If we choose to use "native" actions, we'll have to use JS which probably implies that we'll need to just proxy a shell-script call. But it would also mean that we'd need to somehow force users to maintain proper pre-requisites or install things ourselves.

@pradyunsg any thoughts?

[webknjaz]

@breznak please feel free to send a PR improving the README :)

[breznak]

Thanks for swift reply!

Wow, that's a good point. I didn't realize that GitHub doesn't run the docker daemon in macOS VMs. Maybe it's even a bug in GitHub workflows.

I don't know enough of the Actions' internals, would you investigate, please?
https://github.com/softprops/action-gh-release has a similar functionality (uploads to GH Releases) and works cross-platform. (But seems they don't use docker?)

I think one of the ways to deal with it is to build dists in whatever jobs you want and then use https://github.com/actions/upload-artifact & https://github.com/actions/download-artifact to bypass artifacts (maybe built using different jobs) to the final job that does the upload itself.

thanks, that's a possible workaround 👍

How would I define dependency of jobs? I couldn't find something as needs/requires in CircleCI's workflows.

...
  strategy:
    matrix:
      os: [a, b, c]

jobs:
  - build #builds the app/artifacts on all platforms
    runs_on: matrix.os
    -name: steps... 
   - uses: upload-artifacts

  - pypi # upload artifacts to pypi
    runs_on: ubuntu-18.04
    #FIXME: how do I specify to run after (all of) "build" completed? 
    - uses: download-artifacts
    - uses: gh-action-pypi-publish

evaluate whether it's really needed.
Usually, you'd build a bunch of artifacts and then do the upload of all of them at once which seems to be possible with those actions for uploading/downloading artifacts

the action-gh-release mentioned above does similar, and uploads files from different build envs. The user does not need to maintain/specify anything.

PS: Zdravim z CZ 👋

[breznak]

I guess this https://github.com/remorses/pypi would have the same problem, since it runs docker (?)

[chrispat]

For needs see https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idneeds

[webknjaz]

@chrispat I'm personally aware of job deps via needs: but FYI it was non-trivial for me to locate it in docs back when I was looking for it. Maybe it could be improved/included in some very basic quickstart? (docs feedback)

@breznak

softprops/action-gh-release has a similar functionality (uploads to GH Releases) and works cross-platform. (But seems they don't use docker?)

Yes. They run JS code using NodeJS which is pre-installed in all CI runner VM. For use to do the same, it's needed to pre-install Python and then exec a Python script and proceed from there. This needs some experimentation atm and honestly, I think that the way with collecting artifacts and uploading via a single job looks way better.

I guess this remorses/pypi would have the same problem, since it runs docker (?)

Totally. It also looks half-baked and does more than it should (like setuptools lock-in which is the same problem that Travis CI has).

PS: Zdravim z CZ 👋

Ahoj

[breznak]

I think that the way with collecting artifacts and uploading via a single job looks way better.

I'm starting to think the same. Maybe this issue could be resolved only with a Readme example on using the needs: and artifact-upload to sync and upload all wheels.

[webknjaz]

Yep. But not only the README. I also have a guide PR pending: pypa/packaging.python.org#647.

Please link the results of your experiments here. I can't promise that I'll get to do it myself soon so a PR is also welcome :)

[breznak]

README. I also have a guide PR pending: pypa/packaging.python.org#647.

hey, that's pretty cool, thanks!
I'm currently testing the artifact upload-download-pypi combination. (will take a bit too, we don't have setup.py packaging working correctly, apparently. For that reason, I hadn't known PEP517, looks better than setuptools)

[webknjaz]

looks better than setuptools

It's not "better". It's just a unified way of invoking other tools which defaults to setuptools anyway. It kinda works on a different level.

FWIW, you can invoke PEP517 build backend manually using pep517 dist from PyPI, like this (wrapped with tox):
https://github.com/sanitizers/octomachinery/blob/006bcb0/tox.ini#L68-L87
https://github.com/sanitizers/octomachinery/blob/006bcb0/.github/workflows/publish-to-test-pypi.yml#L29-L31

[breznak]

Thanks for your help, the workflow with artifacts-upload/download works nice, I'm able to collect all whl/egg files from all platforms.
https://github.com/htm-community/htm.core/runs/237965449#step:7:16

Q: Is PYPI and pypi-publish action ok with multiple wheel & egg files in the dist folder? (from each platform)

Run ls dist*
dist1:
htm.core-2.0.16-cp37-cp37m-linux_x86_64.whl
htm.core-2.0.16-py3.7-linux-x86_64.egg
requirements.txt

dist2:
htm.core-2.0.16-cp37-cp37m-macosx_10_14_x86_64.whl
htm.core-2.0.16-py3.7-macosx-10.14-x86_64.egg
requirements.txt

dist3:
htm.core-2.0.16-cp37-cp37m-win_amd64.whl
htm.core-2.0.16-py3.7-win-amd64.egg
requirements.txt

[webknjaz]

Until #10 is addressed, you should put all of the wheels into dist/ directly. Twine will then pick up all of them.

P.S. I see you're building eggs, it's a decade-deprecated format. Better drop those in favor of wheels. And it's also highly recommended to ship a source dist (a .tar.gz tarball) so that end-users could choose to compile stuff on their side.

[breznak]

put all of the wheels into dist/ directly. Twine will then pick up all of them.

perfect, thanks!

building eggs, it's a decade-deprecated format. Better drop those in favor of wheels

oh, didn't know, I'll drop the eggs 🍳

[breznak]

Please link the results of your experiments here.

Hi, I'm getting back with my results, made this method work:

• build on mulpti-platform OS-matrix,
  • for linux I'm using GH Actions' ubuntu-18.04, not manylinux docker image
• push to artifacts, https://github.com/htm-community/htm.core/blob/master/.github/workflows/htmcore.yml#L88
• on Ubuntu image pull artifacts and push to PyPI, https://github.com/htm-community/htm.core/blob/master/.github/workflows/htmcore.yml#L147

All CI seems to pass,
https://github.com/htm-community/htm.core/runs/246384722

But I still have a problem as pip install does not find the version? htm-community/htm.core#19 (comment)

• PyPI release is here https://test.pypi.org/project/htm.core/2.0.18/
• I follow its install instructions: pip install -i https://test.pypi.org/simple/ htm.core==2.0.18
• but fails with

$ pip install -i https://test.pypi.org/simple/ htm.core==2.0.18
Looking in indexes: https://test.pypi.org/simple/
Collecting htm.core==2.0.18
ERROR: Could not find a version that satisfies the requirement htm.core==2.0.18 (from versions: none)
ERROR: No matching distribution found for htm.core==2.0.18

ERROR: No matching distribution found for htm.core==2.0.18

could this be because my "hack"? https://github.com/htm-community/htm.core/blob/master/.github/workflows/htmcore.yml#L135
where I simply renamed the file to *manylinux1_x86_64* to pass PyPI checks?
Thank you for any help!

[webknjaz]

@pradyunsg ^

[pradyunsg]

Ok, I figured this!

Yay!

OT: maybe pip could be more verbose on which condition was not met?

Yes, yes it could. pypa/pip#6526

[webknjaz]

@breznak

I simply renamed the file to *manylinux1_x86_64* to pass PyPI checks
So the proper fix would be to compile in CI with a oldest possible py3.x version.

You shouldn't do this, it's a very bad idea. You have to use manylinux docker containers to build wheels against every Python interpreter version you want to support + also use auditwheel to eliminate external references.
Renaming the wheel only adjusts Pip's search logic but the shared libraries inside won't work under different interpreter versions. So you're basically tricking Pip into downloading the code that 100% won't work in the env.

[breznak]

So you're basically tricking Pip into downloading the code that 100% won't work in the env.

I wasn't able to upload to PyPI a wheel created on Ubuntu18.04, (which would be fine and enough if pip could recognize). So I got the impression that Linux OSs have to be manylinux for pypi.

You shouldn't do this, it's a very bad idea. You have to use manylinux docker containers to build wheels against every Python interpreter version you want to support

yes, I'm aware what I do isn't optimal solution, we already got some users failing on that,
htm-community/htm.core#702 (comment)

But our library is c++/python + other dependencies, and we require a rather modern c++ standard (c++11,ideally '17).

I thought if we build with -staticlibc++ that it would be fine. pypa/manylinux#118 (comment)

The problem is the manylinux (CentOS 5) c++ environment is quite ancient for modern c++ development, so unless newer manylinux images are released for PyPI, I don't know what to do..
pypa/manylinux#118

• would using manylinux + newer gcc (installed from a repo/compiled/or a 3rd party (security implications) docker image of manylinux+gcc9) + using -static-libc++ be any better than the current solution (of using Ubuntu18.04 image)?

also use auditwheel to eliminate external references.

I'll have to try this, building the binary with -static-libc++ but people still get the error:

ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by

[webknjaz]

manylinux (CentOS 5) c++ environment is quite ancient for modern c++ development

@breznak the idea is to only link against things that are 100% present in the user's OS. So if you can compile all externals statically it should work.

You can improve glibc dep a bit by using manylinux2010 which is based on CentOS 6 but you still have to build against glibc present there.

Regarding non-manylinux1, I'm not sure but maybe you could use just linux. @pradyunsg do you know if that works?

Anyway, publishing wheels which are marked as manylinux1 but are not compliant is disrespectful to users because this standard is a promise for that that the thing will work no matter what.

[TkTech]

• would using manylinux + newer gcc (installed from a repo/compiled/or a 3rd party (security implications) docker image of manylinux+gcc9) + using -static-libc++ be any better than the current solution (of using Ubuntu18.04 image)?

I'd like to think I'm trustable, hah, but in the linked issue I do provide a recipe for easily making your own manylinux-derived images with newer versions of GCC. As long as you use -static-libc++ these will work on 99% of Linux installs.

[maartenbreddels]

Would manylinux 2010 not be a solution?

[webknjaz]

@maartenbreddels manylinux2010 allows using a newer toolchain but I think whether it's a solution or not depends on each project's specific needs.

[maartenbreddels]

One issue could be that it requires pip>=19, for the rest I only see benefits.

[webknjaz]

It's not an issue at all. But everyone needs a solution that fits their needs. There's no universal pill.

[RalfG]

To make actual manylinux wheels, I recenty made this GitHub Action that builds using the manylinux1 container: https://github.com/marketplace/actions/python-wheels-manylinux-build/

So if it fits your project, feel free to use it.

[polm]

Is there a guide on how to upload artifacts somewhere and collect them on a later Linux job? @breznak's links are 404 now and I am having trouble following how that is supposed to work.

[webknjaz]

Hey @polm,

Just use actions/upload-artifact@v2 with the same name from multiple build jobs and then have a job with needs that would use actions/download-artifact@v2 to grab the artifact. v2 upload action merges the artifacts with the same name into one so the download will get an artifact will all the files in it.

Here's how I do it: https://github.com/ansible/pylibssh/blob/17e8683/.github/workflows/build-test-n-publish.yml#L353-L357. This example implements quite a complex workflow (with testing exactly the same artifacts that are going to be uploaded) but it should give you some idea of how to address your question.

[polm]

Got it, thanks - that makes it much clearer.

[henryiii]

@polm this is illustrated somewhat simply in the Scikit-HEP guide on binary wheels, too: https://scikit-hep.org/developer/gha_wheels