Build & release to PyPI via trusted publishing

[hugovk]

We discussed in the last monthly docs meeting to automate PyPI deploys using the new "Trusted Publishing", and the consensus was it's a good idea:

• https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/

This would make releasing easier, and also open the release process to other core devs with the commit bit.

Here's a PR to set up the workflow, we'll also need to set up trusted publishing on TestPyPI and PyPI.

---

This workflow does three things:

It is based on @hynek's neat https://github.com/python-attrs/attrs/blob/main/.github/workflows/pypi-package.yml, which I'm successfully using in my own projects.

1.

For all runs, it builds and lints the PyPI sdist and wheel. This makes sure the non-publishing steps are working smoothly, ahead of uploading. It uses @hynek's excellent https://github.com/hynek/build-and-inspect-python-package action, which creates a nice summary, for example:

We can see it creates three handy zips if we want to check what would be uploaded to PyPI, and three handy toggles that show the contents. Try it here:

https://github.com/python/python-docs-theme/actions/runs/5919197594?pr=148

2.

For commits to main we publish to TestPyPI. This helps us know the publishing process is nicely oiled to avoid surprises on release day. It uses the packages that were already created in step 1.

3.

For published GitHub releases we publish to the real PyPI. Also uses packages created in step 1.

---

TODO

• [n/a] Set up trusted publishing on TestPyPI:
  1. Go to https://test.pypi.org/manage/project/python-docs-theme/settings/publishing/
  2. Add a new publisher like:

screenshot

• Set up trusted publishing on PyPI:
  1. Go to https://pypi.org/manage/project/python-docs-theme/settings/publishing/
  2. Add a new publisher as above

@willingc I see you're the only maintainer for https://test.pypi.org/project/python-docs-theme/, please could you set up the above for TestPyPI? It's probably a good opportunity to add some extra owners/maintainers such as @Mariatta, to improve the bus factor.

@willingc You're also one of the maintainers for https://pypi.org/project/python-docs-theme/, so perhaps you could do PyPI too?

[AA-Turner]

I've reviewed this back-to-front -- sorry. I'm a tad uncertain if we should be going with this though. Overarching thoughts:

• I think that the 'check the package is alright' step could happen via a linting run, e.g. just running twine check, which avoids the kerfuffle with uploading artefacts & to Test PyPI
• On Test PyPI, I'm unconvinced that we should use it for every commit -- we know that the package works, and there are limited resources on Test PyPI, even if the distribution is small, so I think we ought be conservative in what we upload.
• I'm unsure what's behind hynek/build-and-inspect-python-package -- whilst I could go and look it up, it adds a layer of indirection when (to my understanding) this workflow should be replicating the release steps one would run locally -- e.g. python -m build; twine upload. Making it explicit what's going on I think is useful to observability and just having fewer moving parts in general (I'd have the same objection to pypa/gh-action-pypi-publish, save that it implements the OIDC flow for us).

Sorry again for several comments, but I hope this outlines my position to some extent.

A

[hugovk]

• I think that the 'check the package is alright' step could happen via a linting run, e.g. just running twine check, which avoids the kerfuffle with uploading artefacts & to Test PyPI

There is no linter that does all the checks that PyPI does.

twine check only does a subset of the checks. I've seen packages pass twine check yet fail to upload.

It's no kerfuffle, this PR adds automation so machines take care of it for us. 🤖

[hugovk]

• On Test PyPI, I'm unconvinced that we should use it for every commit -- we know that the package works, and there are limited resources on Test PyPI, even if the distribution is small, so I think we ought be conservative in what we upload.

We only know for sure the package works and can be uploaded by uploading it.

These pure Python packages are very small, especially when compared to those with C extensions which may have 60+ multi-MB wheels.

Resources are indeed limited on TestPyPI, and I doubt we'll ever reach the project limit. If we do, we can safely delete old uploads.

Disregard, removed: #148 (comment)

[hugovk]

• I'm unsure what's behind hynek/build-and-inspect-python-package -- whilst I could go and look it up, it adds a layer of indirection when (to my understanding) this workflow should be replicating the release steps one would run locally -- e.g. python -m build; twine upload. Making it explicit what's going on I think is useful to observability and just having fewer moving parts in general (I'd have the same objection to pypa/gh-action-pypi-publish, save that it implements the OIDC flow for us).

I find the build and content summaries can be useful. It does lots of extra lint checks beyond twine check. It makes packages avaliable before upload, so we can test them locally if we like.

Best of all, we don't need to maintain it! We can benefit from @hynek and other contributors maintaining and improving it. We can also re-use it in other repos without duplicating all the extra checks across our repos.

(Here's the source: https://github.com/hynek/build-and-inspect-python-package)

[hugovk]

I removed the TestPyPI upload because we hardcode the version and that would have meant the second and subsequent uploads would fail because you cannot reuse filenames on PyPI.

I don't want to expand the scope to propose using something like hatch-vcs or setuptools_scm in this PR, it's more important to get trusted publishing in place. 👍

[AA-Turner]

Thanks for your patience with my questions--excited to see trusted publishing enabled for python-docs-theme!

A

[willingc]

@hugovk I'm just re-engaging with core development a bit more now that I'm semi-retired. Let me know what you need me to do re: pypi. Thanks.

[hugovk]

Congrats on the semi-retirement!

1. Log in to pypi.org
2. Go to https://pypi.org/manage/project/python-docs-theme/settings/publishing/
3. Add a new publisher by filling in the form like this (leave the last env name box blank):

That's it!

[willingc]

@hugovk Thanks for the great instructions. It's now completed. ✨

[hugovk]

Thank you!

Let's merge this now, and I can try out a release with #153 next week, after 3.12 is out.

[hugovk]

Success!

I merged #153, updated version and changelog in #157 and followed these steps (which I'll update in CONTRIBUTING.md):

• Update CHANGELOG.rst
• Bump version (YYYY.MM) in pyproject.toml
• Commit
• Push to check tests pass on GitHub Actions
• Go to https://github.com/python/python-docs-theme/releases
• Click "Draft a new release"
• Click "Choose a tag", type the next version (for example "2023.09") and select "Create new tag: 2023.9 on publish"
• Click "Generate release notes" and amend as required
• Click "Publish release"
• Check the tagged GitHub Actions build has deployed to PyPI

The tag/release:

• https://github.com/python/python-docs-theme/releases/tag/2023.09

The CI build and deploy:

Screenshot

Screenshot

• https://github.com/python/python-docs-theme/actions/runs/6407907553

The release on PyPI:

• https://pypi.org/project/python-docs-theme/2023.9/

[hugovk]

(which I'll update in CONTRIBUTING.md):

Please see PR #158.

[Mariatta]

Great work. Thanks!