Switch to hatchling backend #4747

vidipsingh · 2025-01-11T16:09:53Z

Replace setuptools with hatchling as build backend
Add hatch-vcs for Git tag-based versioning
Unify package versioning under a single source

Description

This PR switches PyBaMM's build system from setuptools to hatchling and implements Git tag-based versioning using hatch-vcs. This change simplifies our build process and version management now that PyBaMM is a pure Python package.

Related to: #4742

Type of change

New feature (non-breaking change which adds functionality)
Optimization (back-end change that speeds up the code)
Bug fix (non-breaking change which fixes an issue)

Key checklist:

No style issues: $ pre-commit run (or $ nox -s pre-commit)
All tests pass: $ python -m pytest (or $ nox -s tests)
The documentation builds: $ python -m pytest --doctest-plus src (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ nox -s quick.

Further checks:

Code is commented, particularly in hard-to-understand areas
Tests added that prove fix is effective or that feature works

codecov · 2025-01-11T16:28:34Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.69%. Comparing base (817ac83) to head (407289b).
Report is 1 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #4747   +/-   ##
========================================
  Coverage    98.69%   98.69%           
========================================
  Files          303      303           
  Lines        23256    23256           
========================================
  Hits         22953    22953           
  Misses         303      303

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

agriyakhetarpal

Hi @vidipsingh, thanks for taking this up! I have just a few suggestions. Please remove other references of setuptools from this file (lines 174–189) and across other files as well, and replace them with hatch/hatchling as necessary.

pyproject.toml

tests/unit/test_util.py

pyproject.toml

kratman

There seems to be a lot of changes here that are not necessary for just switching to hatch. It should only be a couple lines.

Switching to hatch also requires changing the release workflow, the update version script, and other tools. There is a bunch of other stuff that needs to change

Saransh-cpp

Agree with @kratman. Switching to hatchling should be one PR, and adding hatch-vcs should be another. @vidipsingh can you restrict this PR to just switching to hatchling and getting rid of setuptool configurations?

The hatch-vcs PR can deal with updating the release workflow. Switching to hatchling will not change the release workflow.

In any case, both these PRs should go in after 25.1

vidipsingh · 2025-01-12T09:01:00Z

Agree with @kratman. Switching to hatchling should be one PR, and adding hatch-vcs should be another. @vidipsingh can you restrict this PR to just switching to hatchling and getting rid of setuptool configurations?

The hatch-vcs PR can deal with updating the release workflow. Switching to hatchling will not change the release workflow.

In any case, both these PRs should go in after 25.1

@Saransh-cpp Okay, I will restrict this PR to just switching to hatchling and will remove the setuptools configurations.

I also wanted to ask where else I need to remove references to setuptools apart from pyproject.toml? I’ve noticed it is referred to in the following files:

CONTRIBUTING.md
noxfile.py
.github/workflows/publish_pypi.yml
docs/source/api/parameters/parameter_sets.rst

Since I’m new to the PyBaMM repo, any guidance or suggestions regarding these changes would be greatly appreciated :)

Saransh-cpp · 2025-01-12T13:19:39Z

Sounds right! Some of the occurrences might have to be preserved, but we can point that out while reviewing the PR.

agriyakhetarpal

Thanks, @vidipsingh, looks good to me now!

agriyakhetarpal · 2025-01-13T20:41:57Z

pyproject.toml

+[tool.repo-review]
+


Suggested change

[tool.repo-review]

Should I remove this line?

agriyakhetarpal · 2025-01-13T20:44:38Z

Could you please rename the PR title, and replace the "Fixes" keyword in the PR description with "Related to" or similar so that the linked issue doesn't get closed, as we are no longer adding hatch-vcs in this PR?

agriyakhetarpal · 2025-01-13T20:45:08Z

@all-contributors please add @vidipsingh for infra

allcontributors · 2025-01-13T20:45:18Z

@agriyakhetarpal

I've put up a pull request to add @vidipsingh! 🎉

Actually, wait, I'm dismissing my review because the references to setuptools across other files still remain: #4747 (comment)

kratman · 2025-01-13T20:50:17Z

pyproject.toml

+    "src/pybamm/**/*.csv",
+    "src/pybamm/**/*.py",
+    "src/pybamm/CITATIONS.bib",
+    "src/pybamm/plotting/mplstyle",


I don't see this file in the repo

See my comment below.

kratman · 2025-01-13T20:50:39Z

pyproject.toml

+include = [
+    "src/pybamm/**/*.txt",
+    "src/pybamm/**/*.md",
+    "src/pybamm/**/*.csv",
+    "src/pybamm/**/*.py",
+    "src/pybamm/CITATIONS.bib",
+    "src/pybamm/plotting/mplstyle",


The comment explaining why this is not automatically generated is left below when the code was moved.

It would be nice if we could switch to this being automatic though

Makes sense, it's automatic for setuptools if include_package_data is set to True. Could you check if a similar option is available for hatchling, @vidipsingh?

It is not automatic for setuptools (you need to specify files in MANIFEST.in, as we do right now), but it is automatic for hatch 😉

See my comment below.

pyproject.toml

Saransh-cpp · 2025-01-13T21:39:42Z

pyproject.toml

 # List of files to include as package data. These are mainly the parameter CSV files in
 # the input/parameters/ subdirectories. Other files such as the CITATIONS file, relevant
 # README.md files, and specific .txt files inside the pybamm/ directory are also included.
 # These are specified to be included in the SDist through MANIFEST.in.


Suggested change

# List of files to include as package data. These are mainly the parameter CSV files in

# the input/parameters/ subdirectories. Other files such as the CITATIONS file, relevant

# README.md files, and specific .txt files inside the pybamm/ directory are also included.

# These are specified to be included in the SDist through MANIFEST.in.

Saransh-cpp · 2025-01-13T21:57:29Z

pyproject.toml

+[tool.hatch.build]
+packages = ["src/pybamm"]
+include = [
+    "src/pybamm/**/*.txt",
+    "src/pybamm/**/*.md",
+    "src/pybamm/**/*.csv",
+    "src/pybamm/**/*.py",
+    "src/pybamm/CITATIONS.bib",
+    "src/pybamm/plotting/mplstyle",


include-package-data was used to include the files from MANIFEST.in in the wheel file.

hatchling includes everything not ignored by .gitignore by default. Therefore, we should just include the required files -

Suggested change

[tool.hatch.build]

packages = ["src/pybamm"]

include = [

"src/pybamm/**/*.txt",

"src/pybamm/**/*.md",

"src/pybamm/**/*.csv",

"src/pybamm/**/*.py",

"src/pybamm/CITATIONS.bib",

"src/pybamm/plotting/mplstyle",

[tool.hatch]

build.targets.sdist.include = [

"src/pybamm",

"CITATION.cff",

]

We should not bundle the CITATION.cff file with the wheel (as it is being done now). We should also get rid off MANIFEST.in.

kratman · 2025-01-21T01:03:55Z

@vidipsingh Can you resolve the conflicts here?

- Replace setuptools with hatchling as build backend - Add hatch-vcs for Git tag-based versioning - Unify package versioning under a single source

- Replace setuptools with hatchling as build backend - Remove setuptools configurations - Revert to version 24.11.2

- Replace setuptools with hatch as build backend - Remove setuptools configurations - Add automatic file handling in hatch config

- Replace setuptools with hatchling as build backend - Add hatch-vcs for Git tag-based versioning - Unify package versioning under a single source

- Replace setuptools with hatchling as build backend - Remove setuptools configurations - Revert to version 24.11.2

- Replace setuptools with hatch as build backend - Remove setuptools configurations - Add automatic file handling in hatch config

kratman · 2025-01-21T19:37:42Z

@vidipsingh As a general rule, it is best not to force push once the review has started. It makes it really hard to see the changes. We squash most PRs (except releases), so the merge commits do not matter

kratman

It looks like you might have reverted a bunch of stuff from the previous reviews

kratman · 2025-01-21T19:40:52Z

pyproject.toml

+# For MLIR expression evaluation (IDAKLU Solver)
+iree = [
+    "iree-compiler==20240507.886",
+]


This should not have been re-added

Suggested change

# For MLIR expression evaluation (IDAKLU Solver)

iree = [

"iree-compiler==20240507.886",

]

kratman · 2025-01-21T19:43:53Z

pyproject.toml

+[tool.hatch.version]
+source = "vcs"


If we are using the hatch versioning, then we have to make a bunch of other changes for the release process. I thought the agreement from the earlier review was to switch to hatch first, then change the versioning

noxfile.py

vidipsingh requested review from martinjrobins, Saransh-cpp, agriyakhetarpal, kratman and arjxn-py as code owners January 11, 2025 16:09

agriyakhetarpal requested changes Jan 11, 2025

View reviewed changes

kratman requested changes Jan 12, 2025

View reviewed changes

Saransh-cpp requested changes Jan 12, 2025

View reviewed changes

agriyakhetarpal added the skip release Do not merge until the next release label Jan 12, 2025

agriyakhetarpal previously approved these changes Jan 13, 2025

View reviewed changes

allcontributors bot mentioned this pull request Jan 13, 2025

docs: add vidipsingh as a contributor for infra #4750

Draft

kratman requested changes Jan 13, 2025

View reviewed changes

vidipsingh changed the title ~~Switch to hatchling backend with hatch-vcs versioning~~ Switch to hatchling backend Jan 13, 2025

Saransh-cpp requested changes Jan 13, 2025

View reviewed changes

kratman removed the skip release Do not merge until the next release label Jan 15, 2025

vidipsingh added 6 commits January 22, 2025 00:37

build: switch to hatchling backend with hatch-vcs

cba8576

- Replace setuptools with hatchling as build backend - Add hatch-vcs for Git tag-based versioning - Unify package versioning under a single source

switch to hatchling without VCS integration

a92c4c7

- Replace setuptools with hatchling as build backend - Remove setuptools configurations - Revert to version 24.11.2

switch to hatch with automatic file handling

98a8e86

- Replace setuptools with hatch as build backend - Remove setuptools configurations - Add automatic file handling in hatch config

build: switch to hatchling backend with hatch-vcs

f3e8030

- Replace setuptools with hatchling as build backend - Add hatch-vcs for Git tag-based versioning - Unify package versioning under a single source

switch to hatchling without VCS integration

385674f

- Replace setuptools with hatchling as build backend - Remove setuptools configurations - Revert to version 24.11.2

switch to hatch with automatic file handling

b3a5de5

- Replace setuptools with hatch as build backend - Remove setuptools configurations - Add automatic file handling in hatch config

vidipsingh force-pushed the switch-to-hatchling-and-hatch-vcs branch from 53cda20 to b3a5de5 Compare January 21, 2025 19:31

Merge branch 'develop' into switch-to-hatchling-and-hatch-vcs

407289b

kratman requested changes Jan 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to hatchling backend #4747

Switch to hatchling backend #4747

vidipsingh commented Jan 11, 2025 •

edited

Loading

codecov bot commented Jan 11, 2025 •

edited

Loading

agriyakhetarpal left a comment

kratman left a comment

Saransh-cpp left a comment

vidipsingh commented Jan 12, 2025

Saransh-cpp commented Jan 12, 2025 •

edited

Loading

agriyakhetarpal left a comment

agriyakhetarpal Jan 13, 2025

vidipsingh Jan 13, 2025

agriyakhetarpal Jan 13, 2025

agriyakhetarpal commented Jan 13, 2025

agriyakhetarpal commented Jan 13, 2025

allcontributors bot commented Jan 13, 2025

kratman Jan 13, 2025

Saransh-cpp Jan 13, 2025

kratman Jan 13, 2025

agriyakhetarpal Jan 13, 2025

Saransh-cpp Jan 13, 2025 •

edited

Loading

Saransh-cpp Jan 13, 2025

Saransh-cpp Jan 13, 2025

kratman commented Jan 21, 2025

kratman commented Jan 21, 2025

kratman left a comment

kratman Jan 21, 2025

kratman Jan 21, 2025

Switch to hatchling backend #4747

Are you sure you want to change the base?

Switch to hatchling backend #4747

Conversation

vidipsingh commented Jan 11, 2025 • edited Loading

Description

Type of change

Key checklist:

Further checks:

codecov bot commented Jan 11, 2025 • edited Loading

Codecov Report

agriyakhetarpal left a comment

Choose a reason for hiding this comment

kratman left a comment

Choose a reason for hiding this comment

Saransh-cpp left a comment

Choose a reason for hiding this comment

vidipsingh commented Jan 12, 2025

Saransh-cpp commented Jan 12, 2025 • edited Loading

agriyakhetarpal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agriyakhetarpal commented Jan 13, 2025

agriyakhetarpal commented Jan 13, 2025

allcontributors bot commented Jan 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Saransh-cpp Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kratman commented Jan 21, 2025

kratman commented Jan 21, 2025

kratman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vidipsingh commented Jan 11, 2025 •

edited

Loading

codecov bot commented Jan 11, 2025 •

edited

Loading

Saransh-cpp commented Jan 12, 2025 •

edited

Loading

Saransh-cpp Jan 13, 2025 •

edited

Loading