🐛[BUG]: Install from source fails on Python 3.12 due to NumPy pin

[peterdsharpe]

Version

0.9.0a0

On which installation method(s) does this occur?

Source

Describe the issue

Overview

Modulus installation from source fails on Python 3.12 during installation of its NumPy dependency. I confirm that Modulus install does work correctly on Python 3.11, which suggests that this issue is 3.12-specific.

The root cause is pretty straightforward: Modulus pins numpy>=1.22.4,<1.25 in pyproject.toml, which causes NumPy 1.24.4 to be installed. NumPy 1.24 only supports Python 3.8-3.11, as it depends on pkgutil.ImpImporter, which was removed in Python 3.12 following its deprecation.

This issue is low-priority, but important to track. Given that the default public Anaconda/Miniconda is now Python 3.12, this will start impacting more users.

Related issue: #383 [FEA]: Support for newer numpy

Steps to reproduce

• Running locally
• OS: Ubuntu 24.04.1 LTS, via WSL2 on a Windows 11 machine
• Miniconda, with Python 3.12, for Linux (latest public release)
• Clone https://github.com/NVIDIA/modulus/
• In directory, run pip install .
• The traceback below results.

Suggested fix

A few options:

• Quick-fix: Indicate in the README that supported Python versions are 3.10 and 3.11.
• Longer-term: remove upper version pin on NumPy <1.25, after determining the original motivation for this.

Full Traceback

Result:
```bash
(base) root@NV-PeterSharpe:/mnt/c/Users/psharpe/GitHub/modulus# pip install -e .
Obtaining file:///mnt/c/Users/psharpe/GitHub/modulus
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: torch>=2.0.0 in /root/miniconda3/lib/python3.12/site-packages (from nvidia-modulus==0.9.0a0) (2.5.1)
Collecting numpy<1.25,>=1.22.4 (from nvidia-modulus==0.9.0a0)
  Downloading numpy-1.24.4.tar.gz (10.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.9/10.9 MB 499.4 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      Traceback (most recent call last):
        File "/root/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/root/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/root/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
          backend = _build_backend()
                    ^^^^^^^^^^^^^^^^
        File "/root/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
          obj = import_module(mod_path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "/root/miniconda3/lib/python3.12/importlib/__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 995, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "/tmp/pip-build-env-6t2w3ibq/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
          import setuptools.version
        File "/tmp/pip-build-env-6t2w3ibq/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
          import pkg_resources
        File "/tmp/pip-build-env-6t2w3ibq/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[chrismbryant]

@ktangsali (or @NickGeneva) do you remember why you originally updated the numpy dependency from numpy>=1.22.4 to numpy>=1.22.4,<1.25 on 2023-09-22 here: e3bd802?

Some context that may be helpful is that numpy released version 1.25.0 on 2023-06-17. When you changed the numpy dependency specification here, the latest version that was released was version 1.26.0 (released 2023-09-16) (see: https://pypi.org/project/numpy/#history).

[NickGeneva]

I also ran into this issue the other day here with modulus blocking python 3.12 installs NVIDIA/earth2studio#150

I suspect we were encountering an issues with one of the other dependencies and numpy 1.26.0 so locked it at the time, I do not recall precise details however.

I'd recommend we remove the upper bound this seems to operate fine now.

[saydemr]

I also ran into this issue the other day here with modulus blocking python 3.12 installs NVIDIA/earth2studio#150

I suspect we were encountering an issues with one of the other dependencies and numpy 1.26.0 so locked it at the time, I do not recall precise details however.

I'd recommend we remove the upper bound this seems to operate fine now.

At the time torch was not compatible with newer versions of numpy (see this for example) even though somehow errors were triggered by the use of newer versions of numpy...

[peterdsharpe]

Update on this:

I removed the NumPy pin and am testing the install process on Py3.12. In the meantime, I'm documenting one other (minor) issue here for visibility, which may be helpful for end-users installing Modulus on Python 3.12:

---

One of Modulus' dependencies, nvidia-dali-cuda120, stopped offering pre-built wheels on public PyPI starting with version 1.36.0. This forces pip to try to build DALI from source, which takes a long time (hours on my machine). Instead, you can use NVIDIA's hosted PyPI index, which still has pre-built wheels for DALI up to the latest version (1.43.0 at the time of writing) - saves a lot of time.

So, for end-users, you can install Modulus with:

pip install --extra-index-url https://pypi.nvidia.com nvidia-modulus

Or, for dev installs (cloned locally), one might do:

git clone https://github.com/NVIDIA/modulus.git
cd modulus
pip install --extra-index-url https://pypi.nvidia.com nvidia-dali-cuda120
pip install .[dev]

[ktangsali]

Started this PR to address the issue.

[roy0502]

I have followed the above steps, but why am I still getting the following error:

'Preparing metadata (pyproject.toml) did not run successfully.'