desiutil.setup may be incompatible with recent versions of setuptools

[weaverba137]

During recent documentation tests, imports of desiutil.setup triggered errors related to importing setuptools.

Since this is likely related to the deprecation of setup.py, we may need to abandon backward-compatible support for desiutil enhancements to setup.py sooner rather than later.

We already have replacements for the functionality. The only thing needed here is a decision to abandon backward-compatibility.

Full traceback:

WARNING: autodoc: failed to import module 'setup' from module 'desiutil'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module
    return importlib.import_module(modname)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/benjamin.weaver/Documents/Code/git/desihub/desiutil/py/desiutil/setup.py", line 20, in <module>
    from setuptools import Command
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/__init__.py", line 18, in <module>
    from setuptools.dist import Distribution
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/dist.py", line 41, in <module>
    from setuptools.config import setupcfg, pyprojecttoml
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/__init__.py", line 10, in <module>
    from . import setupcfg
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/setupcfg.py", line 23, in <module>
    from . import expand
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/expand.py", line 53, in <module>
    from setuptools.dist import Distribution  # noqa
ImportError: cannot import name 'Distribution' from partially initialized module 'setuptools.dist' (most likely due to a circular import) (/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/dist.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 88, in import_object
    module = import_module(modname, warningiserror=warningiserror)
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 68, in import_module
    raise ImportError(exc, traceback.format_exc()) from exc
ImportError: (ImportError("cannot import name 'Distribution' from partially initialized module 'setuptools.dist' (most likely due to a circular import) (/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/dist.py)"), 'Traceback (most recent call last):\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module\n    return importlib.import_module(modname)\n  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import\n  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load\n  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked\n  File "<frozen importlib._bootstrap_external>", line 883, in exec_module\n  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed\n  File "/Users/benjamin.weaver/Documents/Code/git/desihub/desiutil/py/desiutil/setup.py", line 20, in <module>\n    from setuptools import Command\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/__init__.py", line 18, in <module>\n    from setuptools.dist import Distribution\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/dist.py", line 41, in <module>\n    from setuptools.config import setupcfg, pyprojecttoml\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/__init__.py", line 10, in <module>\n    from . import setupcfg\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/setupcfg.py", line 23, in <module>\n    from . import expand\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/expand.py", line 53, in <module>\n    from setuptools.dist import Distribution  # noqa\nImportError: cannot import name \'Distribution\' from partially initialized module \'setuptools.dist\' (most likely due to a circular import) (/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/dist.py)\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module
    return importlib.import_module(modname)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/benjamin.weaver/Documents/Code/git/desihub/desiutil/py/desiutil/setup.py", line 20, in <module>
    from setuptools import Command
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/__init__.py", line 36, in <module>
    __version__ = setuptools.version.__version__
AttributeError: partially initialized module 'setuptools' has no attribute 'version' (most likely due to a circular import)

[sbailey]

To abandon backwards compatibility, we need:

• summary table of "instead of running python setup.py blat, run blatfoo instead" (see attempt below)
• confirm that all of those work on every desi package (see desitest/nersc.pyL45 for list)
  • e.g. currently pytest doesn't work as-is on specsim and fiberassign and possibly others
• update GitHub actions configs for every package to use only the new commands
• update all wiki docs, READMEs, etc. to use only the new commands

After those have been completed, then we can remove support for setup.py itself for all packages en-masse, but let's not remove setup.py package-by-package until the new commands work everywhere so that we don't have to keep track of which packages require the old vs. new commands.

@weaverba137 please edit this comment to correct/expand this table as needed:

old command	new command
python setup.py test	pytest
python setup.py version	desi_update_version
python setup.py version --tag X.Y	desi_update_version -t X.Y
python setup.py build_sphinx	sphinx-build doc/ build/sphinx/html (?)
python setup.py install	pip install . (?)
python setup.py develop	pip install --editable . (?)
python setup.py build_ext --inplace	??? (used by fiberassign and specex)

[weaverba137]

The table already exists: https://desiutil.readthedocs.io/en/latest/helpers.html. Is that sufficient?

[weaverba137]

To be clear, we will still need python setup.py build_ext --inplace for fiberassign and specex. I'm only proposing abandoning the enhancements provided by desiutil.setup, not necessarily the top-level setup.py file in packages.

[weaverba137]

The exception above only appeared when compiling documentation with Sphinx. Pinning Sphinx<7 in the test suite eliminates this error.

We still need to do the task of eliminating desiutil from various setup.py files, but for the purposes of desiutil, there is a workaround that will allow documentation builds to succeed.