Python: 3.9 or later features to implement

[jpmckinney]

Use https://pypi.org/project/pyupgrade/ when upgrading

3.7 and 3.8 features have been reviewed and applied to all repositories.

3.7

https://www.python.org/downloads/release/python-370/
https://docs.python.org/3.7/whatsnew/3.7.html

3.7 features

• http.server accepts a --directory argument (e.g. update Sphinx docs instructions)
• The database argument of sqlite3.connect can be a path-like object (can remove str() in kingfisher-archive)
• Data classes (e.g. Item classes in kingfisher-collect and Metadata class in kingfisher-archive)
  • Created Consider using @dataclass in items.py kingfisher-collect#995
  • Can be considered as part of Use pydantic instead of JSON Schema validation where appropriate #52
  • I searched for simple __init__ methods using self\.(\S+) = \1\n +self\.(\S+) = \2\n +self\.(\S+) = \3, but it only turned up one case (ocdskit.schema.Field).
• subprocess.run: capture_output can replace passing subprocess.PIPE as stdout and stderr arguments
  • Leaving as-is (only used in 3 places, and only for stdout), since capture_output is just sugar.
• importlib.resources, intended for migration from pkg_resources.
  • We only use pkg_resources in test_requirements.py, where we also use its feature of parsing requirements files.
• Improvements to asyncio, potentially relevant to Kingfisher Collect

3.8

https://www.python.org/downloads/release/python-380/
https://docs.python.org/3.8/whatsnew/3.8.html

Testing

• Change application repositories to be tested against 3.8 (Upgrade applications to Python 3.10 deploy#346)
• All packages should already be tested against multiple versions (can double-check)

3.8 features

• Assignment expressions, for the cases described in the link
  • As documented: "Try to limit use of the walrus operator to clean cases that reduce complexity and improve readability.". I did a quick search for (\S+) = re\..+\n +if \1 and there were only 5 matches.
• f-strings support = for self-documenting expressions and debugging could be nice for some __repr__ methods (if not already replaced by data classes from 3.7) and log messages
  • Can find cases with (\S+)={\1, but this is mostly useful for print debugging, to see the value of a variable – it shouldn't be used to e.g. construct URLs where the variable name happens to match the query string parameter
• A @functools.cached_property decorator can be used instead of the pattern of using private variables to cache properties, and instead of https://pypi.org/project/cached-property/
  • Opened issue on lib-cove
• @lru_cache() can be simply @lru_cache
  • Covered by pyupgrade
• Document python -m json.tool --json-lines in relevant OCDS documentation for data users.
  • Added to OCDS Kit documentation

[jpmckinney]

3.9

https://www.python.org/downloads/release/python-390/
https://docs.python.org/release/3.9.0/whatsnew/3.9.html

• union operators (more) added to dict (can use instead of copy() followed by update() or d[k] = v)
• str.removeprefix(prefix) instead of \[len.+: (regex)
• str.removesuffix(prefix)
• @cache instead of @lru_cache(maxsize=None)

3.10

Ubuntu 22.04 will install 3.10.

https://www.python.org/downloads/release/python-3100/
https://docs.python.org/release/3.10.0/whatsnew/3.10.html

• structural pattern matching instead of a long if/elif/elif... or complex ast traversal like in test_requirements.py

3.11

https://www.python.org/downloads/release/python-3110/
https://docs.python.org/release/3.11.0/whatsnew/3.11.html

• Nothing notable. Speed improvement.

Need to upgrade:

• orjson
• six (cove)
• pyparsing (matplotlib, httplib2)
• numpy (matplotlib)
• psycopg2 (maybe)
• pip-compile -P coverage -P coveralls requirements_dev.in

Also:

• Run pyupgrade
• Run pytest -W error (in non-unittest/Django projects)
• In Django projects, run ./manage.py migrate --check to see if imports succeed.

[jpmckinney]

Most projects have been updated to Python 3.10 (if running on the host on Ubuntu 22.04) or Python 3.11 (if running in Docker).

Exceptions include credere-backend and the standard/profiles. Also, kingfisher-colab follows the Python version in Google Colaboratory.

Can find others (excluding GitHub Actions workflows from packages, and PyPy) with:

python(?!-version: \[3\.[78]).*(?<!pypy-)3\.(?!10|11)\d

Exclude these files:

-.readthedocs.yaml,-locale/*