Python 3.11 upgrade #279

dmichaels-harvard · 2023-08-22T19:40:22Z

Bump required PyYAML version to ^6.0.1 to resolve a Mac M1 issue with downstream builds.

As @netsettler notes in discussion on utils PR #277, the PyYAML Change Log says this about major version 6:

6.0 (2021-10-13)

* https://github.com/yaml/pyyaml/pull/327 -- Change README format to Markdown
* https://github.com/yaml/pyyaml/pull/483 -- Add a test for YAML 1.1 types
* https://github.com/yaml/pyyaml/pull/497 -- fix float resolver to ignore `.` and `._`
* https://github.com/yaml/pyyaml/pull/550 -- drop Python 2.7
* https://github.com/yaml/pyyaml/pull/553 -- Fix spelling of “hexadecimal”
* https://github.com/yaml/pyyaml/pull/556 -- fix representation of Enum subclasses
* https://github.com/yaml/pyyaml/pull/557 -- fix libyaml extension compiler warnings
* https://github.com/yaml/pyyaml/pull/560 -- fix ResourceWarning on leaked file descriptors
* https://github.com/yaml/pyyaml/pull/561 -- always require `Loader` arg to `yaml.load()`
* https://github.com/yaml/pyyaml/pull/564 -- remove remaining direct distutils usage

Based on this, he reasons:

It looks like really dropping Python 2 is the major incompatible change, though the "always require Loader arg to yaml.load()" might require some research to make sure we weren't breaking something.

coveralls · 2023-08-22T19:42:29Z

Pull Request Test Coverage Report for Build 6421775176

0 of 0 changed or added relevant lines in 0 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.009%) to 77.65%

Files with Coverage Reduction	New Missed Lines	%
dcicutils/s3_utils.py	1	87.41%

Totals
Change from base Build 6354871296:	-0.009%
Covered Lines:	8477
Relevant Lines:	10917

💛 - Coveralls

netsettler · 2023-08-22T20:02:47Z

CHANGELOG.rst

@@ -7,6 +7,12 @@ Change Log
 ----------


+7.9.0


I think a major version 8.0.0 would be wise, so that if there's an issue, it's easy for someone to use ^7.x. kinds of notation to select stuff that doesn't have this until a fix is ready. Probably not an issue, but that thing about change in required arg worries me.

@willronchetti do you have an opinion on this?

I'm a little worried about that required arg. I note that if you search for calls to yaml.load in 4dn-dcic (which is only one of our 3 orgs), you'll see several calls that don't pass an explicit Loader argument. Will that work or are you forcing code updates? Maybe those updates are fine. If there is an obvious loader, maybe we should have something that someone can call instead.

Yeah, looking at those results as well as for dbmi-bgm and smaht-dac, I don't see any uses within a repo where dcicutils is used (where the version of pyyaml it uses will increments due to this dcicutils change). The only affected repos are cwltools and parliament2 - neither of which has any reference to dcicutils anywhere.

I built a cgap-portal(branch: dcicutils-with-pyyaml-6-and-sheet-utils | version: 14.2.0.1b1) with this dcicutils/pyyaml update and deployed to cgap-supertest and it seems fine (login/logout, create access-key, and ran submit-metadata-bundle).

netsettler

As @willronchetti noted on slack, testing workflows need to be updated to test the right things.

Also, the categories at top of pyproject.toml mention Python versions, so those need to change.

netsettler · 2023-09-20T14:42:38Z

pyproject.toml

@@ -36,7 +36,7 @@ classifiers = [


 [tool.poetry.dependencies]
-python = ">=3.7,<3.10"
+python = ">=3.7,<3.12"


Suggested change

python = ">=3.7,<3.12"

python = ">=3.8,<3.12"

This will also need update of the categories in pyproject.toml, and of the workflows in GA in order to get tests to pass (since 3.7 is now excluded) but also to test new versions we weren't previously testing.

netsettler · 2023-09-20T14:43:13Z

CHANGELOG.rst

+* Changed PyYAML version to ^6.0.1; Mac M1 has issues building 5.4.1 (though 5.3.1 works).
+  See PyYAML 6.0 change log here: https://github.com/yaml/pyyaml/blob/master/CHANGES
+  The only incompatible change seems to be that yaml.load now requires a Loader argument;
+  and searching our repos (4dn-dcic, dbmi-bgm, smaht-dac) the only ones which might be affected


Suggested change

and searching our repos (4dn-dcic, dbmi-bgm, smaht-dac) the only ones which might be affected

and searching our GitHub organizations (4dn-dcic, dbmi-bgm, smaht-dac) the only ones which might be affected

netsettler

Yeah, this seems fine.

netsettler · 2023-09-20T20:22:48Z

pyproject.toml

@@ -29,7 +29,6 @@ classifiers = [

    # Specify the Python versions you support here.
    'Programming Language :: Python :: 3',
-    'Programming Language :: Python :: 3.7',
    'Programming Language :: Python :: 3.8',
    'Programming Language :: Python :: 3.9',


Suggested change

'Programming Language :: Python :: 3.9',

'Programming Language :: Python :: 3.9',

'Programming Language :: Python :: 3.10',

'Programming Language :: Python :: 3.11',

netsettler · 2023-09-20T20:23:13Z

.github/workflows/main.yml

@@ -23,7 +23,7 @@ jobs:
    runs-on: ubuntu-20.04
    strategy:
      matrix:
-        python_version: [3.7, 3.8, 3.9]
+        python_version: [3.8, 3.9, 3.11]


Why not 3.10?

Just to save time in GitHub actions really; not sure 3.10 is in wide usage; can add if you like.

netsettler · 2023-09-20T20:25:20Z

pyproject.toml

@@ -70,7 +69,8 @@ coverage = ">=7.2.3"
 # coveralls = ">=3.3.1"
 flake8 = ">=3.9.2"
 flaky = ">=3.7.0"
-pip-licenses = "^3.5.5"
+pip-licenses = "^4.3.3"
+pip = "23.2.1"


I would order this before pip-licenses, but that's minor.

Is it really a good idea to list pip here? Is poetry undoing the version we load in the Makefile? (In that case, maybe it is necessary to list it here. But it seems like something we may regret having to maintain.

Further, do you really mean to pin it to so specific a version? If so, could you add a comment saying why so we can know when to unpin it?

You're right, I had that (pip) in there temporarily - I will remove it. Thanks.

willronchetti

Approving recent changes for 3.11, all looks good but the pyproject python versions should be updated (take Kent's suggestion)

…sure it passes again

…2.04

…type return from mimetypes.guess_type on Ubuntu 22.04.

dmichaels-harvard added 2 commits August 22, 2023 15:39

Up pyyaml version to ^6.0.1 for Mac M1 issue with build.

7e142b1

Up pyyaml version to ^6.0.1 for Mac M1 issue with build.

ece924a

dmichaels-harvard requested review from willronchetti and netsettler August 22, 2023 19:41

netsettler reviewed Aug 22, 2023

View reviewed changes

dmichaels-harvard added 5 commits August 24, 2023 09:41

CHANGELOG changes.

4a8f051

Merge from master

2e99cf7

version update

2f49bcb

Fix to test_s3_utils from kmp_sheet_utils_schema_hinting branch.

829e5b0

flake8 update

53c8e6d

willronchetti approved these changes Aug 29, 2023

View reviewed changes

dmichaels-harvard added 3 commits September 20, 2023 09:34

Merge from master

69093fa

Update to Python 3.11!

6cc26bc

Update to Python 3.11!

abe2140

netsettler reviewed Sep 20, 2023

View reviewed changes

dmichaels-harvard added 4 commits September 20, 2023 13:08

Update boto3/botocore versions.

a7fa068

Update boto3/botocore versions.

669e51f

CHANGELOG comments

4b70bd2

Update to Python 3.11

927862e

netsettler approved these changes Sep 20, 2023

View reviewed changes

dmichaels-harvard added 4 commits September 20, 2023 15:23

Remove Python 3.7 from pyproject and GA

702aedb

Updated pip-licenses

5b4c84b

Updated pip-licenses

4120faa

update python to 3.9 in static-checks.yml

a25959b

netsettler reviewed Sep 20, 2023

View reviewed changes

Removed pip from pyproject

07bbdd0

willronchetti approved these changes Sep 21, 2023

View reviewed changes

dmichaels-harvard added 2 commits September 21, 2023 11:43

Put pytest to latest version and added pytest-redis to ^3.0.2.

ba13cbf

Update version

7ed1511

dmichaels-harvard added 16 commits September 26, 2023 08:22

Updated boto versions.

88f3b89

Updated boto versions.

3e5cf5f

Added Python 3.10, 3.11 to pyproject Python list

e24b3e0

Update Ubuntu to 22.04 for GA

c760707

Update Ubuntu to 22.04 for GA and poetry update

7b3daaf

back out ubuntu to 22.04 for now to test - ga failing all of a sudden

099a32e

put back update to ubuntu 20.04 -> 22.04 for ga and see if ga fails now

25a4b58

ga failed with ubuntu 22.04 for python 3.11 - put back 20.04 to make …

96cc528

…sure it passes again

put ubuntu 22.04 back for ga and add testing hack in qa_utils.py

97f1173

flake8 error

b226d61

more hacking of qa_utils for troubleshooting ga failure with ubuntu 2…

45064b2

…2.04

Updated comments

bd6a797

Fix to qa_utils for application/vnd.software602.filler.form+xml mime …

96a10cb

…type return from mimetypes.guess_type on Ubuntu 22.04.

Merged in master

af72ea9

poetry update

f7c05b5

Bumped to Python 3.11 in static-checks.yml

27e11b6

dmichaels-harvard changed the title ~~Up pyyaml version to ^6.0.1 for Mac M1 issue with build.~~ Python 3.11 upgrade Sep 29, 2023

Update version to 8.0.0; ready to merge to master.

2699a4d

dmichaels-harvard merged commit ab88729 into master Oct 5, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.11 upgrade #279

Python 3.11 upgrade #279

dmichaels-harvard commented Aug 22, 2023 •

edited by netsettler

Loading

coveralls commented Aug 22, 2023 •

edited

Loading

netsettler Aug 22, 2023

netsettler Aug 22, 2023

netsettler Aug 22, 2023

dmichaels-harvard Aug 23, 2023 •

edited

Loading

dmichaels-harvard Aug 24, 2023

netsettler left a comment

netsettler Sep 20, 2023

dmichaels-harvard Sep 20, 2023

netsettler Sep 20, 2023

netsettler Sep 20, 2023

dmichaels-harvard Sep 20, 2023

netsettler left a comment

netsettler Sep 20, 2023

dmichaels-harvard Sep 28, 2023

netsettler Sep 20, 2023

dmichaels-harvard Sep 28, 2023

netsettler Sep 20, 2023

dmichaels-harvard Sep 20, 2023

willronchetti left a comment

	and searching our repos (4dn-dcic, dbmi-bgm, smaht-dac) the only ones which might be affected
	and searching our GitHub organizations (4dn-dcic, dbmi-bgm, smaht-dac) the only ones which might be affected

Python 3.11 upgrade #279

Python 3.11 upgrade #279

Conversation

dmichaels-harvard commented Aug 22, 2023 • edited by netsettler Loading

coveralls commented Aug 22, 2023 • edited Loading

Pull Request Test Coverage Report for Build 6421775176

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmichaels-harvard Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netsettler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

netsettler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willronchetti left a comment

Choose a reason for hiding this comment

dmichaels-harvard commented Aug 22, 2023 •

edited by netsettler

Loading

coveralls commented Aug 22, 2023 •

edited

Loading

dmichaels-harvard Aug 23, 2023 •

edited

Loading