Bump deps (auth, orquesta, lint/test, etc) to support newer python #6268

cognifloyd · 2024-10-24T18:31:22Z

This PR is easiest to review per-commit.

A couple of commits were cherry-picked from #6236. I updated some things farther than #6236 did, so I didn't include the commits that updated those deps. I believe that covers all of the the requirements changes from #6236 except the switch from nose to pytest.

This makes a variety of requirements updates, especially orquesta and deps on lint/test tooling.

Use st2-auth-backend-flatfile recently released to pypi: https://pypi.org/project/st2-auth-backend-flat-file/0.3.0/
Bump orquesta and related packages to latest orquesta commit to pull in python compat updates
Use the latest version of st2flake8 as it supports flake8 7.0.0: https://pypi.org/project/st2flake8/0.2.0/
Drop the MarkupSafe constraint. Jinja2 v3+ has fixed their usage of MarkupSafe to support MarkupSafe 2.1+
Remove lockfiles/pytest.lock and just have pants use lockfiles/st2.lock when running pytest.
Bump bandit

Regenerated Lockfile Summaries

`lockfiles/bandit.lock`

==                    Upgraded dependencies                     ==

  bandit                         1.7.0        -->   1.7.10
  gitpython                      3.1.18       -->   3.1.43
  pbr                            6.0.0        -->   6.1.0
  pyyaml                         6.0.1        -->   6.0.2
  setuptools                     69.2.0       -->   75.2.0
  stevedore                      5.2.0        -->   5.3.0

==                      Added dependencies                      ==

  markdown-it-py                 3.0.0
  mdurl                          0.1.2
  pygments                       2.18.0
  rich                           13.9.3
  typing-extensions              4.12.2

==                     Removed dependencies                     ==

  six                            1.16.0

`lockfiles/flake8.lock`

==                    Upgraded dependencies                     ==

  flake8                         4.0.1        -->   7.0.0
  flake8-copyright               0.2.2        -->   0.2.4
  mccabe                         0.6.1        -->   0.7.0
  pycodestyle                    2.8.0        -->   2.11.1
  pyflakes                       2.4.0        -->   3.2.0
  setuptools                     69.2.0       -->   75.2.0
  st2flake8                      0.1.0        -->   0.2.0

==                     Removed dependencies                     ==

  flake8-polyfill                1.0.2

`lockfiles/pytest.lock`

Removed - merged with lockfiles/st2.lock. (see below)

`lockfiles/st2.lock`

st2-auth-ldap was also updated to a later commit, but it doesn't show in the lockfile diff output.

==                !! Downgraded dependencies !!                 ==

  pytest                         8.3.3        -->   7.0.1

==                    Upgraded dependencies                     ==

  chardet                        3.0.4        -->   5.2.0
  flask                          2.1.3        -->   3.0.3
  markupsafe                     2.0.1        -->   2.1.5
  networkx                       2.8.8        -->   3.1
  orjson                         3.10.7       -->   3.10.10
  redis                          5.1.1        -->   5.2.0
  st2-auth-backend-flat-file     0.2.0        -->   0.3.0
  webob                          1.8.8        -->   1.8.9
  werkzeug                       2.1.2        -->   3.0.4

==                      Added dependencies                      ==

  blinker                        1.8.2
  coverage                       7.4.4
  execnet                        2.1.1
  icdiff                         2.0.7
  pprintpp                       0.4.0
  py                             1.11.0
  py-cpuinfo                     9.0.0
  pygal                          3.0.5
  pygaljs                        1.0.2
  pytest-benchmark               3.4.1
  pytest-cov                     3.0.0
  pytest-forked                  1.6.0
  pytest-icdiff                  0.9
  pytest-xdist                   2.5.0

==                     Removed dependencies                     ==

  exceptiongroup                 1.2.2

Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == orjson 3.10.7 --> 3.10.10 st2-auth-backend-flat-file 0.2.0 --> 0.3.0 webob 1.8.8 --> 1.8.9 Also bumps locked commit for st2-auth-ldap

cognifloyd · 2024-10-24T18:33:00Z

BUILD.tools

@@ -34,7 +35,7 @@ python_requirement(

 python_requirement(
    name="pytest-reqs",
-    resolve="pytest",
+    resolve="st2",


This line (and some changes in pants.toml) moves the pytest versions from lockfiles/pytest.lock into lockfiles/st2.lock.

cognifloyd · 2024-10-24T18:34:17Z

lockfiles/st2-constraints.txt

+# REASON: 7.5 was causing errors with orquesta integration tests
+# NOTE: This was probably interaction w/ nose. Try to remove this after switch to pytest.
+# DROPS RESOLVED VERSION: 7.6.1
+coverage<7.5


I moved this constraint from test-requirements.txt to here, as it is now part of the lockfiles/st2.lock. Then, I put the locked version back in test-requirements.txt.

cognifloyd · 2024-10-24T18:34:58Z

lockfiles/st2-constraints.txt

-# REQUIRED BY: jinja2
-# REASON: Fix MarkupSafe to < 2.1.0 as 2.1.0 removes soft_unicode >=0.23 was from jinja2
-# NOTE: try to remove constraint later.
-# DROPS RESOLVED VERSION: unknown
-MarkupSafe<2.1.0,>=0.23


Jinja2 3+ fixed support for MarkupSafe 2.1+. We're on Jinja2 ~3.1

cognifloyd · 2024-10-24T18:36:12Z

pants.toml

@@ -115,6 +115,7 @@ pants_plugins_interpreter_constraints = "CPython==3.9.*"

 # For tools, we have to include python versions for BOTH st2 and pants-plugins
 tool_interpreter_constraints = "CPython>=3.8,<3.10"
+flake8_interpreter_constraints = "CPython>=3.8.1,<3.10"


flake8 requires python >=3.8.1. It would probably be better to apply that python version constraint to all our lockfiles, but this PR already touches several lockfiles and I didn't want to regenerate all of them.

cognifloyd · 2024-10-24T18:36:57Z

pants.toml

@@ -224,7 +223,7 @@ args = [
 ]

 [pytest]
-install_from_resolve = "pytest"
+install_from_resolve = "st2"


This is the line that tells pants to use lockfiles/st2.lock instead of lockfiles/pytest.lock (which is deleted above).

Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == chardet 3.0.4 --> 5.2.0 networkx 2.8.8 --> 3.1 redis 5.1.1 --> 5.2.0

Lockfile diff: lockfiles/flake8.lock [flake8] == Upgraded dependencies == flake8 4.0.1 --> 7.0.0 flake8-copyright 0.2.2 --> 0.2.4 mccabe 0.6.1 --> 0.7.0 pycodestyle 2.8.0 --> 2.11.1 pyflakes 2.4.0 --> 3.2.0 setuptools 69.2.0 --> 75.2.0 st2flake8 0.1.0 --> 0.2.0 == Removed dependencies == flake8-polyfill 1.0.2

Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == flask 2.1.3 --> 3.0.3 markupsafe 2.0.1 --> 2.1.5 werkzeug 2.1.2 --> 3.0.4 == Added dependencies == blinker 1.8.2

Lockfile diff: lockfiles/st2.lock [st2] == !! Downgraded dependencies !! == pytest 8.3.3 --> 7.0.1 == Added dependencies == coverage 7.4.4 execnet 2.1.1 icdiff 2.0.7 pprintpp 0.4.0 py 1.11.0 py-cpuinfo 9.0.0 pygal 3.0.5 pygaljs 1.0.2 pytest-benchmark 3.4.1 pytest-cov 3.0.0 pytest-forked 1.6.0 pytest-icdiff 0.9 pytest-xdist 2.5.0 == Removed dependencies == exceptiongroup 1.2.2

Lockfile diff: lockfiles/bandit.lock [bandit] == Upgraded dependencies == bandit 1.7.0 --> 1.7.10 gitpython 3.1.18 --> 3.1.43 pbr 6.0.0 --> 6.1.0 pyyaml 6.0.1 --> 6.0.2 setuptools 69.2.0 --> 75.2.0 stevedore 5.2.0 --> 5.3.0 == Added dependencies == markdown-it-py 3.0.0 mdurl 0.1.2 pygments 2.18.0 rich 13.9.3 typing-extensions 4.12.2 == Removed dependencies == six 1.16.0

Test results: >> Issue: [B507:ssh_no_host_key_verification] Paramiko call with policy set to automatically trust the unknown host key. Severity: High Confidence: Medium CWE: CWE-295 (https://cwe.mitre.org/data/definitions/295.html) More Info: https://bandit.readthedocs.io/en/1.7.10/plugins/b507_ssh_no_host_key_verification.html Location: ./st2common/st2common/runners/paramiko_ssh.py:781:8 780 client = paramiko.SSHClient() 781 client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) 782 -------------------------------------------------- Code scanned: Total lines of code: 21204 Total lines skipped (#nosec): 0 Total potential issues skipped due to specifically being disabled (e.g., #nosec BXXX): 0 Run metrics: Total issues (by severity): Undefined: 0 Low: 15 Medium: 13 High: 1 Total issues (by confidence): Undefined: 0 Low: 3 Medium: 13 High: 13 Files skipped (0):

cognifloyd added 2 commits October 24, 2024 10:06

use st2-auth-backend-flatfile release from pypi

4a84805

regenerate lockfiles/st2.lock

0b73a00

Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == orjson 3.10.7 --> 3.10.10 st2-auth-backend-flat-file 0.2.0 --> 0.3.0 webob 1.8.8 --> 1.8.9 Also bumps locked commit for st2-auth-ldap

cognifloyd added external dependency python3 infrastructure: ci/cd maintenance labels Oct 24, 2024

cognifloyd added this to the 3.9.0 milestone Oct 24, 2024

cognifloyd requested review from nzlosh, guzzijones and a team October 24, 2024 18:31

cognifloyd self-assigned this Oct 24, 2024

pull-request-size bot added the size/XXL PR that changes 1000+ lines. You should absolutely split your PR into several. label Oct 24, 2024

cognifloyd commented Oct 24, 2024

View reviewed changes

cognifloyd and others added 11 commits October 24, 2024 14:01

bump orquesta to latest commit

f385ad3

Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == chardet 3.0.4 --> 5.2.0 networkx 2.8.8 --> 3.1 redis 5.1.1 --> 5.2.0

drop extra comment char in test-requirements.txt

731cdd1

Bump linting tools

36e91d7

Update requirements for py3.8, py3.9 and py3.10

69bb9f6

drop MarkupSafe constraint that was only needed for Jinja2<3

5304a20

Lockfile diff: lockfiles/st2.lock [st2] == Upgraded dependencies == flask 2.1.3 --> 3.0.3 markupsafe 2.0.1 --> 2.1.5 werkzeug 2.1.2 --> 3.0.4 == Added dependencies == blinker 1.8.2

add merge conflict magnet

36fddec

dedup pygments dependency definition

f2a72b0

cognifloyd force-pushed the bump-reqs branch from 86b5495 to fab2180 Compare October 24, 2024 19:02

nzlosh approved these changes Oct 24, 2024

View reviewed changes

guzzijones approved these changes Oct 24, 2024

View reviewed changes

guzzijones merged commit 791ce96 into master Oct 24, 2024
29 checks passed

guzzijones deleted the bump-reqs branch October 24, 2024 19:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump deps (auth, orquesta, lint/test, etc) to support newer python #6268

Bump deps (auth, orquesta, lint/test, etc) to support newer python #6268

cognifloyd commented Oct 24, 2024 •

edited

Loading

cognifloyd Oct 24, 2024

cognifloyd Oct 24, 2024

cognifloyd Oct 24, 2024

cognifloyd Oct 24, 2024

cognifloyd Oct 24, 2024

Bump deps (auth, orquesta, lint/test, etc) to support newer python #6268

Bump deps (auth, orquesta, lint/test, etc) to support newer python #6268

Conversation

cognifloyd commented Oct 24, 2024 • edited Loading

Regenerated Lockfile Summaries

lockfiles/bandit.lock

lockfiles/flake8.lock

lockfiles/pytest.lock

lockfiles/st2.lock

cognifloyd Oct 24, 2024

Choose a reason for hiding this comment

cognifloyd Oct 24, 2024

Choose a reason for hiding this comment

cognifloyd Oct 24, 2024

Choose a reason for hiding this comment

cognifloyd Oct 24, 2024

Choose a reason for hiding this comment

cognifloyd Oct 24, 2024

Choose a reason for hiding this comment

cognifloyd commented Oct 24, 2024 •

edited

Loading

`lockfiles/bandit.lock`

`lockfiles/flake8.lock`

`lockfiles/pytest.lock`

`lockfiles/st2.lock`