[FIX] Make Python version explicit in pre-commit workflow

[carmenbianca]

Recently (or recently enough that I've just begun noticing it), the nodeenv dependency had troubles installing. Sometimes. It wasn't very reliably failing.

This is related to ekalinin/nodeenv#328.

By explicitly setting the Python version to something <=3.10, we can safely pretend the problem doesn't exist.

Ideally the Python versions match up to the ones used in test.yml, but they don't, and I fear it might be too intrusive to fix that in this commit.

[yajo]

Do you have any explanation on why the workaround works? 🤔

[carmenbianca]

Faintest idea.

(A part of) the stacktrace in the linked issue:

  File "/usr/local/lib/python3.11/site-packages/nodeenv.py", line 573, in tarfile_open
    tf = tarfile.open(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/tarfile.py", line 1630, in open
    saved_pos = fileobj.tell()
                ^^^^^^^^^^^^
AttributeError: 'bytes' object has no attribute 'tell'

I have to assume that something goes wrong in tarfile_open such that fileobj is passed to tarfile.open(cls, name=None, mode="r", fileobj=None, bufsize=RECORDSIZE, **kwargs) (via *args or **kwargs) as a bytes object instead of a FileIO interface. And I have to assume that this only happens in Python ≥3.11.

In any case, setting an explicit version is Good, Actually. If I had more time than I do, I would research this further and contribute to upstream.

[yajo]

According to ekalinin/nodeenv#328 (comment) it seems this is a network problem, and it seems the next release for nodeenv will include the patch mentioned there as a workaround and as a fix for the flaky error message.

I mean that, AFAICS, probably we'll be still getting the same error every now and then. Or maybe a slightly different one, but still an error. Actually I'm having the same failures on some internal pipelines that still don't use precommix, and all I need is to retry, and see them go green.

[carmenbianca]

All I know is that coopiteasy/addons#262 (which uses a minorly modified oca-addons-repo-template) consistently failed to merge (but did NOT fail the regular tests on a fastforwardable branch) until I set an explicit Python version.

[pedrobaeza]

That error had appeared to me several times (but a lot percentage), but retrying fixes the problem, so I was considering this a glitch.

[yajo]

Just now I hit the error in our internal CI:

$ cat $PRE_COMMIT_HOME/pre-commit.log || true

version information

pre-commit version: 2.16.0
sys.version:
    3.9.9 (main, Nov 19 2021, 00:00:00) 
    [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
sys.executable: /usr/bin/python3
os.name: posix
sys.platform: linux

error information

An unexpected error has occurred: CalledProcessError: command: ('/usr/bin/python3', '-mnodeenv', '--prebuilt', '--clean-src', '/builds/moduon/customer-projects/turbointernacional-odoo/.cache/pre-commit/repoibozj_gv/node_env-14.14.0', '-n', '14.14.0')
return code: 1
expected return code: 0
stdout: (none)
stderr:
     * Install node (14.14.0Traceback (most recent call last):
      File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/usr/lib/python3.9/site-packages/nodeenv.py", line 1043, in <module>
        main()
      File "/usr/lib/python3.9/site-packages/nodeenv.py", line 891, in main
        create_environment(env_dir, opt)
      File "/usr/lib/python3.9/site-packages/nodeenv.py", line 732, in create_environment
        install_node(env_dir, src_dir, opt)
      File "/usr/lib/python3.9/site-packages/nodeenv.py", line 601, in install_node
        download_node(node_url, src_dir, env_dir, opt)
      File "/usr/lib/python3.9/site-packages/nodeenv.py", line 490, in download_node
        tar_contents = io.BytesIO(urlopen(node_url).read())
      File "/usr/lib64/python3.9/http/client.py", line 476, in read
        s = self._safe_read(self.length)
      File "/usr/lib64/python3.9/http/client.py", line 628, in _safe_read
        raise IncompleteRead(b''.join(s), amt)
    http.client.IncompleteRead: IncompleteRead(1089536 bytes read, 32687884 more expected)
    

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/pre_commit/error_handler.py", line 65, in error_handler
    yield
  File "/usr/lib/python3.9/site-packages/pre_commit/main.py", line 396, in main
    return run(args.config, store, args)
  File "/usr/lib/python3.9/site-packages/pre_commit/commands/run.py", line 417, in run
    install_hook_envs(to_install, store)
  File "/usr/lib/python3.9/site-packages/pre_commit/repository.py", line 224, in install_hook_envs
    _hook_install(hook)
  File "/usr/lib/python3.9/site-packages/pre_commit/repository.py", line 82, in _hook_install
    lang.install_environment(
  File "/usr/lib/python3.9/site-packages/pre_commit/languages/node.py", line 97, in install_environment
    cmd_output_b(*cmd)
  File "/usr/lib/python3.9/site-packages/pre_commit/util.py", line 154, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/usr/bin/python3', '-mnodeenv', '--prebuilt', '--clean-src', '/builds/moduon/customer-projects/turbointernacional-odoo/.cache/pre-commit/repoibozj_gv/node_env-14.14.0', '-n', '14.14.0')
return code: 1
expected return code: 0
stdout: (none)
stderr:
     * Install node (14.14.0Traceback (most recent call last):
      File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/usr/lib/python3.9/site-packages/nodeenv.py", line 1043, in <module>
        main()
      File "/usr/lib/python3.9/site-packages/nodeenv.py", line 891, in main
        create_environment(env_dir, opt)
      File "/usr/lib/python3.9/site-packages/nodeenv.py", line 732, in create_environment
        install_node(env_dir, src_dir, opt)
      File "/usr/lib/python3.9/site-packages/nodeenv.py", line 601, in install_node
        download_node(node_url, src_dir, env_dir, opt)
      File "/usr/lib/python3.9/site-packages/nodeenv.py", line 490, in download_node
        tar_contents = io.BytesIO(urlopen(node_url).read())
      File "/usr/lib64/python3.9/http/client.py", line 476, in read
        s = self._safe_read(self.length)
      File "/usr/lib64/python3.9/http/client.py", line 628, in _safe_read
        raise IncompleteRead(b''.join(s), amt)
    http.client.IncompleteRead: IncompleteRead(1089536 bytes read, 32687884 more expected)

As you see, it's a network error. Retrying makes it go ✅ . And this CI is running on python 3.9. My recommendation is to set up a cache, so successful runs cache the pre-commit installation and avoid fetching from this buggy endpoint. Or... well, use precommix 😆 #175.

Closing because, although the issue is real, the fix doesn't seem to be real AFAICS. Please don't hesitate to continue the conversation or reopen if needed.

[carmenbianca]

although the issue is real, the fix doesn't seem to be real AFAICS.

Agreed. Just bad luck on my part regarding the consistent failure.

Even disregarding the linked issue, though, I think it would be good to pin the Python version. When Python 3.12 is released (and GitHub's setup-python switches to 3.12 by default), inevitably pre-commit is going to break, again. We can get ahead of that here.

[carmenbianca]

Pinging @yajo to review the above comment.