Unifying tool lockfiles and user lockfiles?

[benjyw]

Kicking off a thread to discuss how Python tool lockfiles are different from user lockfiles, and why, and whether we can simplify and unify these mechanisms.

This is particularly in relation to export, which currently requires a lot of jumping through hoops in order to support the --resolve= flag uniformly for user and tool lockfiles.

Basically, for a user lockfile it is easy to go from resolve name to lockfile content. But for a tool lockfile, we have to go via ExportPythonToolSentinel -> ExportPythonTool -> PexRequest

As far as I can tell, this is so that tools can tinker with the interpreter constraints in various ways. E.g.,

pants/src/python/pants/backend/python/lint/black/subsystem.py

Line 151 in 7251de9

constraints = await _black_interpreter_constraints(black, python_setup)

.

But I'm wondering if this is strictly necessary? There are already interpreter constraints baked into the lockfile, and if the repo has custom interpreter constraints on the tool's subsystem then they must regenerate the lockfile anyway.

So a lot of complexity seems to exist only to support the feature of automatically adopting python3.8 interpreter constraints for black and mypy (and maybe there are other examples?) without forcing you to regenerate the 3.7+ lockfile we ship with. And I'm wondering if it's worth it. We could, for example, ship a python3.8+ tool lockfile, and if you must run on 3.7 then you must generate a tool lockfile?

But, I'm also wondering if I'm fully understanding this, and if I am missing more reasons why this complexity is needed. I have a feeling I'm not seeing the full picture.

[benjyw]

What I'd like to get to is to have one unified lockfile mechanism for tools and user lockfiles. Where you can even do things like resolve a tool from a user lockfile (e.g., pytest, which otherwise has to appear in both).

[ryanking]

I think this is related to an issue we see, which is that we have to put tools and related libraries (like type stubs) in our default resolve, so that their version stay consistent.

[benjyw]

Note that this relates to this feature request: #17721. Implementing that, and other export-related features, would be substantually simpler if we could streamline tool lockfiles.

[danxmoran]

I think it would be great to unify the concepts of user and tool resolves (not sure if that's exactly the same as unifying the lockfile mechanism, or just very closely related). I have occasionally wanted to mark sources as belonging to a tool's resolve (i.e. to have in-repo pylint plugins belong to the pylint resolve). This could also provide an alternative mechanism for #12449 - instead of having pants.toml point at in-repo addresses, you could set the resolve on your python_requirement targets to be == the target tool's resolve.

[cognifloyd]

Yes. I want to unify the resolve for pylint and pylint-plugins. So, unifying tool and user lockfiles would be an excellent step in that direction.

[cognifloyd]

We are much closer to merging tool and user lockfiles. In 2.18.x, we finally removed support for using export with target specs / without the --resolve option. That means that there is a lot less code that depends on the difference between tool and user lockfiles.

• Shorten the deprecation for export without --resolve. #19000
• Some 2.18.0.dev0 deprecations #19073
• Remove more 2.18.0.dev0 deprecations (follow up to #19073) #19077

What else is required to finish merging tool and user lockfiles?

[cognifloyd]

The only tools that seem to have special logic around the PythonToolExport are:

• bandit: _find_all_unique_interpreter_constraints
• black: _find_all_unique_interpreter_constraints + some
• flake8: _find_all_unique_interpreter_constraints
• mypy: _find_all_unique_interpreter_constraints + some
• pylint: _find_all_unique_interpreter_constraints
• pytest: _find_all_unique_interpreter_constraints

Now that we can run tools from user resolves, do we need to make the tool-specific interpreter constraints calculations somehow usable with the user resolves? Or are we just saying: Use your own resolve and you're on your own?

[benjyw]

I'm mopping up the last of this right now! I think we no longer need any distinction. Tool lockfiles are just user lockfiles pointed to in the tool's install_from_resolve option.

[benjyw]

This has been implemented.