Project.toml formatting test not sensible in julia pre-1.7

[lgoettgens]

Pkg.jl my alter a Project.toml file when instantiating before julia 1.7 (cf. JuliaLang/Pkg.jl#3481 (comment)).
Thus, the current test is basically a no-op returning pass on julia 1.6 and before (in CI), as the Project.toml gets formatted when instantiating the project and thus everything is already formatted when Aqua inspects it.

If a package uses package extensions, this is even worse, as the automatic formatting on julia 1.6 and before is incompatible with the one required by the Aqua test. Therefore, this test always fails in these cases.

For the previous discussion, please refer to #105 (comment) and the comments below it.

[fingolfin]

In my opinion, the whole Project.toml formatting test is not sensible, period. In any Julia version. At least not in a project about "quality assurance" -- the order in which things appear in Project.toml are not indicative for quality. There is only one reason to have things ordered, and that is to minimize diffs when using Pkg to add entries to it.

As such, my preferred "fix" for this "problem" would be to remove the Project.toml formatting test completely.

[lgoettgens]

@gdalle @fingolfin Let's please discuss #206, #209, and #210 here until we decide on one of them to keep the discussion in one place

[lgoettgens]

I think the change of default order from julia 1.6 to 1.7 is a good indicator that this test is actually needed. Imagine having two people working on a project together but with different julia versions that enforce different formatting. In this case, even though there is no "real" change to the Project.toml, even no-ops (as add and then rm a dep) bring new large diffs. The Aqua test then decides on one of the two formatting that should be used consistently.

Thus, my preferred solution fix would be to exclude this test from all julia/Pkg.jl versions that modify the Project.toml when running resolve or instantiate (see #210).

[gdalle]

Sounds good to me

[devmotion]

I just ran into this issue again in a private project on Julia 1.9. The setup is a bit special (exported Preferences.jl preferences in the Project.toml - maybe the issue was caused by JuliaPackaging/Preferences.jl#52?) but nevertheless this means I'll just have to disable this check in yet another repo. IMO this Aqua test should just be removed completely since apparently not even libraries such as Pkg and Preferences can agree on a specific order.

[gdalle]

Maybe instead of the other one could just check the presence of various components? And their relations? As in, if there's a target section there must be extras, etc

[lgoettgens]

Ok, so the existence of JuliaPackaging/Preferences.jl#52 (and possibly many similar things in other packages) is a dealbreaker for me. I would thus proceed with #209.
If we want to test a subset of it (alphabetical order in each category, presence/relation of components, etc.), I would suggest to add that as a default-disabled test later in the 0.8.x cycle. I would declare it as "subject to change" so we can gradually add features, and release it as default-enabled in 0.9.

@fingolfin @gdalle Objections?

[gdalle]

I agree that it seems like it does more harm than good these days

[fingolfin]

Yes I agree (unsurprising since #209 is mine). And I would be fully in support for a structural test that reports things like missing compat entries etc., as suggested by @gdalle .

[lgoettgens]

Yes I agree (unsurprising since #209 is mine). And I would be fully in support for a structural test that reports things like missing compat entries etc., as suggested by @gdalle .

Just FYI, missing compat entries are already checked by test_deps_compat. But there is other stuff to check.