Add persistent_tasks test

[timholy]

This checks whether a package spawns persistent Tasks that might block precompilation of dependents of the package. It also provides a convenience utility, test_persistent_tasks_deps, for checking all the dependencies of a given package, to simplify discovery. If one runs one of the tests in this PR without annotating one as an expected failure, here is the output on Julia 1.10-beta:

julia> Aqua.test_persistent_tasks_deps(getid("UsesBoth"))
  Generating  project jl_jqh8hMTOr6:
    /tmp/jl_jqh8hMTOr6/Project.toml
    /tmp/jl_jqh8hMTOr6/src/jl_jqh8hMTOr6.jl
  Activating project at `/tmp/jl_jqh8hMTOr6`
   Resolving package versions...
    Updating `/tmp/jl_jqh8hMTOr6/Project.toml`
  [94ae9332] + TransientTask v0.1.0 `~/.julia/dev/Aqua/test/pkgs/PersistentTasks/TransientTask`
    Updating `/tmp/jl_jqh8hMTOr6/Manifest.toml`
  [94ae9332] + TransientTask v0.1.0 `~/.julia/dev/Aqua/test/pkgs/PersistentTasks/TransientTask`
  Generating  project jl_FI7OrSlgVA:
    /tmp/jl_FI7OrSlgVA/Project.toml
    /tmp/jl_FI7OrSlgVA/src/jl_FI7OrSlgVA.jl
  Activating project at `/tmp/jl_FI7OrSlgVA`
   Resolving package versions...
    Updating `/tmp/jl_FI7OrSlgVA/Project.toml`
  [e5c298b6] + PersistentTask v0.1.0 `~/.julia/dev/Aqua/test/pkgs/PersistentTasks/PersistentTask`
    Updating `/tmp/jl_FI7OrSlgVA/Manifest.toml`
  [e5c298b6] + PersistentTask v0.1.0 `~/.julia/dev/Aqua/test/pkgs/PersistentTasks/PersistentTask`
PersistentTask [e5c298b6-d81d-47aa-a9ed-5ea983e22986] persistent_tasks: Test Failed at /home/tim/.julia/dev/Aqua/src/persistent_tasks.jl:77
  Expression: fails ⊻ precompile_wrapper(result, tmax)

Stacktrace:
 [1] macro expansion
   @ Aqua ~/.julia/juliaup/julia-1.10.0-beta2+0.x64.linux.gnu/share/julia/stdlib/v1.10/Test/src/Test.jl:672 [inlined]
 [2] macro expansion
   @ Aqua ~/.julia/dev/Aqua/src/persistent_tasks.jl:77 [inlined]
 [3] macro expansion
   @ Aqua ~/.julia/juliaup/julia-1.10.0-beta2+0.x64.linux.gnu/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
 [4] test_persistent_tasks(package::PkgId; tmax::Int64, fails::Bool)
   @ Aqua ~/.julia/dev/Aqua/src/persistent_tasks.jl:75
Test Summary:                                                             | Pass  Fail  Total   Time
/home/tim/.julia/dev/Aqua/test/pkgs/PersistentTasks/UsesBoth/Project.toml |    1     1      2  10.2s
  TransientTask [94ae9332-58b0-4342-989c-0a7e44abcca9] persistent_tasks   |    1            1   2.5s
  PersistentTask [e5c298b6-d81d-47aa-a9ed-5ea983e22986] persistent_tasks  |          1      1   7.7s
ERROR: Some tests did not pass: 1 passed, 1 failed, 0 errored, 0 broken.

Closes JuliaLang/julia#50914

CC @vtjnash

[codecov]

Codecov Report

Merging #174 (3f15ae5) into master (69a7655) will decrease coverage by 0.52%.
Report is 1 commits behind head on master.
The diff coverage is 86.20%.

❗ Current head 3f15ae5 differs from pull request most recent head 77f7fd7. Consider uploading reports for the commit 77f7fd7 to get more accurate results

@@            Coverage Diff             @@
##           master     #174      +/-   ##
==========================================
- Coverage   89.63%   89.12%   -0.52%     
==========================================
  Files          10       11       +1     
  Lines         415      570     +155     
==========================================
+ Hits          372      508     +136     
- Misses         43       62      +19     

Flag	Coverage Δ
unittests	89.12% <86.20%> (-0.52%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
src/Aqua.jl	72.41% <66.66%> (-27.59%)	⬇️
src/persistent_tasks.jl	87.27% <87.27%> (ø)

... and 9 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[lgoettgens]

Overall, this looks like a good improvement for Aqua. Thanks! Unfortunately, the tests did not succeed and the reviewdog has some annotations. I have added some further remarks below.

[fingolfin]

Thank @timholy . Unfortunately CI tests fail in Julia 1.4-1.10 due to a CI timeout -- it seems the tests get suck somehow

[timholy]

Now all passing except Windows. I will investigate later, other duties call.

[timholy]

Yay, green.

[fingolfin]

Thank you very much for this wonderful contribution.

I have one remaining concern: there is an established pattern for signalling a broken test case, using a broken kwarg; it would be nice if this PR also used it, instead of using fails for essentially the same purpose. Would you mind doing that? If not, I'll try to make the necessary adjustments but it may take me a while to find the time (sorry)

[timholy]

I changed the name to broken but I'm no longer sure that was the right thing to do. Is broken intended to used by packages that are using Aqua for their tests? I.e., "gosh, I want this Aqua test to pass, but it doesn't, some day I should fix my package." In contrast, fails was purely for veracity of Aqua's own internal testing; it's not intended to be used by "consumers" of Aqua.

[timholy]

I see a "needs changelog" label but I don't see a "NEWS" file?

[lgoettgens]

I see a "needs changelog" label but I don't see a "NEWS" file?

It has only been introduced this week. If you rebase your branch onto latest master, you should find a CHANGELOG.md.

[lgoettgens]

I'll later have a look at your issue with broken and fails

[lgoettgens]

I have slightly altered the interface. Many of the issues here can easily be solved by splitting test functions up into one computing something and another one checking it with @test.
has_persistent_tasks now returns a boolean. test_persistent_tasks just checks this boolean (depending of a kwarg broken with @test or @test_broken). Internal testing just tests has_persistent_tasks.
Similar to test_persistent_tasks_deps: I changed it to a function returning a vector of the failing packages instead. This is almost as readable as an output but gives users access to the results (if they want to use it somehow), and makes testing this a lot cleaner.

Similar adaptations are happening one after the other to the other, already for longer time existing tasks as well. I hope these changes are fine for you. Please check that I didn't break everything. Thanks!

[lgoettgens]

I noticed some caveats/errors with this new test. Aqua tests itself, and when enabling test_persistent_tasks, that task fails. That may possibly be some bug in the persisten_tasks code.

Furthermore, find_persistent_tasks_deps fails if a dependency is in the sysimg. For example, try to call it with Aqua as its argument.

It would be great if you could look into these two things and fix them.

[lgoettgens]

Rebased this on top of the 0.7.3 release

[lgoettgens]

I found the culprits (see comments below). Furthermore I already pushed a quality of life change, that restores the active environment after the test.

@timholy please have a look, if this looks good to you. If the suggestions get committed, this is good to go from my pov.

[timholy]

Sorry for my long absence here, and many thanks for carrying this forward. Just to check (since there have been many commits), are the comments in #174 (review) still relevant? They seem like nice changes to me. If you want I can update this, or feel free to do so yourself. Aside from some travel later this week, I am getting back to a point of having time to finish this off.

[lgoettgens]

Sorry for my long absence here, and many thanks for carrying this forward. Just to check (since there have been many commits), are the comments in #174 (review) still relevant? They seem like nice changes to me. If you want I can update this, or feel free to do so yourself. Aside from some travel later this week, I am getting back to a point of having time to finish this off.

Yes, these comments/suggestions are still relevant. I just rebase this onto master to resolve some conflicts. I think accepting all suggestions should resolve the remaining ci failures

[lgoettgens]

You don't need to worry about the failures in julia 1.3 and before, as we will drop support anyway (see #216).

[timholy]

Thanks for all the help, @lgoettgens, and others for their reviews!