CI Conda: Add to known-test-failures

[mkoeppe]

Not all of the added failures are conda-specific, but the 6 instances of CI Conda amplify the error probability (#36694 (comment))

We also add info (ideally a link to the GitHub Issue where the failure is reported) to the failures and arrange for it to be printed by the doctester as part of the [failed in baseline] message.

Marked as blocker so it takes effect in CI runs -- to reduce the noise from failures of the Conda CI that PR authors and reviewers find distracting.

📝 Checklist

• The title is concise, informative, and self-explanatory.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation accordingly.

⌛ Dependencies

• Depends on src/sage/doctest/forker.py: Show '# [failed in baseline]' earlier #36936 (which refactors the [failed in baseline] printing)

[tobiasdiez]

Ignoring the whole file just because there is a single failing test runs against the goal of making the conda tests more stable. Hence motion to close. For the same reason this is not suitable as blocker.

[mkoeppe]

Ignoring the whole file just because there is a single failing test

That's not at all what it does, as we have explained to you at length in #36678.

We have also explained to you at length why your approach of #36372 is not suitable.
It simply has no support by any reviewer.
It is highly inappropriate to use the blocker label to have it take effect in every PR.

[mkoeppe]

As I said in #36372 (comment) and before, Tobias, these continued abuses of your Triage team privileges have to stop.

[kwankyu]

I think that for each record of a file in .github/workflows/ci-conda-known-test-failures.json, there should be a github issue associated with it so that we track the history of the failures in the file.

Is there an issue for

    "sage.parallel.map_reduce": {
        "failed": true
    },

?

[kwankyu]

and for

    "sage.structure.coerce_actions": {
        "failed": true
    },

?

[kwankyu]

Since json does not support comments, we may add an "issue" field so that "issue: 12345" will point to the github issue with the number.

[mkoeppe]

fine with me, but I think i'd prefer a free-form text that can contain an URL or whatever for flexibility.

[kwankyu]

Yes, URL would work better.

[mkoeppe]

i added this info where available to the commit messages, so you can see it via "git blame"

[kwankyu]

fine with me, but I think i'd prefer a free-form text that can contain an URL or whatever for flexibility.

Then

    "sage.structure.coerce_actions": {
        "failed": true
        "comment": "some text, preferably an URL"
    },

for example.

[kwankyu]

i added this info where available to the commit messages, so you can see it via "git blame"

My git blame only shows the sha and the committer's name and the date. But I can see them in the git log (just for this PR).

[kwankyu]

@tobiasdiez Let's go this way. You may want to add those comments.

Matthias is working in #36558 to make these failures in baseline more visible to developers (without failing checks).

[tobiasdiez]

Yeah sure, why wouldn't we want to exclude even more files from the conda tests? For consistency, you should also add a similar file-wide exclusion mechanism for the tests that are known to fail sometimes in the Build & Test.

So stupid, instead of viewing the conda environment as a means to harden the test suit, we now massively swap the failures under the carpet. And that in a way that doesn't even allow to answer the simple question "is this the only example that fails, or are other examples failing as well"!

I also don't understand why I should add these comments. My solution keeps very well track of what example fails. It's one of the major shortcomings of Mathias solution, as I've pointed out multiple times. So at the very least he should do the work himself.

[tobiasdiez]

The concern of one developer to confirm whether the failures marked up in #36960 cover everything is secondary. Monitoring this can be done just using the logs and does not have to rely on signaling test failures to all other developers.

I now went through the logs and added a few more exceptions to #36960. So every failure (that occurred more than once) is now excluded in #36860. Could you please mirror the changes here. Thanks!

[mkoeppe]

Completeness is not part of the scope of this PR.

[mkoeppe]

I now went through the logs and added a few more exceptions to #36960.

Thanks for doing this work. Do you have a script for this - retrieving and parsing logs?

You may be interested in:

• CI: Upload test stats as artifacts, improve output of "List Docker images" #36970

[github-actions]

Documentation preview for this PR (built with commit 8f4bc66; changes) is ready! 🎉

[mkoeppe]

Let's merge this please.

[kwankyu]

OK. Hopefully save us from chronic conda failures in CI.