Switch from MultipleFailures to PEP-654 ExceptionGroup

[Zac-HD]

Using the new features from PEP-654 and PEP-678, exception groups and enriching exceptions with notes.

Draft because I'm not quite done updating tests yet - and will want to do some further manual confirmation that the output looks good too - but this will close #3175 when merged.

• Experiment with ExceptionGroup
• Write PEP-678 so we can attach notes (failing examples, etc) to each exception
• Core implementation work (this PR)
• Patch format-exception-only too agronholm/exceptiongroup#8 (traceback.format_exception_only()) makes Pytest show PEP-678 notes
• BuildContext.close() could raise ExceptionGroup for failed cleanup tasks instead of just printing tracebacks
  • check for other loops with a try/except in them.
• We have some tests which check that stdout doesn't contain some value; inspect each usage of capsys, or capture_out() to ensure that they're still correct with use of notes.
• Pytest does not show ExceptionGroup.exceptions (i.e. the actual errors!) unless you use --tb=native Pytest does not show inner exceptions in PEP-654 ExceptionGroups pytest-dev/pytest#9159
  • Disable the report_multiple_bugs reporting when running on old pytest without --tb=native
• (nice-to-have) rich.traceback support [REQUEST] rich.traceback should handle ExceptionGroup and __notes__ Textualize/rich#2238

[Zac-HD]

@HypothesisWorks/hypothesis-python-contributors - with Python 3.11 rc1 next Monday, I'm hoping to merge this soon.

• 🙏 standard code-review for a big change would be appreciated.
• 👂 just try using it! Do you like the new reporting? Has anything gone missing or broken for your use-cases?
• 🤔 I've disabled multiple-bug reporting for Pytest traceback modes which don't support ExceptionGroup yet. Thoughts?

[honno]

Had been using this for a bit last week for a test suite that uses Hypothesis extensively (but not particularly weirdly), and this didn't break anything... take that as you will!

The only change I've felt is yeah, not seeing the multiple tracebacks when there are multiple failures in a typical pytest run.

Leaving sub-exception traceback display for pytest to support properly sounds reasonable to me. Even if pytest did print the tracebacks properly right now, there'd still be a lag in users using it, so it's not like we can escape the disruption. I certainly see for our/your sanity that we don't want to build/maintain some hacky interim solution either.

Just to note, the current --tb=native ASCII tree representation of "exception hierarchy" is nice, but looks a bit jarring for sub-exceptions that has text exceeding your terminals columns.

  + Exception Group Traceback (most recent call last):
  |   ...
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "..............................................
.............................................."
    |   etc.

I suppose this is the kind of stuff that has got you pytest folks interested in adopting Rich heh. In any case, I still think this tree is preferable to how Hypothesis currently displays multiple tracebacks.

EDIT: oh the tree is just the traceback message from ExceptionGroup? ahh

[Zac-HD]

Knowing that it works in practice is super useful, thank you! I know it's well-tested, but whether the tests cover all the use-cases is harder to check 😅

Even if pytest did print the tracebacks properly right now, there'd still be a lag in users using it, so it's not like we can escape the disruption.

Indeed - download stats indicate that ~40% downloads are for the latest version, which was released in April, and based on other projects I track loosely there's something like 30% of downloads usually unpinned.

[Zac-HD]

@honno: https://github.com/data-apis/dataframe-interchange-tests/blob/73e4f74eb53eb04bc9632ea243d8b167c458dbfe/tests/test_from_dataframe.py#L11-L14

Looking at this test, you get exhaustive coverage of A -> B -> A - but it's also possible that there are bugs of the form A -> B -> C -> D -> A! I'd consider pulling out the body of this test into a helper function, and then sharing it with a new test that does multiple conversions with st.lists(st.sampled_from(library_infos), min_size=2, unique=True).

[honno]

Looking at this test, you get exhaustive coverage of A -> B -> A - but it's also possible that there are bugs of the form A -> B -> C -> D -> A! I'd consider pulling out the body of this test into a helper function, and then sharing it with a new test that does multiple conversions with st.lists(st.sampled_from(library_infos), min_size=2, unique=True).

Ayup, I think with just the single round-trips I wrote a gazillion bug reports for adopting libraries, so I kinda forgot to do this heh.