Improve test collection and output

[edoardob90]

Changes how the test report is collected and the output message printed to the user.

Beware that the format_assertion_error function might not handle all cases. By default, if no special case is found with the assertion, the message is printed as-is, converted to HTML.

Lines 124-126 of testsuite.py will be irrelevant when the PR on async testing is merged, but I needed them for testing purposes.

[edoardob90]

@despadam It's a small modification of your implementation. Comments/suggestions are welcome!

[baffelli]

Thanks @edoardob90 🙏👍 I will review it tomorrow

[edoardob90]

Thanks @edoardob90 🙏👍 I will review it tomorrow

Thanks. Have a look and try it out, there might be some edge cases I did not handle. Also: we're collecting the full traceback with your TestResult class, but maybe it's not necessary.

We can then change the draft status and do some cleanup/rewriting some bits in a better way.

[baffelli]

Thanks @edoardob90 🙏👍 I will review it tomorrow

Thanks. Have a look and try it out, there might be some edge cases I did not handle. Also: we're collecting the full traceback with your TestResult class, but maybe it's not necessary.

We can then change the draft status and do some cleanup/rewriting some bits in a better way.

It's a great starting point and we did good progress today. We can merge it soon and later change as needed. it is great to have structured information on test failure and not having to extract it from the very verbose stdout

[baffelli]

I would prefer that the formatting happens at the time of construction of the test output.
We could store the excinfo here and call the method later.

[baffelli]

Excellent work @edoardob90. I have a few ideas, with your agreement I will just create a new commit with those.

[baffelli]

I would add a status variable instead of success. This could have the following states:

• Passed
• Failed
• Syntax error
  In this way we can also report syntax errors or any other error that happens during the test collection phase but before the test is run.

[baffelli]

Following changes should be added:

• Suppress all Ipython messages and replace them with the Ipytest output
• Handle syntax error, test collection failure and test results more uniformly through a data class
• Handle output with a single function

[baffelli]

Following changes should be added:

• Suppress all Ipython messages and replace them with the Ipytest output
• Handle syntax error, test collection failure and test results more uniformly through a data class
• Handle output with a single function

A lot of this was achieved today in a monster refactor with @edoardob90 🔥. Missing:

• a single function to format the output

[baffelli]

Added the following:

• Found a cleaner way to suppress tracebacks
• Fixed the missing widget output

[edoardob90]

Tidbits from the last commit:

• Everything, except bundling the output, happens in the TestMagic class
• The final output (test results + solution code) is delegated to the helper class TestResultOutput (perhaps to rename)
• Only one function to format an exception: format_error from the testsuite_helpers.py module

[baffelli]

Great work today @edoardob90 🔥. Let's merge this soon and get back to the course materials

[edoardob90]

In the last commit:

• ResultCollector now catches an exception raised when running a test. It creates a TestCaseResult with outcome=TestOutcome.TEST_ERROR. This has to be handled somehow in the output, and it's matched by the case IPytestOutcome.FINISHED because the tests indeed finished running.
• If the call phase runs with no exceptions, then we are able to capture stdout or stderr. They are formatted in two collapsible sections, with each stdout/err in a separate box.

It might not be necessary to capture the outputs/errors from all the tests, but we cannot know in advance if some output is tied to the input of the solution function. For example, in this case, it's redundant:

def solution_power2(x: int) -> int:
    print("name")
    return x * 2

because the same string will be printed every time. While in the following case it's not:

def solution_power2(x: int) -> int:
    print("x is", x)
    return x * 2

I don't see another way of dealing with this.

[baffelli]

Thanks @edoardob90 , we did good progress today! Now missing:

• finish display all possible outcomes
• merge
• add the async tests (Allow the function under test to be asynchronous #159 )

[baffelli]

Shall we merge this?

[edoardob90]

There's still one TODO left: catching if an exception is raised in the solution function. Check out #165 .

Right now, if that happens, the test outcome is set to TEST_ERROR, while it should be FAILED.

[edoardob90]

Great! 🚀 Thanks @baffelli @despadam 👏🏻