ConsoleLauncher should output diff of expected and actual String values #3139

carolosf · 2023-01-30T14:14:17Z

As a developer if I am running JUnit tests using the Console Launcher on the command line if two large strings are found to be different and I get "expected:" "but was:" if I want to see what was different I need to copy the strings and manually diff them.

It would be easier if there was --details flag or another flag (I think another flag is a better option) that would show me a diff between the two strings for actionable results. In most cases using an IDE is suitable but in workflows such as Bazel and CI it would be useful if a diff would be displayed.

Apologies if this is already supported didn't find anything obvious in the docs.

marcphilipp · 2023-02-17T11:49:43Z

Potential library to implement this: https://github.com/java-diff-utils/java-diff-utils

@carolosf We've briefly discussed this in our team call today and think this would be feasible since AssertionFailedError from opentest4j carries actual and expected in separate fields.

We'll have to take a closer look first to decide whether --details diff would be the right way of enabling this. Maybe it could even be enabled by default.

carolosf · 2023-02-17T12:08:54Z

@marcphilipp

If it becomes the default behaviour it might break IDEs and plug-ins - which is why I suggested a totally separate flag.

I didn't spend much time on it but I noticed previously that the Bazel plugin for IntelliJ does a regex for "expected but was" and splits the results.

Thanks for discussing though - if implemented it would significantly reduce my debugging time!

marcphilipp · 2023-04-22T12:48:29Z

@carolosf IDEs and build tools usually do not run tests via ConsoleLauncher but use the Launcher API directly so this shouldn't be a problem.

marcphilipp · 2023-05-05T10:56:07Z

Team decision: Change ConsoleLauncher output for test failures to include diff if actual and expected of AssertionFailedError are of type CharSequence.

marcphilipp · 2023-05-05T10:56:32Z

@carolosf Would you be interested to work on a PR for this?

bechte · 2023-07-04T10:24:57Z

Hey @marcphilipp & @carolosf - if ok for you, I'd like to provide a PR for this. Any veto?

carolosf · 2023-07-04T10:57:43Z

@carolosf Would you be interested to work on a PR for this?

Would love to but I don't have any capacity to work on this in the near future right now.

carolosf · 2023-07-04T11:10:02Z

Hey @marcphilipp & @carolosf - if ok for you, I'd like to provide a PR for this. Any veto?

I'm not working on this.

@marcphilipp Anything to add?

marcphilipp · 2023-07-07T10:36:48Z

@bechte Sure, go ahead

bechte · 2023-07-18T18:59:13Z

To introduce a new dependency to a library, I assume that I am extending the list in gradle/libs.versions.toml, e.g.:

java-diff-utils = { module = "io.github.java-diff-utils:java-diff-utils", version = "4.12" }

And with that plus running a build, I am able to add the following to the platform console project:

dependencies {
	/*...*/
	implementation(libs.java.diff.utils)
	/*...*/
}

Questions to the team, FYI: @marcphilipp, @sbrannen

Is that the current approach we are taking on dependencies?
Furthermore, would you like to shadow this dependency?

With this commit, a dependency to https://github.com/java-diff-utils/java-diff-utils is introduced. java-diff-utils is used to create a diff from the expected and the actual result and report it. See #3139

bechte · 2023-07-21T07:21:53Z

I have added some questions within the PR #3397.

Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence To do list: 1. Add diff function dependency 2. Using diff to generate the output desired

XJ114514 · 2024-08-31T22:19:44Z

Hi, I am currently a college student that wants to contribute to Junit 5 project.
Can I do a PR for this issue? Please assign to Xj114514(green avatar.)
As I had just made a commit that successfully located the ConsoleLauncher output and print out expected and actual value as the Team decision stated.

I tested the code by building a console standalone jar to run one test case I wrote.
I spent the entire day to search for it as I knows nothing about Gradle, Junit 5 , and next I will learn how to use the diff library you provide to finish this issue.

ps: Can someone look at my code to see if I did anything wrong? it only change a tiny part of original code.

Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence To do list: 1. Add diff function dependency 2. Using diff to generate the output desired Issue: junit-team#3139

Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence 3. Add diff function dependency 4. Using diff to generate the output desired Issue: junit-team#3139

delete one line of extra code

XJ114514 · 2024-09-01T00:10:09Z

About the auto test case, I can not think a good way to test it, Could you please give me some idea how to implement the execution below?
A fail execute with AssertionFailedError AND the expected and the actual value are both charSequence.

Fixed different profile issue. Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence 3. Add diff function dependency 4. Using diff to generate the output desired ToDo: 1.Automatic test case for diff output Issue: junit-team#3139

Issue: junit-team#3139

Add external diff module to module documentation For unknown reason, the test cases can not find the external module while the gradle can build normally. Issue: junit-team#3139

XJ114514 · 2024-09-14T19:38:41Z

Conclusion of the problems that I had encountered:

4 failed Test cases about adding the external module io.github.javadiffutils as dependency
- All the test cases seem to result in Module Not Found, while the gradle can successfully build and fetch the jar normally.
  - If I remove the io.github.javadiffutils from the module-info.java (documenting purpose), the project can build and the standalone will have the extra diff functionality when comparing two different strings.
- list of failed test cases
  - JarDescribeModuleTests [ [7] module=junit-platform-console ]
  - ModularUserGuideTests [ runTestsFromUserGuideWithinModularBoundaries(Path) ]
  - StandaloneTests [ printVersionViaModule() ]
  - ToolProviderTests [ findAndRunJUnitOnTheModulePath() ]
No clues in how to write Auto Test cases
- The DemoHierarchicalTestEngine dummyTestEngine does not support making a test comparing two strings.

I will try my best to see if I can solve them. Because of the enormous amount of codes without any annotation, this could be all I can do in making solution this issue.

marcphilipp · 2024-09-23T07:07:41Z

@XJ114514 Could you please open a draft PR so it's easier for us to take a look?

XJ114514 · 2024-09-23T12:12:03Z

@XJ114514 Could you please open a draft PR so it's easier for us to take a look?

Thank you for getting back to me. I made the draft PR after testing the generated standalone jar.

1. Add diffPrinter class 2. replace diff in consoleTestExecutor with diffPrinter - will be deleted evetually 3. mark the location of adding diff in flatPrinter and verboseTreePrinter. Issue: junit-team#3139

1. Add a `testPlan` getter to `platform/launcher/listeners/SummaryGeneratingListener.java`. 2. DiffPrinter now needs testPlan to display the testMethod info. Issue: junit-team#3139

github-actions bot added the status: new label Jan 30, 2023

marcphilipp added theme: reporting component: Platform status: team discussion and removed status: new labels Feb 17, 2023

marcphilipp changed the title ~~ConsoleLauncher output --details diff flag~~ ConsoleLauncher should output diff of expected and actual String values May 5, 2023

marcphilipp added this to the 5.11 M1 milestone May 5, 2023

marcphilipp added up-for-grabs and removed status: team discussion labels May 5, 2023

marcphilipp added the type: enhancement label May 5, 2023

bechte self-assigned this Jul 4, 2023

marcphilipp mentioned this issue Jul 17, 2023

assertEquals with a diff #3192

Open

bechte mentioned this issue Jul 18, 2023

Add diff section to report of AssertionFailedError failures #3397

Closed

6 tasks

bechte linked a pull request Jul 18, 2023 that will close this issue

Add diff section to report of AssertionFailedError failures #3397

Closed

6 tasks

marcphilipp unassigned bechte Jan 12, 2024

marcphilipp modified the milestones: 5.11 M1, 5.12 Feb 2, 2024

marcphilipp modified the milestones: 5.11 M2, 5.11 M3 May 17, 2024

marcphilipp modified the milestones: 5.11 M3, 5.12 Jul 5, 2024

XJ114514 pushed a commit to XJ114514/junit5 that referenced this issue Aug 31, 2024

Issue: junit-team#3139 part 2 update

2fa9a8e

delete one line of extra code

XJ114514 added a commit to XJ114514/junit5 that referenced this issue Sep 14, 2024

Try to Fix Dependency issue

af16685

Issue: junit-team#3139

marcphilipp added the status: in progress label Sep 23, 2024

marcphilipp assigned XJ114514 Sep 23, 2024

XJ114514 linked a pull request Sep 23, 2024 that will close this issue

Output diff of expected and actual values in ConsoleLauncher #4017

Draft

6 tasks

marcphilipp linked a pull request Sep 23, 2024 that will close this issue

Output diff of expected and actual values in ConsoleLauncher #4017

Draft

6 tasks

XJ114514 added a commit to XJ114514/junit5 that referenced this issue Oct 25, 2024

Issue: junit-team#3139 Add number reference to each test

a3f8f25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ConsoleLauncher should output diff of expected and actual String values #3139

ConsoleLauncher should output diff of expected and actual String values #3139

carolosf commented Jan 30, 2023 •

edited

Loading

marcphilipp commented Feb 17, 2023

carolosf commented Feb 17, 2023 •

edited

Loading

marcphilipp commented Apr 22, 2023

marcphilipp commented May 5, 2023

marcphilipp commented May 5, 2023

bechte commented Jul 4, 2023

carolosf commented Jul 4, 2023

carolosf commented Jul 4, 2023

marcphilipp commented Jul 7, 2023

bechte commented Jul 18, 2023 •

edited

Loading

bechte commented Jul 21, 2023

XJ114514 commented Aug 31, 2024 •

edited

Loading

XJ114514 commented Sep 1, 2024 •

edited

Loading

XJ114514 commented Sep 14, 2024

All the test cases seem to result in `Module Not Found`, while the gradle can successfully build and fetch the jar normally.

marcphilipp commented Sep 23, 2024

XJ114514 commented Sep 23, 2024

ConsoleLauncher should output diff of expected and actual String values #3139

ConsoleLauncher should output diff of expected and actual String values #3139

Comments

carolosf commented Jan 30, 2023 • edited Loading

marcphilipp commented Feb 17, 2023

carolosf commented Feb 17, 2023 • edited Loading

marcphilipp commented Apr 22, 2023

marcphilipp commented May 5, 2023

marcphilipp commented May 5, 2023

bechte commented Jul 4, 2023

carolosf commented Jul 4, 2023

carolosf commented Jul 4, 2023

marcphilipp commented Jul 7, 2023

bechte commented Jul 18, 2023 • edited Loading

bechte commented Jul 21, 2023

XJ114514 commented Aug 31, 2024 • edited Loading

XJ114514 commented Sep 1, 2024 • edited Loading

XJ114514 commented Sep 14, 2024

Conclusion of the problems that I had encountered:

All the test cases seem to result in Module Not Found, while the gradle can successfully build and fetch the jar normally.

marcphilipp commented Sep 23, 2024

XJ114514 commented Sep 23, 2024

carolosf commented Jan 30, 2023 •

edited

Loading

carolosf commented Feb 17, 2023 •

edited

Loading

bechte commented Jul 18, 2023 •

edited

Loading

XJ114514 commented Aug 31, 2024 •

edited

Loading

XJ114514 commented Sep 1, 2024 •

edited

Loading

All the test cases seem to result in `Module Not Found`, while the gradle can successfully build and fetch the jar normally.