[UT] Automatically generate skiplist when on a new platform

[AshburnLee]

Functionality

issue#985 & issue#986 are solved by manually add skiplists, this PR automatically generate skiplist whenever on a new platform.

Result

A set of skiplists is generated and replace all pytest.skip() calls:

/skiplist/${GPU}_${AGAMA}/language.txt
                         /operators.txt
                         /subprocess.txt
                         /regression.txt
                         /...

Note

New script in this PR will be only executed when on a new platform, a new platform means that there exist no such a directory: /skiplist/${GPU}_${AGAMA}, script generates this directory for this new platform.

Fixes #1423

[pbchekin]

Not sure if I understand what this PR does. Where exactly you replace pytest.skip() calls?

[AshburnLee]

Not sure if I understand what this PR does. Where exactly you replace pytest.skip() calls?

The logic is this: if code is running on a new platform let's say xxx, there is no corresponding skiplist /skiplist/xxx/....txt. This PR executes run_core_tests & run_regression_tests to get pytest log and parse the log to generate:

/skiplist/xxx/language.txt
                    /operators.txt
                    /subprocess.txt
                    /regression.txt
                    /...

Those are generated from the SKIPPED in the log, which is the result from pytest.skip(). Then execute run_core_tests & run_regression_tests again, in this run, all pytest.skip() calls will be ignored, as the cases are in skiplist.

Once we have this /skiplist/xxx/....txt, the next time run code on the same platform, only run run_core_tests & run_regression_tests one time with skiplist, all pytest.skip() calls will be ignored.

===

Let me put it another way, there are two types of execution situations here:

1. On new platforms, execute run_core_tests & run_regression_tests twice, we get the skiplist from the first run, we use this skiplist in the second run.
2. Not on new platforms, execute run_core_tests & run_regression_tests one time, because skiplist is already there, just use it. As skiplist is given, all pytest.skip() calls are ignored.

===

So "replace" here is not real, "ignore" is more correct. As I think we are not supposed to modify the test code in PR, so this is a way to ignore.

[AshburnLee]

Oh, I see Vlad linked this PR to #882, but this PR is not target to this issue.

Again, this PR description says "#985 & #986 are solved by manually add skiplists, this PR automatically generate skiplist whenever on a new platform."

[pbchekin]

The steps for preparing a skip list for a new platform need to be different, see #987. For a new platform, you need to

1. "un-skip" all skipped tests: remove or ignore pytest.skip() in the Python code.
2. "un-deselect" all deselected tests: use an empty skip list.
3. run all tests, ignore test failures, save the test report.
4. parse the test report to generate a skip list by de-selecting all failed tests (put them to the new skip list).

Also no need to parse the pytest output, it is for humans. The existing script allows saving the test report in JUnit XML format that is machine readable.

[AshburnLee]

The steps for preparing a skip list for a new platform need to be different, see #987. For a new platform, you need to

1. "un-skip" all skipped tests: remove or ignore pytest.skip() in the Python code.
2. "un-deselect" all deselected tests: use an empty skip list.
3. run all tests, ignore test failures, save the test report.
4. parse the test report to generate a skip list by de-selecting all failed tests (put them to the new skip list).

Also no need to parse the pytest output, it is for humans. The existing script allows saving the test report in JUnit XML format that is machine readable.

Good to know, let me do it manually and see what's the difference from my "naive" way ...

[AshburnLee]

I choose a smaller scope to reproduce the existing skiplist on PVC+Rolling, the scope is language/test_core.py. The command is pytest -vvv -n 8 --device xpu language/test_core.py.
Here is what I've done:

1. "un-skip" all skipped tests: remove or ignore pytest.skip() in the Python code.

Done, replaced pytest.skip() with pass in language/test_core.py.

2. "un-deselect" all deselected tests: use an empty skip list.

Done, I used the original pytest command, there is no --deselect-from-file specified.

3. run all tests, ignore test failures, save the test report.

Done, I saved the report only with the following content:

...
[gw5] [ 97%] PASSED language/test_core.py::test_convert2d[dst_layout8-interm_layout3-src_layout7-float16-64-64] 
[gw8] [ 97%] FAILED language/test_core.py::test_fp8_dot_acc[0-float8e4b15] 
[gw10] [ 97%] FAILED language/test_core.py::test_dot3d[int8-int8-64-64-64-1-4] 
[gw4] [ 97%] PASSED language/test_core.py::test_dot[1-64-128-128-2-True-False-none-tf32-float16-float32-1_1] 
[gw9] [ 97%] PASSED language/test_core.py::test_convert2d[dst_layout8-interm_layout3-src_layout4-float16-128-128] 
[gw5] [ 97%] PASSED language/test_core.py::test_convert2d[dst_layout8-interm_layout3-src_layout7-float16-128-128] 
[gw9] [ 97%] XFAIL language/test_core.py::test_convert2d[dst_layout8-interm_layout3-src_layout4-float16-1-64] 
[gw5] [ 97%] XFAIL language/test_core.py::test_convert2d[dst_layout8-interm_layout3-src_layout7-float16-1-64]
...

It has PASSED, FAILED, XFAIL, no SKIPPED as there were no pytest.skip().

4. parse the test report to generate a skip list by de-selecting all failed tests (put them to the new skip list).

Here is what I think is weird, "skip list is generated by de-selecting all failed tests", then my skip list contains all PASSED & XFAIL case, but the reference is only 4 FAILED cases,

My "naive" way is put all SKIPPED cases into the existing skiplist, which ready contains 4 FAILED cases. That is why I added 180 SKIPPED cases in the other PR (https://github.com/intel/intel-xpu-backend-for-triton/pull/1093/files#diff-4fa89506bbe5b7f399da9acb4709a0ab504aa68c09fddd90bb29f09b84aa4311)

I haven‘t tried the JUnit XML script, but the result should be the same. (will try)

I think I must have misunderstood something, could you point it out, thanks @pbchekin

[AshburnLee]

Implemented here:#1596