-
Notifications
You must be signed in to change notification settings - Fork 121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add CTS macro for marking tests as known failures in code #2392
Conversation
83dad48
to
0ea7e17
Compare
0ea7e17
to
28cfbbb
Compare
28cfbbb
to
ba57ce9
Compare
The new `UUR_KNOWN_FAILURE_ON` macro can be used to skip tests on devices where the test is known to fail. This can be done for a devices in an adapter or by substring match of the device name. For example: ```cpp UUR_KNOWN_FAILURE_ON(uur::OpenCL{"Intel(R) UHD Graphics 770"}); ``` This invocation is also used in a few places in order to have clean CTS runs on machines with this iGPU.
ba57ce9
to
5e376ec
Compare
deviceName); \ | ||
const char *alsoRunKnownFailures = \ | ||
std::getenv("UR_CTS_ALSO_RUN_KNOWN_FAILURES"); \ | ||
if (alsoRunKnownFailures && (alsoRunKnownFailures == "1"sv || \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could getting and checking the environment variable be done in a separate function? Just to avoid having a huge macro in each failing test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've done this in another branch which is based on top of the work to support multiple adapters/paltforms/devices at once. This work will likely get rolled into those changes.
if (result != UR_RESULT_SUCCESS) { | ||
return result; | ||
} | ||
out_value = std::string(data.data(), data.size()); | ||
out_value = value.substr(0, value.find_last_of('\0')); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A malfunctioning adapter could return a string that isn't null terminated, resulting in a crash here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll chuck a std::min(value.find_last_of('\0'), value.size())
on this.
does this have a way to account for parameter specific fails? we still have a lot of expected fails in |
You mean based on |
yeah kind of, unfortunately we have this hack that's going to be hard to get rid of:
GetParam() gets you a std::tuple containing the device and whatever T the parameter is, getParam just gets you the parameter
|
Okay sure, or if (getParam() == UR_DEVICE_INFO_GLOBAL_MEM_FREE) {
UUR_KNOWN_FAILURE_ON(uur::LevelZeroV2{});
} |
As well, the need for this shouldn't be quite as prevalent after #2290 is addressed. |
I've closing this as the changes will be incorporated into the work @aarongreig is doing to support multi-platform/device testing which will also remove the match files. |
The new
UUR_KNOWN_FAILURE_ON
macro can be used to skip tests on devices where the test is known to fail. This can be done for a devices in an adapter or by substring match of the device name. For example:This invocation is also used in a few places in order to have clean CTS runs on machines with this iGPU.