chore(test): lower p_value_limit to decrease test failure probability #1787
Conversation
| let p_value_limit: f64 = 0.001; | ||
| let p_value_limit: f64 = 0.000_01; | ||
|
|
There was a problem hiding this comment.
how much does this slow the test down ?
There was a problem hiding this comment.
Not at all, the number of samples has not changed
It's just more permissive
This new value is a bit arbitrary, like the previous one was
There was a problem hiding this comment.
I see, what sort of impact does this have on the "tolerance" ?
There was a problem hiding this comment.
I don't see how this is better than the test_random_from_distribution_custom_mod test function that verifies the actual p-value
There was a problem hiding this comment.
Decreasing the p_value_limit means there are less false positives (test fails where it shouldn't) but more false negatives.
False positives rate is the p_value_limit which is well known.
False negative rates depend on the real distribution (if it's not uniform). It's hard to quantify since we don't know what this distribution is.
For a fixed p_value_limit, increasing the number of samples reduces the false negative rates (of all other distributions) but again it's hard to quantify.
So it's hard to know by how much the number of samples needs to be increased to compensate for the decrease of the p_value_limit impact on the false negative rates.
Compared to combining n sub-tests (while keeping the same total number of samples), we can detect correlated deviations in sample-sets of sub-tests which individually appear plausible (even when combined as if they were independent) but when analyzed together are much less plausible, which means the detection is finer.
No description provided.