MONGOCRYPT-712 Improve double handling with precision #869
Conversation
test/test-mc-range-encoding.c
Outdated
| .precision = OPT_U32_C(0), | ||
| .expect = 9007199254740996, | ||
| .expectMax = OPT_U64_C(36028797018963967)}, | ||
| {.value = 5, |
There was a problem hiding this comment.
After thinking this through a bit more, although we know that this is probably correct, we still need a few more test cases. We basically need to show that the imprecision that we see in the range [2^53 < abs(n) < 2^63] does not impact the precision in the range of [-(2^53) < val < 2^53]. To make that happen, we should probably run a few test cases.
We should test some random bounds where min E [-2^53, -2^54, -2^55, ... -2^63] and max E [2^53, 2^54, 2^55, ... 2^63] and show that when value = [0, 1, 2, 3, 4, 5], we have .expect = [max + value].
I know that we've poured over the math, but I still have some qualms about us possibly losing precision here. I also have some general qualms about floating point imprecision in a fractional number, but I am not sure that those will ever really go away :(
There was a problem hiding this comment.
Doubles lose precision for their significant between 2^52 and 2^63. The lose precision at 2^(i-52) where i = bits. I generated test cases from the server code and added a few more test cases. One for each number between 52-62 with values from 0-4. I did not add 40 tests, just 10 because 40 would be a waste.
markbenvenuto
force-pushed
the
markbenvenuto/mongocrypt-712
branch
from
cfa0e57
to
d970422
Compare
Min and max integer of a double is -2^53 and 2^53. Remember double has a separate sign bit unlike integers. Javascript's Number constants are not correct for doubles.
Depends on Shreyas' change to compute the right max value.