[Backport release/3.9] gdalwarp: be more robust to numerical instability when selecting overviews

[Backport release/3.9] gdalwarp: be more robust to numerical instability when selecting overviews #6725