[experimental] Run crosshair in CI

[Zac-HD]

See #3914

To reproduce this locally, you can run make check-crosshair-cover/nocover/niche for the same command as in CI, but I'd recommend pytest --hypothesis-profile=crosshair hypothesis-python/tests/{cover,nocover,datetime} -m xf_crosshair --runxfail to select and run only the xfailed tests.

Hypothesis' problems

• Vast majority of failures are Flaky: Inconsistent results from replaying a failing test... - mostly backend-specific failures; we've both
  • improved reporting in this case to show the crosshair-specific traceback
  • got most of the affected tests passing
• Invalid internal boolean probability, e.g. "hypothesis/internal/conjecture/data.py", line 2277, in draw_boolean assert p > 2 ** (-64), fixed in 1f845e0 (#4049)
• many of our test helpers involved nested use of @given, fixed in 3315be6
• symbolic outside context
  • due to charmap, fixed in 48e89a6
  • due to target(), fixed in 85712ad (#4049)
• avoid uninstalling typing_extensions when crosshair depends on it
• tests which are not really expected to pass on other backends. I'm slowly applying a backend-specific xfail decorator to them, @xfail_on_crosshair(...).
  • tests which expect to raise a healthcheck, and fail because our crosshair profile disables healthchecks. Disable only .too_slow and .filter_too_much, and skip remaining affected tests under crosshair.
  • undo some over-broad skips, e.g. various xfail decorators, pytestmarks, -k 'not decimal' once we're closer
• provide a special exception type for when running the test or realizing values would hit a PathTimeout; see Rare PathTimeout errors in provider.realize(...) pschanely/hypothesis-crosshair#21 and Stable support for symbolic execution #3914 (comment)
  • and something to signal that we've exhausted Crosshair's ability to explore the test. If this is sound, we've verified the function and can stop! (and should record that in the stop_reason). If unsound, we can continue testing with Hypothesis' default backend - so it's important to distinguish.
    Add BackendCannotProceed to improve integration #4092

Probably Crosshair's problems

• Repeated-registration error, see Duplicate type "<class 'array.array'>" registered from repeated imports? pschanely/hypothesis-crosshair#17
• RecursionError, see RecursionError in _issubclass pschanely/CrossHair#294
• unsupported operand type(s) for -: 'float' and 'SymbolicFloat' in test_float_clamper
• TypeError: descriptor 'keys' for 'dict' objects doesn't apply to a 'ShellMutableMap' object (or 'values' or 'items'). Fixed in Implement various fixes for hypothesis integration pschanely/CrossHair#269
• TypeError: _int() got an unexpected keyword argument 'base'
• Buffer not realized for hash function, fixed in hashlib requires the buffer protocol, which symbolics bytes don't provide pschanely/CrossHair#272
• Internal error for case-insensitive regex Internal error for case-insensitive regex where changing case can change string length pschanely/CrossHair#274
• typing.get_type_hints() raises ValueError, see typing.get_type_hints() raises ValueError when used inside Crosshair pschanely/CrossHair#275
• json round-trip error below
• TypeError in bytes regex, see TypeError in bytes regex pschanely/CrossHair#276
• Invalid args to provider.draw_boolean() inside FeatureStrategy, see Invalid combination of arguments to draw_boolean(...) pschanely/hypothesis-crosshair#18
• Support dict(name=value), see Support named dict init syntax pschanely/CrossHair#279
• Error in PurePath constructor, see PurePath(LazyIntSymbolicStr) error pschanely/CrossHair#280
• zlib.compress() not symbolic, see a bytes-like object is required, not SymbolicBytes when calling zlib.compress(b'') pschanely/CrossHair#286
• int.from_bytes(map(...), ...), see Accept map() object - or any iterable - in int.from_bytes() pschanely/CrossHair#291
• base64 support, see Support base64.b64encode() and friends pschanely/CrossHair#293
• TypeError: conversion from SymbolicInt to Decimal is not supported; see also snan below
• TypeVar problem, see z3.z3types.Z3Exception: b'parser error' from interaction with TypeVar pschanely/CrossHair#292
• Crash on way-too-large integer, see Crash when integer has (way) too many digits pschanely/CrossHair#285
• RecursionError inside Lark, see Weird failures using sets pschanely/CrossHair#297
• Investigate some more hypothesis tests pschanely/CrossHair#307

Error in operator.eq(Decimal('sNaN'), an_int)

____ test_rewriting_does_not_compare_decimal_snan ____
  File "hypothesis/strategies/_internal/strategies.py", line 1017, in do_filtered_draw
    if self.condition(value):
TypeError: argument must be an integer
while generating 's' from integers(min_value=1, max_value=5).filter(functools.partial(eq, Decimal('sNaN')))

Cases where crosshair doesn't find a failing example but Hypothesis does

Seems fine, there are plenty of cases in the other direction. Tracked with @xfail_on_crosshair(Why.undiscovered) in case we want to dig in later.

Nested use of the Hypothesis engine (e.g. given-inside-given)

This is just explicitly unsupported for now. Hypothesis should probably offer some way for backends to declare that they don't support this, and then raise a helpful error message if you try anyway.

[pschanely]

@Zac-HD your triage above is SO great. I am investigating.

[pschanely]

Knocked out a few of these in 0.0.60.
I think that means current status on my end is:

• TypeError: conversion from SymbolicInt to Decimal is not supported
• Unsupported operand type(s) for -: 'float' and 'SymbolicFloat' in test_float_clamper
• TypeError: descriptor 'keys' for 'dict' objects doesn't apply to a 'ShellMutableMap' object (or 'values' or 'items').
• TypeError: _int() got an unexpected keyword argument 'base'
• Symbolic not realized (in e.g. test_suppressing_filtering_health_check)
• Error in operator.eq(Decimal('sNaN'), an_int)
• Zac's cursed example below!

More soon.

[Zac-HD]

Ah - the Flaky failures are of course because we had some failure under the Crosshair backend, which did not reproduce under the Hypothesis backend. This is presumably going to point to a range of integration bugs, but is also something that we'll want to clearly explain to users because integration bugs are definitely going to happen in future and users will need to respond (by e.g. using a different backend, ignoring the problem, whatever).

• improve the reporting around Flaky failures where the differing or missing errors are related to a change of backend while shrinking. See also Change Flaky to be an ExceptionGroup #4040.
• triage all the current failures so we can fix them

[tybug]

Most/all of the "expected x, got symbolic" errors are symptoms of an underlying error in my experience (often operation on symbolic while not tracing). In this case running with export HYPOTHESIS_NO_TRACEBACK_TRIM=1 reveals limited_category_index_cache in cm.query is at fault.

[Zac-HD]

ah-ha, seems like we might want some #4029 - style 'don't cache on backends with avoid_realize=True' logic.

[pschanely]

Still here and excited about this! I am on a detour of doing a real symbolic implementation of the decimal module - should get that out this weekend.

[Zac-HD]

Triaging a pile of the Flaky erorrs, most were due to getting a RecursionError under crosshair and then passing under Hypothesis - and it looks like most of those were in turn because of all our nested-@given() test helpers.

So I've tried de-nesting those, which seems to work nicely and even makes things a bit faster by default; and when CI finishes we'll see how much it helps on crosshair 🤞

[Zac-HD]

most of the currently failing tests look like they might be crosshair issues, cc @pschanely:

• tests/cover/test_filter_rewriting.py::test_regex_filter_rewriting[binary(min_size=5, max_size=10)-b'ab+c'-match] - crosshair.statespace.NotDeterministic
• tests/nocover/test_explore_arbitrary_languages.py::test_explore_an_arbitrary_language - crosshair.statespace.NotDeterministic
• tests/cover/test_lookup.py::test_resolves_builtin_types[BaseException] - crosshair.util.CrosshairInternal: Possibly transient value found in memo (and [object])

and I'll skip the database test - crosshair just finishes exploring sooner than that test expects 😁

[Zac-HD]

@pschanely huge progress from recent updates! The BackendCannotProceed mechanism entirely fixed several classes of issues, the floats changes have been great (signed zero ftw!), from_type() generates instances more often, I'm no longer skipping categories of stuff, and overall we've dropped from about +350 to +250 lines of code in this PR 🎊

At this point my only real reason to avoid merging is that crosshair updates often cause a fair bit of churn, causing some tests to start failing and some to start xpassing - it's net-good, but would be toil in our CI. I feel like we've crossed from an alpha-version which is a neat proof of concept, to a beta-version which is still early but already both useful and clearly on a path to stability and wider adoption. Incredibly excited about this ✨

---

If you want to pull out Crosshair issues,

• this PR is probably useful as a pre-release test, to check whether there are any regressions you didn't expect
• there's a commit marking some things that look like Crosshair bugs to me, and many more where Crosshair just doesn't find a failure that Hypothesis does (within the test budget, and which might or might not be a problem)
• there's a commit full of tests skipped because they were very slow, if you want to look at performance issues. I haven't audited it lately but would guess at least a third are still slow + also Crosshair's problem.
• the last big commit is pretty messy, probably best to ignore that for now

[pschanely]

@pschanely huge progress from recent updates! The BackendCannotProceed mechanism entirely fixed several classes of issues, the floats changes have been great (signed zero ftw!), from_type() generates instances more often, I'm no longer skipping categories of stuff, and overall we've dropped from about +350 to +250 lines of code in this PR 🎊

So great.

At this point my only real reason to avoid merging is that crosshair updates often cause a fair bit of churn, causing some tests to start failing and some to start xpassing - it's net-good, but would be toil in our CI.

Frankly, I'm not sure it makes sense to block hypothesis on a crosshair-related failure, even in a very distant, stable future. Would love your ideas making the integration more "eventually" correct. Maybe a dedicated testing repo that pulls the hypothesis source and has these pytest markers externally applied? (or submodules? but those scare me)

If you want to pull out Crosshair issues,

Always. Thanks for the commit breakdown. More updates soon!

[Zac-HD]

Frankly, I'm not sure it makes sense to block hypothesis on a crosshair-related failure, even in a very distant, stable future. Would love your ideas making the integration more "eventually" correct. Maybe a dedicated testing repo that pulls the hypothesis source and has these pytest markers externally applied? (or submodules? but those scare me)

For clarity, "blocking" would mean 'when we update our pinned dependencies, if Crosshair has changed we'll update the xfail markers accordingly and report any issues upstream, or maybe add a != requirement for that version'. Similarly, if a Hypothesis PR doesn't work with Crosshair I'd prefer to learn that at the time so I can decide to either xfail the tests, or do some extra work to support it - and my guess is that the converse would be useful for you too.

In practice I expect I'll just keep updating this PR for now, and you can grab a local copy of the branch if you want to run the tests before a Crosshair release 😁 (and note the test-selection tips at the top of the pr!)

[pschanely]

For clarity, "blocking" would mean 'when we update our pinned dependencies, if Crosshair has changed we'll update the xfail markers accordingly and report any issues upstream, or maybe add a != requirement for that version'. Similarly, if a Hypothesis PR doesn't work with Crosshair I'd prefer to learn that at the time so I can decide to either xfail the tests, or do some extra work to support it - and my guess is that the converse would be useful for you too.

Fair enough! I was concerned about how much churn in CrossHair pass/fails you'll see for unrelated hypothesis changes, but it's also true that I want to know about what you see. Current plan SGTM.

In practice I expect I'll just keep updating this PR for now, and you can grab a local copy of the branch if you want to run the tests before a Crosshair release 😁 (and note the test-selection tips at the top of the pr!)

Yup! I've been doing this a little already; works for me.