Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fast path for replaying already shrunk test cases #4155

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

DRMacIver
Copy link
Member

This adds a fast path so that when you rerun a test that has previously been fully shrunk it does not attempt to shrink it again.

The logic is a bit fussy to avoid a couple of failure modes. In particular:

  • We need to make sure that when we replay multiple failures we do see all of these failures
  • If the test changes in a way that means that we should shrink after all, we try to do so. This impossible to detect in general without actually running the shrinker and seeing, but we can detect major test changes.

I've added corresponding tests for a bunch of the fussy different interactions this can have.

This did cause me to find a bug in the fuzzing support where it was putting the data in the primary key (which is for already shrunk examples) rather than the secondary key (which is for interesting unshrunk examples), which caused some problems with this. This means that previous users of hypothesis fuzzing will run into a problem with this.

I thought about trying to fix that but decided against it. I think it's a niche use case that can be solved by shrinking them under an old version of Hypothesis before upgrading, and I don't think this is actually a widely used feature? Let me know if you want me to do anything about it and we'll figure something out though.

@DRMacIver DRMacIver force-pushed the DRMacIver/no-shrink-on-replay branch 2 times, most recently from c3fe418 to 89ca4d2 Compare November 2, 2024 10:42
@DRMacIver DRMacIver force-pushed the DRMacIver/no-shrink-on-replay branch from 89ca4d2 to 3a74210 Compare November 2, 2024 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant