Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Random execution order of test cases (shuffle) #2

Closed
shreyasbharath opened this issue Jul 13, 2012 · 6 comments · May be fixed by #714
Closed

Random execution order of test cases (shuffle) #2

shreyasbharath opened this issue Jul 13, 2012 · 6 comments · May be fixed by #714
Labels
feature request

Comments

@shreyasbharath
Copy link

This is a very useful feature that I've come across in GoogleTest (using the --gtest-shuffle option). The main reason you'd want to shuffle tests is to expose any tests that depend on any other test in any way.

I can't use GoogleTest on the embedded target I am working on because it is too heavy weight. Would this be a difficult feature to add?

I'll try toying with an implementation on my fork and submit a pull request when it is ready.

Thanks.
@sw17ch
Copy link
Contributor

sw17ch commented Oct 3, 2012

I like this idea a lot and would be happy to get a pull request with this goal accepted.

@ghost ghost assigned silentbicycle Nov 19, 2012
@ghost
Copy link

ghost commented Mar 12, 2013

I've implemented this "random execution order of test cases" feature on my fork and I'm going to submit this pull request very soon.

@barneywilliams
Copy link
Member

Can your implementation be fed with a seed to reproduce a "randomized"
sequence verbatim? That is definitely necessary for this to be a
responsible addition.

Thanks much for contributing! This will definitely be a good addition to
the framework. We have discovered many initialization issues, both in tests
and production code via randomization.

Cheers,
Greg

On Tue, Mar 12, 2013 at 6:04 AM, pbhar notifications@github.com wrote:

I've implemented this "random execution order of test cases" feature on my
fork and I'm going to submit this pull request very soon.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-14767140
.

@mvandervoord mvandervoord mentioned this issue May 21, 2014
Closed
adiorion pushed a commit to adiorion/Unity that referenced this issue Oct 28, 2015
@leftbyte
Merge pull request ThrowTheSwitch#2 from OnBeep/dan-fix_no_message_as…
58b5acb 
…sert_catches

fix no message assert catches
@mvandervoord mvandervoord mentioned this issue May 11, 2016
Merged
jeremyhannon added a commit to jeremyhannon/Unity that referenced this issue Feb 10, 2018
@jeremyhannon
unity_internals MISRA 19.10: macro param parens ThrowTheSwitch#2
7bfcc54 
MISRA 2004 rule 19.10: inside macros, surround each parameter usage with parentheses.

commit2/3
jeremyhannon added a commit to jeremyhannon/Unity that referenced this issue Feb 10, 2018
@jeremyhannon
unity_internals MISRA 19.10: macro param parens ThrowTheSwitch#2
decbea3 
MISRA 2004 rule 19.10: inside macros, surround each parameter usage with parentheses.

commit2/3
@jamestiotio
Copy link

Hi folks, I stumbled upon this issue while trying to check whether Unity supports this shuffled random execution order of tests or not. This issue was closed as completed by @mvandervoord, but as far as I am aware, I do not think Unity supports this feature yet, and there are no corresponding relevant pull requests or commits linked to this issue that introduced this feature into Unity.

Can I confirm whether this feature has been added to Unity or not? If yes, can I ask how to activate it (and perhaps add those steps to the docs)? If not, then I am willing to raise a PR to add this feature, including adding the aforementioned possibility of using a seed for deterministic reproduction.

Thanks!

@mvandervoord
Copy link
Member

This was previously supported in the test runner generator, but was long ago removed from it. I'd be interested in seeing this feature reintroduced in a clean way. :)

@silentbicycle silentbicycle removed their assignment Apr 10, 2023
@jamestiotio
Copy link

This was previously supported in the test runner generator, but was long ago removed from it.

Ah alright then.

I'd be interested in seeing this feature reintroduced in a clean way. :)

Sure thing, I will work on this whenever I manage to get around to it.

@jamestiotio jamestiotio mentioned this issue Jan 30, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add support for randomizing test execution order jamestiotio/Unity
6 participants
@mvandervoord @sw17ch @silentbicycle @barneywilliams @shreyasbharath @jamestiotio