pytest-dynamicrerun is a pytest plugin to rerun tests dynamically based off of test outcome and output.
This plugin is tested against the following Python and pytest versions. Each Python version is tested against all pytest versions. Please file an issue to request additional targets.
| Python Versions: | python 3.5, python 3.6, python 3.7, python 3.8, pypy3 |
|---|---|
| Pytest Versions: | 5.4.0, 5.4.1, 5.4.2, 5.4.3, 6.0.0, 6.0.1 |
Install this plugin from PyPI by running the following:
$ pip install pytest-dynamicrerun
By default, one rerun attempt is made. You can set the amount of times to attempt a rerun by passing the --dynamic-rerun-attempts flag when invoking pytest or including the dynamic_rerun_attempts INI key.
To pass the flag:
python3 -m pytest --dynamic-rerun-attempts=3
To set the INI key add the following to your config file's [pytest] section:
[pytest] dynamic_rerun_attempts = 123
Passing a non positive integer value will set the number of rerun attempts to the default.
By default, all failed tests are rerun. You can change this behavior by either passing the --dynamic-rerun-triggers flag when invoking pytest or including the dynamic_rerun_triggers INI key. Note that regular expressions are allowed.
To pass the flag:
python3 -m pytest --dynamic-rerun-triggers="a triggering trace"
To set the INI key add the following to your config file's [pytest] section:
[pytest] dynamic_rerun_triggers = a triggering trace
You can accumulate values by either providing the flag multiple times or appending to the INI key. For example, the below two snippets would cause this plugin to trigger on both foo and bar:
python3 -m pytest --dynamic-rerun-triggers="foo" --dynamic-rerun-triggers="bar"
[pytest]
dynamic_rerun_triggers = foo
bar
Note that at this time only stdout, stderr, and exceptions are checked.
You can specify an interval to rerun tests on by either passing the --dynamic-rerun-schedule flag to python when invoking pytest or including the dyanmic_rerun_schedule INI key.
Internally, this plugin uses croniter to schedule wait times. Because of this, we are able to schedule wait times with second level granularity. Visit the croniter repository README to find out more information on this.
To pass the flag:
python3 -m pytest --dynamic-rerun-schedule="* * * * * *"
To set the INI key add the following to your config file's [pytest] section:
[pytest] dynamic_rerun_schedule = * * * * * *
Note that any valid cron schedule is accepted. If this flag is not passed or set in the INI file, this plugin will not take effect. Passing an invalid value will force the interval to default to * * * * * * ( every second ).
You can ignore this plugin by passing the --dynamic-rerun-disabled flag to python when invoking pytest or including the dynamic_rerun_disabled INI key.
To pass the flag:
python3 -m pytest --dynamic-rerun-disabled="True"
To set the INI key add the following to your config file's [pytest] section:
[pytest] dynamic_rerun_disabled = True
Note that if this flag is omitted, we do not disable the plugin ( so it is equivalent to passing --dynamic-rerun-disabled=False )
We can achieve the above functionality through markers as well. This plugin defines the dynamicrerun mark, which can be used as follows:
@pytest.mark.dynamicrerun(attempts=10, disabled=False, schedule="* * * * * *", triggers="foo")
def test_print_foo():
print("foo")
Mark arguments correspond to INI keys as follows:
attemptscorresponds todynamic_rerun_attemptsdisabledcorresponds todynanic_rerun_disabledschedulecorresponds todynamic_rerun_scheduletriggerscorresponds todynamic_rerun_triggers
To pass multiple values to the triggers argument, provide a list as so:
@pytest.mark.dynamicrerun(attempts=10, disabled=False, schedule="* * * * * *", triggers=["foo", "bar", "baz"])
def test_print_foo():
print("foo")
In the above example, reruns will be triggered on foo, bar, and baz.
Note that first we check for arguments in markers, then command line switches, and only then do we check in INI files. Values found at lower levels ( those checked first ) take precedence over values defined at a higher level ( those checked later ).
For example, if we define the number of rerun attempts as 10 when invoking pytest from the command line, but later in a marker define the number of rerun attempts as 3, 3 would take precedence over 10 since we check markers before we check command line flags.
This plugin exposes the following attributes on the item object:
dynamic_rerun_run_times ( list ): The list of times this item was run by the plugin. Note this includes the original non dynamically rerun run.dynamic_rerun_schedule(string): The schedule to rerun this item on. See the sectionSpecifying a rerun intervalabove for more details.dynamic_rerun_sleep_times (list): A list of timedelta objects representing the time slept in between reruns for the itemdynamic_rerun_triggers (list): The rerun triggers for this specific item. See the sectionSpecifying what to rerun onabove for more details.max_allowed_dynamic_rerun_attempts(int): The maximum amount of times we are allowed to rerun this item. See the sectionSpecifying how many times to rerunabove for more details.num_dynamic_reruns_kicked_off (int): The amount of reruns launched at the moment of inspection for this item.
This plugin exposes the following attributes on the session object:
dynamic_rerun_items (list): The list of items that are set to be dynamically rerun on the next iteration
Contributions are always welcome. Tests can be run with tox.
Please remember to add a changelog entry when adding a non-trivial feature.
pre-commit is used to ensure basic checks pass.
Distributed under the terms of the MIT license, "pytest-dynamicrerun" is free and open source software
If you encounter any problems, please file an issue along with a detailed description.