parallel_tests

Speedup Minitest + RSpec + Turnip + Cucumber + Spinach by running parallel on multiple CPU cores.
ParallelTests splits tests into balanced groups (by number of lines or runtime) and runs each group in a process with its own database.

Setup for Rails

RailsCasts episode #413 Fast Tests

Install

Gemfile:

gem 'parallel_tests', group: [:development, :test]

Add to `config/database.yml`

ParallelTests uses 1 database per test-process.

Process number	1	2	3
ENV['TEST_ENV_NUMBER']	''	'2'	'3'

test:
  database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>

Create additional database(s)

rake parallel:create

(Multi-DB) Create individual database

rake parallel:create:<database>
rake parallel:create:secondary

Copy development schema (repeat after migrations)

rake parallel:prepare

Run migrations in additional database(s) (repeat after migrations)

rake parallel:migrate

(Multi-DB) Run migrations in individual database

rake parallel:migrate:<database>

Setup environment from scratch (create db and loads schema, useful for CI)

rake parallel:setup

Drop all test databases

rake parallel:drop

(Multi-DB) Drop individual test database

rake parallel:drop:<database>

Run!

rake parallel:test          # Minitest
rake parallel:spec          # RSpec
rake parallel:features      # Cucumber
rake parallel:features-spinach       # Spinach

rake "parallel:test[1]" --> force 1 CPU --> 86 seconds
rake parallel:test    --> got 2 CPUs? --> 47 seconds
rake parallel:test    --> got 4 CPUs? --> 26 seconds
...

Test by pattern with Regex (e.g. use one integration server per subfolder / see if you broke any 'user'-related tests)

rake "parallel:test[^test/unit]" # every test file in test/unit folder
rake "parallel:test[user]"  # run users_controller + user_helper + user tests
rake "parallel:test['user|product']"  # run user and product related tests
rake "parallel:spec['spec\/(?!features)']" # run RSpec tests except the tests in spec/features

Example output

2 processes for 210 specs, ~ 105 specs per process
... test output ...

843 examples, 0 failures, 1 pending

Took 29.925333 seconds

Run an arbitrary task in parallel

RAILS_ENV=test parallel_test -e "rake my:custom:task"
# or
rake "parallel:rake[my:custom:task]"
# limited parallelism
rake "parallel:rake[my:custom:task,2]"

Running setup or teardown once

require "parallel_tests"

# preparation:
# affected by race-condition: first process may boot slower than the second
# either sleep a bit or use a lock for example File.lock
ParallelTests.first_process? ? do_something : sleep(1)

# cleanup:
# last_process? does NOT mean last finished process, just last started
ParallelTests.last_process? ? do_something : sleep(1)

at_exit do
  if ParallelTests.first_process?
    ParallelTests.wait_for_other_processes_to_finish
    undo_something
  end
end

Even test group runtimes

Test groups will often run for different times, making the full test run as slow as the slowest group.

Step 1: Use these loggers (see below) to record test runtime

Step 2: The next test run will use the recorded test runtimes (use --runtime-log <file> if you picked a location different from below)

Step 3: Automate upload/download of test runtime from your CI system example (chunks need to be combined, an alternative is amend)

RSpec

Rspec: Add to your .rspec_parallel (or .rspec) :

--format progress
--format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log

To use a custom logfile location (default: tmp/parallel_runtime_rspec.log), use the CLI: parallel_test spec -t rspec --runtime-log my.log

Minitest

Add to your test_helper.rb:

require 'parallel_tests/test/runtime_logger' if ENV['RECORD_RUNTIME']

results will be logged to tmp/parallel_runtime_test.log when RECORD_RUNTIME is set, so it is not always required or overwritten.

Loggers

RSpec: SummaryLogger

Log the test output without the different processes overwriting each other.

Add the following to your .rspec_parallel (or .rspec) :

--format progress
--format ParallelTests::RSpec::SummaryLogger --out tmp/spec_summary.log

RSpec: FailuresLogger

Produce pasteable command-line snippets for each failed example. For example:

rspec /path/to/my_spec.rb:123 # should do something

Add to .rspec_parallel or use as CLI flag:

--format progress
--format ParallelTests::RSpec::FailuresLogger --out tmp/failing_specs.log

(Not needed to retry failures, for that pass --only-failures to rspec)

RSpec: VerboseLogger

Prints a single line for starting and finishing each example, to see what is currently running in each process.

# PID, parallel process number, spec status, example description
[14403] [2] [STARTED] Foo foo
[14402] [1] [STARTED] Bar bar
[14402] [1] [PASSED] Bar bar

Add to .rspec_parallel or use as CLI flag:

  --format ParallelTests::RSpec::VerboseLogger

Cucumber: FailuresLogger

Log failed cucumber scenarios to the specified file. The filename can be passed to cucumber, prefixed with '@' to rerun failures.

Usage:

cucumber --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log

Or add the formatter to the parallel: profile of your cucumber.yml:

parallel: --format progress --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log

Note if your cucumber.yml default profile uses <%= std_opts %> you may need to insert this as follows parallel: <%= std_opts %> --format progress...

To rerun failures:

cucumber @tmp/cucumber_failures.log

Setup for non-rails

gem install parallel_tests
# go to your project dir
parallel_test
parallel_rspec
parallel_cucumber
parallel_spinach

use ENV['TEST_ENV_NUMBER'] inside your tests to select separate db/memcache/etc. (docker compose: expose it)
Only run a subset of files / folders:

parallel_test test/bar test/baz/foo_text.rb
Pass test-options and files via --:

parallel_rspec -- -t acceptance -f progress -- spec/foo_spec.rb spec/acceptance
Pass in test options, by using the -o flag (wrap everything in quotes):

parallel_cucumber -n 2 -o '-p foo_profile --tags @only_this_tag or @only_that_tag --format summary'

Options are:

-n [PROCESSES]                   How many processes to use, default: available CPUs
-p, --pattern [PATTERN]          run tests matching this regex pattern
    --exclude-pattern [PATTERN]  exclude tests matching this regex pattern
    --group-by [TYPE]            group tests by:
                                 found - order of finding files
                                 steps - number of cucumber/spinach steps
                                 scenarios - individual cucumber scenarios
                                 filesize - by size of the file
                                 runtime - info from runtime log
                                 default - runtime when runtime log is filled otherwise filesize
-m, --multiply-processes [FLOAT] use given number as a multiplier of processes to run
-s, --single [PATTERN]           Run all matching files in the same process
-i, --isolate                    Do not run any other tests in the group used by --single(-s)
    --isolate-n [PROCESSES]      Use 'isolate'  singles with number of processes, default: 1
    --highest-exit-status        Exit with the highest exit status provided by test run(s)
    --failure-exit-code [INT]    Specify the exit code to use when tests fail
    --specify-groups [SPECS]     Use 'specify-groups' if you want to specify multiple specs running in multiple
                                 processes in a specific formation. Commas indicate specs in the same process,
                                 pipes indicate specs in a new process. Cannot use with --single, --isolate, or
                                 --isolate-n.  Ex.
                                 $ parallel_tests -n 3 . --specify-groups '1_spec.rb,2_spec.rb|3_spec.rb'
                                   Process 1 will contain 1_spec.rb and 2_spec.rb
                                   Process 2 will contain 3_spec.rb
                                   Process 3 will contain all other specs
    --only-group INT[,INT]       Only run the given group numbers.
                                 Changes `--group-by` default to 'filesize'.
-e, --exec [COMMAND]             execute this code parallel and with ENV['TEST_ENV_NUMBER']
-o, --test-options '[OPTIONS]'   execute test commands with those options
-t, --type [TYPE]                test(default) / rspec / cucumber / spinach
    --suffix [PATTERN]           override built in test file pattern (should match suffix):
                                 '_spec.rb$' - matches rspec files
                                 '_(test|spec).rb$' - matches test or spec files
    --serialize-stdout           Serialize stdout output, nothing will be written until everything is done
    --prefix-output-with-test-env-number
                                 Prefixes test env number to the output when not using --serialize-stdout
    --combine-stderr             Combine stderr into stdout, useful in conjunction with --serialize-stdout
    --non-parallel               execute same commands but do not in parallel, needs --exec
    --no-symlinks                Do not traverse symbolic links to find test files
    --ignore-tags [PATTERN]      When counting steps ignore scenarios with tags that match this pattern
    --nice                       execute test commands with low priority.
    --runtime-log [PATH]         Location of previously recorded test runtimes
    --allowed-missing [INT]      Allowed percentage of missing runtimes (default = 50)
    --allow-duplicates           When detecting files to run, allow duplicates
    --unknown-runtime [FLOAT]    Use given number as unknown runtime (otherwise use average time)
    --first-is-1                 Use "1" as TEST_ENV_NUMBER to not reuse the default test environment
    --fail-fast                  Stop all groups when one group fails (best used with --test-options '--fail-fast' if supported
    --verbose                    Print debug output
    --verbose-command            Combines options --verbose-process-command and --verbose-rerun-command
    --verbose-process-command    Print the command that will be executed by each process before it begins
    --verbose-rerun-command      After a process fails, print the command executed by that process
    --quiet                      Print only tests output
-v, --version                    Show Version
-h, --help                       Show this.

You can run any kind of code in parallel with -e / --exec

parallel_test -n 5 -e 'ruby -e "puts %[hello from process #{ENV[:TEST_ENV_NUMBER.to_s].inspect}]"'
hello from process "2"
hello from process ""
hello from process "3"
hello from process "5"
hello from process "4"

	1 Process	2 Processes	4 Processes
RSpec spec-suite	18s	14s	10s
Rails-ActionPack	88s	53s	44s

TIPS

RSpec

Add a .rspec_parallel to use different options, e.g. no --drb
Remove --loadby from .rspec
Instantly see failures (instead of just a red F) with rspec-instafail
Use rspec-retry (not rspec-rerun) to rerun failed tests.
JUnit formatter configuration
Use parallel_split_test to run multiple scenarios in a single spec file, concurrently. (parallel_tests works at the file-level and intends to stay that way)

Cucumber

Add a parallel: foo profile to your config/cucumber.yml and it will be used to run parallel tests
ReportBuilder can help with combining parallel test results
- Supports Cucumber 2.0+ and is actively maintained
- Combines many JSON files into a single file
- Builds a HTML report from JSON with support for debug msgs & embedded Base64 images.

General

[ZSH] use quotes to use rake arguments rake "parallel:prepare[3]"
[Memcached] use different namespaces
e.g. config.cache_store = ..., namespace: "test_#{ENV['TEST_ENV_NUMBER']}"
Debug errors that only happen with multiple files using --verbose and cleanser
export PARALLEL_TEST_PROCESSORS=13 to override default processor count
Shell alias: alias prspec='parallel_rspec -m 2 --'
[Spring] Add the spring-commands-parallel-tests gem to your Gemfile to get parallel_tests working with Spring.
--first-is-1 will make the first environment be 1, so you can test while running your full suite.
export PARALLEL_TEST_FIRST_IS_1=true will provide the same result
email_spec and/or action_mailer_cache_delivery
zeus-parallel_tests
Distributed Parallel Tests on CI systems) learn how parallel_tests can run on distributed servers such as Travis and GitLab-CI. Also shows you how to use parallel_tests without adding TEST_ENV_NUMBER-backends
Capybara setup
Sphinx setup
Capistrano setup let your tests run on a big box instead of your laptop

Contribute your own gotchas to the Wiki or even better open a PR :)

Authors

inspired by pivotal labs

Contributors

Charles Finkel
Indrek Juhkam
Jason Morrison
jinzhu
Joakim Kolsjö
Kevin Scaldeferri
Kpumuk
Maksim Horbul
Pivotal Labs
Rohan Deshpande
Tchandy
Terence Lee
Will Bryant
Fred Wu
xxx
Levent Ali
Michael Kintzer
nathansobo
Joe Yates
asmega
Doug Barth
Geoffrey Hichborn
Trae Robrock
Lawrence Wang
Sean Walbran
Lawrence Wang
Potapov Sergey
Łukasz Tackowiak
Pedro Carriço
Pablo Manrubia Díez
Slawomir Smiechura
Georg Friedrich
R. Tyler Croy
Ulrich Berkmüller
Grzegorz Derebecki
Florian Motlik
Artem Kuzko
Zeke Fast
Joseph Shraibman
David Davis
Ari Pollak
Aaron Jensen
Artur Roszczyk
Caleb Tomlinson
Jawwad Ahmad
Iain Beeston
Alejandro Pulver
Felix Clack
Izaak Alpert
Micah Geisel
Exoth
sidfarkus
Colin Harris
Wataru MIYAGUNI
Brandon Turner
Matt Hodgson
bicarbon8
seichner
Matt Southerden
Stanislaw Wozniak
Dmitry Polushkin
Samer Masry
Volodymyr Mykhailyk
Mike Mueller
Aaron Jensen
Ed Slocomb
Cezary Baginski
Marius Ioana
Lukas Oberhuber
Ryan Zhang
Rhett Sutphin
Doc Ritezel
Alexandre Wilhelm
Jerry
Aleksei Gusev
Scott Olsen
Andrei Botalov
Zachary Attas
David Rodríguez
Justin Doody
Sandeep Singh
Calaway
alboyadjian
Nathan Broadbent
Vikram B Kumar
Joshua Pinter
Zach Dennis
Jon Dufresne
Eric Kessler
Adis Osmonov
Josh Westbrook
Jay Dorsey
hatsu

Michael Grosser
michael@grosser.it
License: MIT

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Readme.md

Readme.md

parallel_tests

Setup for Rails

Install

Add to `config/database.yml`

Create additional database(s)

(Multi-DB) Create individual database

Copy development schema (repeat after migrations)

Run migrations in additional database(s) (repeat after migrations)

(Multi-DB) Run migrations in individual database

Setup environment from scratch (create db and loads schema, useful for CI)

Drop all test databases

(Multi-DB) Drop individual test database

Run!

Example output

Run an arbitrary task in parallel

Running setup or teardown once

Even test group runtimes

RSpec

Minitest

Loggers

RSpec: SummaryLogger

RSpec: FailuresLogger

RSpec: VerboseLogger

Cucumber: FailuresLogger

Setup for non-rails

TIPS

RSpec

Cucumber

General

Authors

Contributors

Files

Readme.md

Latest commit

History

Readme.md

File metadata and controls

parallel_tests

Setup for Rails

Install

Add to config/database.yml

Create additional database(s)

(Multi-DB) Create individual database

Copy development schema (repeat after migrations)

Run migrations in additional database(s) (repeat after migrations)

(Multi-DB) Run migrations in individual database

Setup environment from scratch (create db and loads schema, useful for CI)

Drop all test databases

(Multi-DB) Drop individual test database

Run!

Example output

Run an arbitrary task in parallel

Running setup or teardown once

Even test group runtimes

RSpec

Minitest

Loggers

RSpec: SummaryLogger

RSpec: FailuresLogger

RSpec: VerboseLogger

Cucumber: FailuresLogger

Setup for non-rails

TIPS

RSpec

Cucumber

General

Authors

Contributors

Add to `config/database.yml`