Parallelizing github actions

[jordandsullivan]

Apologies if this is a duplicate, but I wanted to know if we had decided on a tool for parallelizing the docs build part of the github actions. My understanding is that it's actually running all the notebooks that takes a long time (as pointed out by @natestemen in #2247, these example notebooks are essentially our integration testing at present).

Locally I have started using this tool https://pypi.org/project/pytest-xdist/ when I run pytest, which has an automatic mode to parallelize across as many CPUs as are available.

[natestemen]

Docs

We have #2235 opened from @cosenal which is currently blocked by the sphinx theme that we use. Alessandro has done some testing for the pydata-sphinx-theme and found our docs look okay even when built in parallel. See the comment here.

Tests

When running make test the following command is run:

mitiq/Makefile

Line 63 in e76ef08

pytest -n auto -v --cov=mitiq --cov-report=term --cov-report=xml --ignore=mitiq/interface/mitiq_pyquil

In particular the -n auto option is provided by the pytest-xdist package and is described in pytest --help:

  -n numprocesses, --numprocesses=numprocesses
                        Shortcut for '--dist=load --tx=NUM*popen'. With 'auto',
                        attempt to detect physical CPU count. With 'logical',
                        detect logical CPU count. If physical CPU count cannot
                        be found, falls back to logical count. This will be 0
                        when used with --pdb.

You can see pytest-xdist is already in our dev_requirements.txt 😃. It helps the tests run so much faster locally! Usually when I'm just testing the changes I've made I run a single test file, or a series of files in a single module, but when running more than that I almost always use -n auto.

[jordandsullivan]

I see. So is make test already being run in parallel on the Github runner.

[jordandsullivan]

Reference to the place in the Makefile where this is written:

mitiq/Makefile

Lines 61 to 67 in 6ff6c08

	.PHONY: test
	test:
	pytest -n auto -v --cov=mitiq --cov-report=term --cov-report=xml --ignore=mitiq/interface/mitiq_pyquil
	.PHONY: test-%
	test-%:
	pytest -n auto -v --cov=mitiq --cov-report=term --cov-report=xml */$(*)/*