diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000..a1ff62d --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,23 @@ +name: CI +on: [push, pull_request] + +jobs: + test: + name: Test + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@master + - name: Install Rust + run: | + rustup set profile minimal + rustup toolchain install stable + rustup default stable + - name: Install latest mdbook + run: | + tag=$(curl 'https://api.github.com/repos/rust-lang/mdbook/releases/latest' | jq -r '.tag_name') + url="https://github.com/rust-lang/mdbook/releases/download/${tag}/mdbook-${tag}-x86_64-unknown-linux-gnu.tar.gz" + mkdir bin + curl -sSL $url | tar -xz --directory=bin + echo "$(pwd)/bin" >> $GITHUB_PATH + - name: Run tests + run: mdbook test doc diff --git a/.github/workflows/publish-docs.yml b/.github/workflows/publish-docs.yml new file mode 100644 index 0000000..70edbae --- /dev/null +++ b/.github/workflows/publish-docs.yml @@ -0,0 +1,13 @@ +name: Deploy mdBook +on: [push] +jobs: + build: + runs-on: ubuntu-latest + permissions: + contents: write # To push a branch + pull-requests: write # To create a PR from that branch + steps: + - uses: actions/checkout@v4 + - uses: XAMPPRocky/deploy-mdbook@v1.1 + with: + token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 99ce9b4..54d2ba7 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -42,10 +42,6 @@ jobs: run: | python -W once -m testtools.run testrepository.tests.test_suite - - name: Docs - run: | - make -C doc html - success: needs: build runs-on: ubuntu-latest diff --git a/.markdownlint.yml b/.markdownlint.yml new file mode 100644 index 0000000..26d0c3b --- /dev/null +++ b/.markdownlint.yml @@ -0,0 +1,2 @@ +"MD025": false +"MD013": false \ No newline at end of file diff --git a/COPYING b/COPYING index 632a79e..ca0d9e3 100644 --- a/COPYING +++ b/COPYING @@ -27,7 +27,7 @@ limitations under that license. A concordance of contributors is maintained here to provide an easy reference for distributions such as Debian that wish to list all the copyright holders in their metadata: -* Robert Collins , 2009 +* Robert Collins , 2009, 2024 * Hewlett-Packard Development Company, L.P., 2013 * IBM Corp., 2013 diff --git a/doc/.gitignore b/doc/.gitignore new file mode 100644 index 0000000..7585238 --- /dev/null +++ b/doc/.gitignore @@ -0,0 +1 @@ +book diff --git a/doc/MANUAL.txt b/doc/MANUAL.txt deleted file mode 100644 index d0b298b..0000000 --- a/doc/MANUAL.txt +++ /dev/null @@ -1,422 +0,0 @@ -Test Repository users manual -++++++++++++++++++++++++++++ - -Overview -~~~~~~~~ - -Test repository is a small application for tracking test results. Any test run -that can be represented as a subunit stream can be inserted into a repository. - -Typical workflow is to have a repository into which test runs are inserted, and -then to query the repository to find out about issues that need addressing. -testr can fully automate this, but lets start with the low level facilities, -using the sample subunit stream included with testr:: - - # Note that there is a .testr.conf already: - ls .testr.conf - # Create a store to manage test results in. - $ testr init - # add a test result (shows failures) - $ testr load < doc/example-failing-subunit-stream - # see the tracked failing tests again - $ testr failing - # fix things - $ testr load < doc/example-passing-subunit-stream - # Now there are no tracked failing tests - $ testr failing - -Most commands in testr have comprehensive online help, and the commands:: - - $ testr help [command] - $ testr commands - -Will be useful to explore the system. - -Configuration -~~~~~~~~~~~~~ - -testr is configured via the '.testr.conf' file which needs to be in the same -directory that testr is run from. testr includes online help for all the -options that can be set within it:: - - $ testr help run - -Python ------- - -If your test suite is written in Python, the simplest - and usually correct -configuration is:: - - [DEFAULT] - test_command=python -m subunit.run discover . $LISTOPT $IDOPTION - test_id_option=--load-list $IDFILE - test_list_option=--list - -Running tests -~~~~~~~~~~~~~ - -testr is taught how to run your tests by interepreting your .testr.conf file. -For instance:: - - [DEFAULT] - test_command=foo $IDOPTION - test_id_option=--bar $IDFILE - -will cause 'testr run' to run 'foo' and process it as 'testr load' would. -Likewise 'testr run --failing' will automatically create a list file listing -just the failing tests, and then run 'foo --bar failing.list' and process it as -'testr load' would. failing.list will be a newline separated list of the -test ids that your test runner outputs. If there are no failing tests, no test -execution will happen at all. - -Arguments passed to 'testr run' are used to filter test ids that will be run - -testr will query the runner for test ids and then apply each argument as a -regex filter. Tests that match any of the given filters will be run. Arguments -passed to run after a ``--`` are passed through to your test runner command -line. For instance, using the above config example ``testr run quux -- bar ---no-plugins`` would query for test ids, filter for those that match 'quux' and -then run ``foo bar --load-list tempfile.list --no-plugins``. Shell variables -are expanded in these commands on platforms that have a shell. - -Having setup a .testr.conf, a common workflow then becomes:: - - # Fix currently broken tests - repeat until there are no failures. - $ testr run --failing - # Do a full run to find anything that regressed during the reduction process. - $ testr run - # And either commit or loop around this again depending on whether errors - # were found. - -The --failing option turns on ``--partial`` automatically (so that if the -partial test run were to be interrupted, the failing tests that aren't run are -not lost). - -Another common use case is repeating a failure that occured on a remote -machine (e.g. during a jenkins test run). There are two common ways to do -approach this. - -Firstly, if you have a subunit stream from the run you can just load it:: - - $ testr load < failing-stream - # Run the failed tests - $ testr run --failing - -The streams generated by test runs are in .testrepository/ named for their test -id - e.g. .testrepository/0 is the first stream. - -If you do not have a stream (because the test runner didn't output subunit or -you don't have access to the .testrepository) you may be able to use a list -file. If you can get a file that contains one test id per line, you can run -the named tests like this: - - $ testr run --load-list FILENAME - -This can also be useful when dealing with sporadically failing tests, or tests -that only fail in combination with some other test - you can bisect the tests -that were run to get smaller and smaller (or larger and larger) test subsets -until the error is pinpointed. - -``testr run --until-failure`` will run your test suite again and again and -again stopping only when interrupted or a failure occurs. This is useful -for repeating timing-related test failures. - -Listing tests -~~~~~~~~~~~~~ - -It is useful to be able to query the test program to see what tests will be -run - this permits partitioning the tests and running multiple instances with -separate partitions at once. Set 'test_list_option' in .testr.conf like so:: - - test_list_option=--list-tests - -You also need to use the $LISTOPT option to tell testr where to expand things: - - test_command=foo $LISTOPT $IDOPTION - -All the normal rules for invoking test program commands apply: extra parameters -will be passed through, if a test list is being supplied test_option can be -used via $IDOPTION. - -The output of the test command when this option is supplied should be a subunit -test enumeration. For subunit v1 that is a series of test ids, in any order, -``\n`` separated on stdout. For v2 use the subunit protocol and emit one event -per test with each test having status 'exists'. - -To test whether this is working the `testr list-tests` command can be useful. - -You can also use this to see what tests will be run by a given testr run -command. For instance, the tests that ``testr run myfilter`` will run are shown -by ``testr list-tests myfilter``. As with 'run', arguments to 'list-tests' are -used to regex filter the tests of the test runner, and arguments after a '--' -are passed to the test runner. - -Parallel testing -~~~~~~~~~~~~~~~~ - -If both test listing and filtering (via either IDLIST or IDFILE) are configured -then testr is able to run your tests in parallel:: - - $ testr run --parallel - -This will first list the tests, partition the tests into one partition per CPU -on the machine, and then invoke multiple test runners at the same time, with -each test runner getting one partition. Currently the partitioning algorithm -is simple round-robin for tests that testr has not seen run before, and -equal-time buckets for tests that testr has seen run. NB: This uses the anydbm -Python module to store the duration of each test. On some platforms (to date -only OSX) there is no bulk-update API and performance may be impacted if you -have many (10's of thousands) of tests. - -To determine how many CPUs are present in the machine, testrepository will -use the multiprocessing Python module (present since 2.6). On operating systems -where this is not implemented, or if you need to control the number of workers -that are used, the --concurrency option will let you do so:: - - $ testr run --parallel --concurrency=2 - -A more granular interface is available too - if you insert into .testr.conf:: - - test_run_concurrency=foo bar - -Then when testr needs to determine concurrency, it will run that command and -read the first line from stdout, cast that to an int, and use that as the -number of partitions to create. A count of 0 is interpreted to mean one -partition per test. For instance in .test.conf:: - - test_run_concurrency=echo 2 - -Would tell testr to use concurrency of 2. - -When running tests in parallel, testrepository tags each test with a tag for -the worker that executed the test. The tags are of the form ``worker-%d`` -and are usually used to reproduce test isolation failures, where knowing -exactly what test ran on a given backend is important. The %d that is -substituted in is the partition number of tests from the test run - all tests -in a single run with the same worker-N ran in the same test runner instance. - -To find out which slave a failing test ran on just look at the 'tags' line in -its test error:: - - ====================================================================== - label: testrepository.tests.ui.TestDemo.test_methodname - tags: foo worker-0 - ---------------------------------------------------------------------- - error text - -And then find tests with that tag:: - - $ testr last --subunit | subunit-filter -s --xfail --with-tag=worker-3 | subunit-ls > slave-3.list - -Grouping Tests -~~~~~~~~~~~~~~ - -In certain scenarios you may want to group tests of a certain type together -so that they will be run by the same backend. The group_regex option in -.testr.conf permits this. When set, tests are grouped by the group(0) of any -regex match. Tests with no match are not grouped. - -For example, extending the python sample .testr.conf from the configuration -section with a group regex that will group python tests cases together by -class (the last . splits the class and test method):: - - [DEFAULT] - test_command=python -m subunit.run discover . $LISTOPT $IDOPTION - test_id_option=--load-list $IDFILE - test_list_option=--list - group_regex=([^\.]+\.)+ - - -Remote or isolated test environments -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A common problem with parallel test running is test runners that use global -resources such as well known ports, well known database names or predictable -directories on disk. - -One way to solve this is to setup isolated environments such as chroots, -containers or even separate machines. Such environments typically require -some coordination when being used to run tests, so testr provides an explicit -model for working with them. - -The model testr has is intended to support both developers working -incrementally on a change and CI systems running tests in a one-off setup, -for both statically and dynamically provisioned environments. - -The process testr follows is: - -1. The user should perform any one-time or once-per-session setup. For instance, - checking out source code, creating a template container, sourcing your cloud - credentials. -2. Execute testr run. -3. testr queries for concurrency. -4. testr will make a callout request to provision that many instances. - The provisioning callout needs to synchronise source code and do any other - per-instance setup at this stage. -5. testr will make callouts to execute tests, supplying files that should be - copied into the execution environment. Note that instances may be used for - more than one command execution. -6. testr will callout to dispose of the instances after the test run completes. - -Instances may be expensive to create and dispose of. testr does not perform -any caching, but the callout pattern is intended to facilitate external -caching - the provisioning callout can be used to pull environments out of -a cache, and the dispose to just return it to the cache. - -Configuring environment support -------------------------------- - -There are three callouts that testrepository depends on - configured in -.testr.conf as usual. For instance:: - - instance_provision=foo -c $INSTANCE_COUNT - instance_dispose=bar $INSTANCE_IDS - instance_execute=quux $INSTANCE_ID $FILES -- $COMMAND - -These should operate as follows: - -* instance_provision should start up the number of instances provided in the - $INSTANCE_COUNT parameter. It should print out on stdout the instance ids - that testr should supply to the dispose and execute commands. There should - be no other output on stdout (stderr is entirely up for grabs). An exit code - of non-zero will cause testr to consider the command to have failed. A - provisioned instance should be able to execute the list tests command and - execute tests commands that testr will run via the instance_execute callout. - Its possible to lazy-provision things if you desire - testr doesn't care - - but to reduce latency we suggest performing any rsync or other code - synchronisation steps during the provision step, as testr may make multiple - calls to one environment, and re-doing costly operations on each command - execution would impair performance. - -* instance_dispose should take a list of instance ids and get rid of them - this might mean putting them back in a pool of instances, or powering them - off, or terminating them - whatever makes sense for your project. - -* instance_execute should accept an instance id, a list of files that need to - be copied into the instance and a command to run within the instance. It - needs to copy those files into the instance (it may adjust their paths if - desired). If the paths are adjusted, the same paths within $COMMAND should be - adjusted to match. Execution that takes place with a shared filesystem can - obviously skip file copying or adjusting (and the $FILES parameter). When the - instance_execute terminates, it should use the exit code that the command - used within the instance. Stdout and stderr from instance_execute are - presumed to be that of $COMMAND. In particular, stdout is where the subunit - test output, and subunit test listing output, are expected, and putting other - output into stdout can lead to surprising results - such as corrupting the - subunit stream. - instance_execute is invoked for both test listing and test executing - callouts. - -Hiding tests -~~~~~~~~~~~~ - -Some test runners (for instance, zope.testrunner) report pseudo tests having to -do with bringing up the test environment rather than being actual tests that -can be executed. These are only relevant to a test run when they fail - the -rest of the time they tend to be confusing. For instance, the same 'test' may -show up on multiple parallel test runs, which will inflate the 'executed tests' -count depending on the number of worker threads that were used. Scheduling such -'tests' to run is also a bit pointless, as they are only ever executed -implicitly when preparing (or finishing with) a test environment to run other -tests in. - -testr can ignore such tests if they are tagged, using the filter_tags -configuration option. Tests tagged with any tag in that (space separated) list -will only be included in counts and reports if the test failed (or errored). - -Automated test isolation bisection -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -As mentioned above, its possible to manually analyze test isolation issues by -interrogating the repository for which tests ran on which worker, and then -creating a list file with those tests, re-running only half of them, checking -the error still happens, rinse and repeat. - -However that is tedious. testr can perform this analysis for you:: - - $ testr run --analyze-isolation - -will perform that analysis for you. (This requires that your test runner is -(mostly) deterministic on test ordering). The process is: - -1. The last run in the repository is used as a basis for analysing against - - tests are only cross checked against tests run in the same worker in that - run. This means that failures accrued from several different runs would not - be processed with the right basis tests - you should do a full test run to - seed your repository. This can be local, or just testr load a full run from - your Jenkins or other remote run environment. - -2. Each test that is currently listed as a failure is run in a test process - given just that id to run. - -3. Tests that fail are excluded from analysis - they are broken on their own. - -4. The remaining failures are then individually analysed one by one. - -5. For each failing, it gets run in one work along with the first 1/2 of the - tests that were previously run prior to it. - -6. If the test now passes, that set of prior tests are discarded, and the - other half of the tests is promoted to be the full list. If the test fails - then other other half of the tests are discarded and the current set - promoted. - -7. Go back to running the failing test along with 1/2 of the current list of - priors unless the list only has 1 test in it. If the failing test still - failed with that test, we have found the isolation issue. If it did not - then either the isolation issue is racy, or it is a 3-or-more test - isolation issue. Neither of those cases are automated today. - -Forcing isolation -~~~~~~~~~~~~~~~~~ - -Sometimes it is useful to force a separate test runner instance for each test -executed. The ``--isolated`` flag will cause testr to execute a separate runner -per test:: - - $ testr run --isolated - -In this mode testr first determines tests to run (either automatically listed, -using the failing set, or a user supplied load-list), and then spawns one test -runner per test it runs. To avoid cross-test-runner interactions concurrency -is disabled in this mode. ``--analyze-isolation`` supercedes ``--isolated`` if -they are both supplied. - -Repositories -~~~~~~~~~~~~ - -A testr repository is a very simple disk structure. It contains the following -files (for a format 1 repository - the only current format): - -* format: This file identifies the precise layout of the repository, in case - future changes are needed. - -* next-stream: This file contains the serial number to be used when adding another - stream to the repository. - -* failing: This file is a stream containing just the known failing tests. It - is updated whenever a new stream is added to the repository, so that it only - references known failing tests. - -* #N - all the streams inserted in the repository are given a serial number. - -* repo.conf: This file contains user configuration settings for the repository. - ``testr repo-config`` will dump a repo configration and - ``test help repo-config`` has online help for all the repository settings. - -setuptools integration -~~~~~~~~~~~~~~~~~~~~~~ - -testrepository provides a setuptools commands for ease of integration with -setuptools-based workflows: - -* testr: - ``python setup.py testr`` will run testr in parallel mode - Options that would normally be passed to testr run can be added to the - testr-options argument. - ``python setup.py testr --testr-options="--failing"`` will append --failing - to the test run. -* testr --coverage: - ``python setup.py testr --coverage`` will run testr in code coverage mode. This - assumes the installation of the python coverage module. -* ``python testr --coverage --omit=ModuleThatSucks.py`` will append - --omit=ModuleThatSucks.py to the coverage report command. diff --git a/doc/Makefile b/doc/Makefile deleted file mode 100644 index 4d42bd7..0000000 --- a/doc/Makefile +++ /dev/null @@ -1,153 +0,0 @@ -# Makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -PAPER = -BUILDDIR = _build - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . -# the i18n builder cannot share the environment and doctrees with the others -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext - -help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " singlehtml to make a single large HTML file" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " devhelp to make HTML files and a Devhelp project" - @echo " epub to make an epub" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " latexpdf to make LaTeX files and run them through pdflatex" - @echo " text to make text files" - @echo " man to make manual pages" - @echo " texinfo to make Texinfo files" - @echo " info to make Texinfo files and run them through makeinfo" - @echo " gettext to make PO message catalogs" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - -clean: - -rm -rf $(BUILDDIR)/* - -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." - -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." - -singlehtml: - $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml - @echo - @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." - -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/TestRepository.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/TestRepository.qhc" - -devhelp: - $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp - @echo - @echo "Build finished." - @echo "To view the help file:" - @echo "# mkdir -p $$HOME/.local/share/devhelp/TestRepository" - @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/TestRepository" - @echo "# devhelp" - -epub: - $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub - @echo - @echo "Build finished. The epub file is in $(BUILDDIR)/epub." - -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make' in that directory to run these through (pdf)latex" \ - "(use \`make latexpdf' here to do that automatically)." - -latexpdf: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through pdflatex..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -text: - $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text - @echo - @echo "Build finished. The text files are in $(BUILDDIR)/text." - -man: - $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man - @echo - @echo "Build finished. The manual pages are in $(BUILDDIR)/man." - -texinfo: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo - @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." - @echo "Run \`make' in that directory to run these through makeinfo" \ - "(use \`make info' here to do that automatically)." - -info: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo "Running Texinfo files through makeinfo..." - make -C $(BUILDDIR)/texinfo info - @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." - -gettext: - $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale - @echo - @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." - -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." diff --git a/doc/book.toml b/doc/book.toml new file mode 100644 index 0000000..963341f --- /dev/null +++ b/doc/book.toml @@ -0,0 +1,6 @@ +[book] +authors = ["Robert Collins"] +language = "en" +multilingual = false +src = "src" +title = "Testrepository" diff --git a/doc/conf.py b/doc/conf.py deleted file mode 100644 index c482fbe..0000000 --- a/doc/conf.py +++ /dev/null @@ -1,242 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Test Repository documentation build configuration file, created by -# sphinx-quickstart on Mon Dec 3 23:24:00 2012. -# -# This file is execfile()d with the current directory set to its containing dir. -# -# Note that not all possible configuration values are present in this -# autogenerated file. -# -# All configuration values have a default; values that are commented out -# serve to show the default. - -import sys, os - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -#sys.path.insert(0, os.path.abspath('.')) - -# -- General configuration ----------------------------------------------------- - -# If your documentation needs a minimal Sphinx version, state it here. -#needs_sphinx = '1.0' - -# Add any Sphinx extension module names here, as strings. They can be extensions -# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = [] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# The suffix of source filenames. -source_suffix = '.txt' - -# The encoding of source files. -#source_encoding = 'utf-8-sig' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'Test Repository' -copyright = u'2012, Testrepository Contributors' - -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. -# -# The short X.Y version. -# version = 'trunk' -# The full version, including alpha/beta/rc tags. -# release = 'trunk' - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -#language = None - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -exclude_patterns = ['_build'] - -# The reST default role (used for this markup: `text`) to use for all documents. -#default_role = None - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] - - -# -- Options for HTML output --------------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. -html_theme = 'nature' - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -#html_theme_options = {} - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -#html_title = None - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -#html_logo = None - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -#html_favicon = None - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_domain_indices = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True - -# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True - -# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None - -# Output file base name for HTML help builder. -htmlhelp_basename = 'TestRepositorydoc' - - -# -- Options for LaTeX output -------------------------------------------------- - -latex_elements = { -# The paper size ('letterpaper' or 'a4paper'). -#'papersize': 'letterpaper', - -# The font size ('10pt', '11pt' or '12pt'). -#'pointsize': '10pt', - -# Additional stuff for the LaTeX preamble. -#'preamble': '', -} - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, author, documentclass [howto/manual]). -latex_documents = [ - ('index', 'TestRepository.tex', u'Test Repository Documentation', - u'Testrepository Contributors', 'manual'), -] - -# The name of an image file (relative to this directory) to place at the top of -# the title page. -#latex_logo = None - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# If true, show page references after internal links. -#latex_show_pagerefs = False - -# If true, show URL addresses after external links. -#latex_show_urls = False - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_domain_indices = True - - -# -- Options for manual page output -------------------------------------------- - -# One entry per manual page. List of tuples -# (source start file, name, description, authors, manual section). -man_pages = [ - ('index', 'testrepository', u'Test Repository Documentation', - [u'Testrepository Contributors'], 1) -] - -# If true, show URL addresses after external links. -#man_show_urls = False - - -# -- Options for Texinfo output ------------------------------------------------ - -# Grouping the document tree into Texinfo files. List of tuples -# (source start file, target name, title, author, -# dir menu entry, description, category) -texinfo_documents = [ - ('index', 'TestRepository', u'Test Repository Documentation', - u'Testrepository Contributors', 'TestRepository', 'One line description of project.', - 'Miscellaneous'), -] - -# Documents to append as an appendix to all manuals. -#texinfo_appendices = [] - -# If false, no module index is generated. -#texinfo_domain_indices = True - -# How to display URL addresses: 'footnote', 'no', or 'inline'. -#texinfo_show_urls = 'footnote' diff --git a/doc/index.txt b/doc/index.txt deleted file mode 100644 index 720cb5f..0000000 --- a/doc/index.txt +++ /dev/null @@ -1,25 +0,0 @@ -.. Test Repository documentation master file, created by - sphinx-quickstart on Mon Dec 3 23:24:00 2012. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to Test Repository's documentation! -=========================================== - -Contents: - -.. toctree:: - :maxdepth: 2 - - MANUAL - DESIGN - DEVELOPERS - - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`search` - diff --git a/doc/make.bat b/doc/make.bat deleted file mode 100644 index fd4663f..0000000 --- a/doc/make.bat +++ /dev/null @@ -1,190 +0,0 @@ -@ECHO OFF - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set BUILDDIR=_build -set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . -set I18NSPHINXOPTS=%SPHINXOPTS% . -if NOT "%PAPER%" == "" ( - set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% - set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% -) - -if "%1" == "" goto help - -if "%1" == "help" ( - :help - echo.Please use `make ^` where ^ is one of - echo. html to make standalone HTML files - echo. dirhtml to make HTML files named index.html in directories - echo. singlehtml to make a single large HTML file - echo. pickle to make pickle files - echo. json to make JSON files - echo. htmlhelp to make HTML files and a HTML help project - echo. qthelp to make HTML files and a qthelp project - echo. devhelp to make HTML files and a Devhelp project - echo. epub to make an epub - echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter - echo. text to make text files - echo. man to make manual pages - echo. texinfo to make Texinfo files - echo. gettext to make PO message catalogs - echo. changes to make an overview over all changed/added/deprecated items - echo. linkcheck to check all external links for integrity - echo. doctest to run all doctests embedded in the documentation if enabled - goto end -) - -if "%1" == "clean" ( - for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i - del /q /s %BUILDDIR%\* - goto end -) - -if "%1" == "html" ( - %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/html. - goto end -) - -if "%1" == "dirhtml" ( - %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. - goto end -) - -if "%1" == "singlehtml" ( - %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. - goto end -) - -if "%1" == "pickle" ( - %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the pickle files. - goto end -) - -if "%1" == "json" ( - %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the JSON files. - goto end -) - -if "%1" == "htmlhelp" ( - %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run HTML Help Workshop with the ^ -.hhp project file in %BUILDDIR%/htmlhelp. - goto end -) - -if "%1" == "qthelp" ( - %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run "qcollectiongenerator" with the ^ -.qhcp project file in %BUILDDIR%/qthelp, like this: - echo.^> qcollectiongenerator %BUILDDIR%\qthelp\TestRepository.qhcp - echo.To view the help file: - echo.^> assistant -collectionFile %BUILDDIR%\qthelp\TestRepository.ghc - goto end -) - -if "%1" == "devhelp" ( - %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. - goto end -) - -if "%1" == "epub" ( - %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The epub file is in %BUILDDIR%/epub. - goto end -) - -if "%1" == "latex" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "text" ( - %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The text files are in %BUILDDIR%/text. - goto end -) - -if "%1" == "man" ( - %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The manual pages are in %BUILDDIR%/man. - goto end -) - -if "%1" == "texinfo" ( - %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. - goto end -) - -if "%1" == "gettext" ( - %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The message catalogs are in %BUILDDIR%/locale. - goto end -) - -if "%1" == "changes" ( - %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes - if errorlevel 1 exit /b 1 - echo. - echo.The overview file is in %BUILDDIR%/changes. - goto end -) - -if "%1" == "linkcheck" ( - %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck - if errorlevel 1 exit /b 1 - echo. - echo.Link check complete; look for any errors in the above output ^ -or in %BUILDDIR%/linkcheck/output.txt. - goto end -) - -if "%1" == "doctest" ( - %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest - if errorlevel 1 exit /b 1 - echo. - echo.Testing of doctests in the sources finished, look at the ^ -results in %BUILDDIR%/doctest/output.txt. - goto end -) - -:end diff --git a/doc/src/SUMMARY.md b/doc/src/SUMMARY.md new file mode 100644 index 0000000..ab94544 --- /dev/null +++ b/doc/src/SUMMARY.md @@ -0,0 +1,25 @@ +# Summary + +[Overview](./overview.md) + +# Manual + +- [Configuration](./configuration.md) +- [Running tests](./running-tests.md) +- [Listing tests](./listing-tests.md) +- [Parallel testing](./parallel-testing.md) +- [Grouping tests](./grouping-tests.md) +- [Remote or isolated test environments](./remote-or-isolated-test-environments.md) +- [Hiding tests](./hiding-tests.md) +- [Automated test isolation bisection](./automated-test-isolation-bisection.md) +- [Forcing isolation](./forcing-isolation.md) +- [Repositories](./repositories.md) +- [Setuptools integration](./setuptools-integration.md) + +# Developers + +- [Development guidelines](./developers/development-guidelines.md) + +# Design + +- [Design](./design/design.md) \ No newline at end of file diff --git a/doc/src/automated-test-isolation-bisection.md b/doc/src/automated-test-isolation-bisection.md new file mode 100644 index 0000000..d938b8c --- /dev/null +++ b/doc/src/automated-test-isolation-bisection.md @@ -0,0 +1,45 @@ +# Automated test isolation bisection + +As mentioned previously, it is possible to manually analyze test isolation +issues by interrogating the repository for which tests ran on which worker, and +then creating a list file with those tests, re-running only half of them, +checking the error still happens, rinse and repeat. + +However that is tedious. testr can perform this analysis for you + +```sh + $ testr run --analyze-isolation +``` + +will perform that analysis for you. (This requires that your test runner is +(mostly) deterministic on test ordering). The process is: + +1. The last run in the repository is used as a basis for analysing against - + tests are only cross checked against tests run in the same worker in that + run. This means that failures accrued from several different runs would not + be processed with the right basis tests - you should do a full test run to + seed your repository. This can be local, or just testr load a full run from + your Jenkins or other remote run environment. + +2. Each test that is currently listed as a failure is run in a test process + given just that id to run. + +3. Tests that fail are excluded from analysis - they are broken on their own. + +4. The remaining failures are then individually analysed one by one. + +5. For each failing, it gets run in one work along with the first 1/2 of the + tests that were previously run prior to it. + +6. If the test now passes, that set of prior tests are discarded, and the + other half of the tests is promoted to be the full list. If the test fails + then other other half of the tests are discarded and the current set + promoted. + +7. Go back to running the failing test along with 1/2 of the current list of + priors unless the list only has 1 test in it. If the failing test still + failed with that test, we have found the isolation issue. If it did not + then either the isolation issue is racy, or it is a 3-or-more test + isolation issue. Neither of those cases are automated today. + +This cannot prove the absence of interactions - for instance, a runner that randomises the order of tests executing combined with a failure that occurs with A before B but not B before A could easily appear to be isolated when it is not. diff --git a/doc/src/chapter_1.md b/doc/src/chapter_1.md new file mode 100644 index 0000000..b743fda --- /dev/null +++ b/doc/src/chapter_1.md @@ -0,0 +1 @@ +# Chapter 1 diff --git a/doc/src/configuration.md b/doc/src/configuration.md new file mode 100644 index 0000000..aef16aa --- /dev/null +++ b/doc/src/configuration.md @@ -0,0 +1,21 @@ +# Configuration + +testr is configured via the `.testr.conf` file which needs to be in the same +directory that testr is run from. testr includes online help for all the +options that can be set within it: + +```sh + $ testr help run +``` + +## Python + +If your test suite is written in Python, the simplest - and usually correct +configuration is: + +```ini + [DEFAULT] + test_command=python -m subunit.run discover . $LISTOPT $IDOPTION + test_id_option=--load-list $IDFILE + test_list_option=--list +``` diff --git a/doc/DESIGN.txt b/doc/src/design/design.md similarity index 64% rename from doc/DESIGN.txt rename to doc/src/design/design.md index 812d2b8..9332577 100644 --- a/doc/DESIGN.txt +++ b/doc/src/design/design.md @@ -1,30 +1,23 @@ -Design / Architecture of Test Repository -++++++++++++++++++++++++++++++++++++++++ +# Design -Values -~~~~~~ +## Values Code reuse. Focus on the project. Do one thing well. -Goals -~~~~~ +## Goals -Achieve a Clean UI, responsive UI, small-tools approach. Simulataneously have +Achieve a Clean UI, responsive UI, small-tools approach. Simultaneously have a small clean code base which is easily approachable. -Data model/storage -~~~~~~~~~~~~~~~~~~ +## Data model/storage testrepository stores subunit streams as subunit streams in .testrespository -with simple additional metadata. See the MANUAL for documentation on the -repository layout. The key design elements are that streams are stored -verbatim, and a testr managed stream called 'failing' is used to track the -current failures. +with simple additional metadata. See the [manual](../repositories.md) for documentation on the repository layout. +The key design elements are that streams are stored verbatim, and a testr managed stream called 'failing' is used to track the current failures. -Code layout -~~~~~~~~~~~ +## Code layout One conceptual thing per module, packages for anything where multiple types are expected (e.g. testrepository.commands, testrespository.ui). @@ -38,15 +31,13 @@ The tests for the code in testrepository.foo.bar is in testrepository.tests.foo.test_bar. Interface tests for testrepository.foo is in testrepository.tests.test_foo. -External integration -~~~~~~~~~~~~~~~~~~~~ +## External integration + Test Repository command, ui, parsing etc objects should all be suitable for reuse from other programs. -Threads/concurrency -~~~~~~~~~~~~~~~~~~~ +## Threads/concurrency In general using any public interface is fine, but keeping syncronisation needs to a minimum for code readability. - diff --git a/doc/DEVELOPERS.txt b/doc/src/developers/development-guidelines.md similarity index 68% rename from doc/DEVELOPERS.txt rename to doc/src/developers/development-guidelines.md index c9d766d..814cc88 100644 --- a/doc/DEVELOPERS.txt +++ b/doc/src/developers/development-guidelines.md @@ -1,14 +1,11 @@ -Development guidelines for Test Repository -++++++++++++++++++++++++++++++++++++++++++ +# Development guidelines -Coding style -~~~~~~~~~~~~ +## Coding style PEP-8 please. We don't enforce a particular style, but being reasonably consistent aids readability. -Copyrights and licensing -~~~~~~~~~~~~~~~~~~~~~~~~ +## Copyrights and licensing Code committed to Test Repository must be licensed under the BSD + Apache-2.0 licences that Test Repository offers its users. Copyright assignment is not @@ -17,38 +14,30 @@ a given source file. Lastly, all copyright holders need to add their name to the master list in COPYING the first time they make a change in a given calendar year. -Testing and QA -~~~~~~~~~~~~~~ +## Testing and QA For Test repository please add tests where possible. There is no requirement for one test per change (because somethings are much harder to automatically test than the benfit from such tests). Fast tests are preferred to slow tests, and understandable tests to fast tests. -http://build.robertcollins.net/ has a job testing every commit made to trunk -of Test Repository, and there is no automated test-before-merge process at the -moment. The quid pro quo for committers is that they should check that the -automated job found their change acceptable after merging it, and either roll -it back or fix it immediately. A broken trunk is not acceptable! +CI is done via Github Actions. A broken trunk is not acceptable! See DESIGN.txt for information about code layout which will help you find where to add tests (and indeed where to change things). -Running the tests ------------------ +### Running the tests Generally just ``make`` is all that is needed to run all the tests. However if dropping into pdb, it is currently more convenient to use ``python -m testtools.run testrepository.tests.test_suite``. -Diagnosing issues ------------------ +### Diagnosing issues The cli UI will drop into pdb when an error is thrown if TESTR_PDB is set in the environment. This can be very useful for diagnosing problems. -Releasing ---------- +### Releasing Update NEWS and testrepository/__init__.py version numbers. Release to pypi. Pivot the next milestone on LP to version, and make a new next milestone. diff --git a/doc/src/forcing-isolation.md b/doc/src/forcing-isolation.md new file mode 100644 index 0000000..fc2d83d --- /dev/null +++ b/doc/src/forcing-isolation.md @@ -0,0 +1,15 @@ +# Forcing isolation + +Sometimes it is useful to force a separate test runner instance for each test +executed. The `--isolated` flag will cause testr to execute a separate runner +per test + +```sh + $ testr run --isolated +``` + +In this mode testr first determines tests to run (either automatically listed, +using the failing set, or a user supplied load-list), and then spawns one test +runner per test it runs. To avoid cross-test-runner interactions concurrency +is disabled in this mode. `--analyze-isolation` supercedes `--isolated` if +they are both supplied. diff --git a/doc/src/grouping-tests.md b/doc/src/grouping-tests.md new file mode 100644 index 0000000..d9c7c94 --- /dev/null +++ b/doc/src/grouping-tests.md @@ -0,0 +1,18 @@ +# Grouping tests + +In certain scenarios you may want to group tests of a certain type together +so that they will be run by the same backend. The group_regex option in +.testr.conf permits this. When set, tests are grouped by the group(0) of any +regex match. Tests with no match are not grouped. + +For example, extending the python sample .testr.conf from the configuration +section with a group regex that will group python tests cases together by +class (the last . splits the class and test method) + +```ini + [DEFAULT] + test_command=python -m subunit.run discover . $LISTOPT $IDOPTION + test_id_option=--load-list $IDFILE + test_list_option=--list + group_regex=([^\.]+\.)+ +``` diff --git a/doc/src/hiding-tests.md b/doc/src/hiding-tests.md new file mode 100644 index 0000000..6c04bf0 --- /dev/null +++ b/doc/src/hiding-tests.md @@ -0,0 +1,15 @@ +# Hiding tests + +Some test runners (for instance, `zope.testrunner`) report pseudo tests having to +do with bringing up the test environment rather than being actual tests that +can be executed. These are only relevant to a test run when they fail - the +rest of the time they tend to be confusing. For instance, the same 'test' may +show up on multiple parallel test runs, which will inflate the 'executed tests' +count depending on the number of worker threads that were used. Scheduling such +'tests' to run is also a bit pointless, as they are only ever executed +implicitly when preparing (or finishing with) a test environment to run other +tests in. + +testr can ignore such tests if they are tagged, using the filter_tags +configuration option. Tests tagged with any tag in that (space separated) list +will only be included in counts and reports if the test failed (or errored). diff --git a/doc/src/listing-tests.md b/doc/src/listing-tests.md new file mode 100644 index 0000000..3c2f012 --- /dev/null +++ b/doc/src/listing-tests.md @@ -0,0 +1,32 @@ +# Listing Tests + +It is useful to be able to query the test program to see what tests will be +run - this permits partitioning the tests and running multiple instances with +separate partitions at once. Set 'test_list_option' in .testr.conf like so: + +```ini + test_list_option=--list-tests +``` + +You also need to use the $LISTOPT option to tell testr where to expand things: + +```ini + test_command=foo $LISTOPT $IDOPTION +``` + +All the normal rules for invoking test program commands apply: extra parameters +will be passed through, if a test list is being supplied test_option can be +used via `$IDOPTION`. + +The output of the test command when this option is supplied should be a subunit +test enumeration. For subunit v1 that is a series of test ids, in any order, +``\n`` separated on stdout. For v2 use the subunit protocol and emit one event +per test with each test having status 'exists'. + +To test whether this is working the `testr list-tests` command can be useful. + +You can also use this to see what tests will be run by a given testr run +command. For instance, the tests that `testr run myfilter` will run are shown +by `testr list-tests myfilter`. As with `run`, arguments to `list-tests` are +used to regex filter the tests of the test runner, and arguments after a `--` +are passed to the test runner. diff --git a/doc/src/overview.md b/doc/src/overview.md new file mode 100644 index 0000000..9869c2e --- /dev/null +++ b/doc/src/overview.md @@ -0,0 +1,33 @@ +# Overview + +Test repository is a small application for tracking test results. Any test run +that can be represented as a subunit stream can be inserted into a repository. + +Typical workflow is to have a repository into which test runs are inserted, and +then to query the repository to find out about issues that need addressing. +testr can fully automate this, but lets start with the low level facilities, +using the sample subunit stream included with testr + +```sh + # Note that there is a .testr.conf already: + ls .testr.conf + # Create a store to manage test results in. + $ testr init + # add a test result (shows failures) + $ testr load < examples/example-failing-subunit-stream + # see the tracked failing tests again + $ testr failing + # fix things + $ testr load < examples/example-passing-subunit-stream + # Now there are no tracked failing tests + $ testr failing +``` + +Most commands in testr have comprehensive online help, and the commands + +```sh + $ testr help [command] + $ testr commands +``` + +Will be useful to explore the system. diff --git a/doc/src/parallel-testing.md b/doc/src/parallel-testing.md new file mode 100644 index 0000000..074f7cb --- /dev/null +++ b/doc/src/parallel-testing.md @@ -0,0 +1,66 @@ +# Parallel testing + +If both test listing and filtering (via either `IDLIST` or `IDFILE`) are configured then testr is able to run your tests in parallel + +```sh + $ testr run --parallel +``` + +This will first list the tests, partition the tests into one partition per CPU +on the machine, and then invoke multiple test runners at the same time, with +each test runner getting one partition. Currently the partitioning algorithm +is simple round-robin for tests that testr has not seen run before, and +equal-time buckets for tests that testr has seen run. NB: This uses the anydbm +Python module to store the duration of each test. On some platforms (to date +only OSX) there is no bulk-update API and performance may be impacted if you +have many (10's of thousands) of tests. + +To determine how many CPUs are present in the machine, testrepository will +use the multiprocessing Python module (present since 2.6). On operating systems +where this is not implemented, or if you need to control the number of workers +that are used, the --concurrency option will let you do so + +```sh + $ testr run --parallel --concurrency=2 +``` + +A more granular interface is available too - if you insert into .testr.conf + +```ini + test_run_concurrency=foo bar +``` + +Then when testr needs to determine concurrency, it will run that command and +read the first line from stdout, cast that to an int, and use that as the +number of partitions to create. A count of 0 is interpreted to mean one +partition per test. For instance in .test.conf + +```ini + test_run_concurrency=echo 2 +``` + +Would tell testr to use concurrency of 2. + +When running tests in parallel, testrepository tags each test with a tag for +the worker that executed the test. The tags are of the form `worker-%d` +and are usually used to reproduce test isolation failures, where knowing +exactly what test ran on a given backend is important. The `%d` that is +substituted in is the partition number of tests from the test run - all tests +in a single run with the same `worker-N` ran in the same test runner instance. + +To find out which slave a failing test ran on just look at the `tags` line in +its test error + +```text + ====================================================================== + label: testrepository.tests.ui.TestDemo.test_methodname + tags: foo worker-0 + ---------------------------------------------------------------------- + error text +``` + +And then find tests with that tag + +```sh + $ testr last --subunit | subunit-filter -s --xfail --with-tag=worker-3 | subunit-ls > slave-3.list +``` diff --git a/doc/src/remote-or-isolated-test-environments.md b/doc/src/remote-or-isolated-test-environments.md new file mode 100644 index 0000000..2d72d27 --- /dev/null +++ b/doc/src/remote-or-isolated-test-environments.md @@ -0,0 +1,79 @@ +# Remote or isolated test environments + +A common problem with parallel test running is test runners that use global +resources such as well known ports, well known database names or predictable +directories on disk. + +One way to solve this is to setup isolated environments such as chroots, +containers or even separate machines. Such environments typically require +some coordination when being used to run tests, so testr provides an explicit +model for working with them. + +The model testr has is intended to support both developers working +incrementally on a change and CI systems running tests in a one-off setup, +for both statically and dynamically provisioned environments. + +The process testr follows is: + +1. The user should perform any one-time or once-per-session setup. For instance, + checking out source code, creating a template container, sourcing your cloud + credentials. +2. Execute testr run. +3. testr queries for concurrency. +4. testr will make a callout request to provision that many instances. + The provisioning callout needs to synchronise source code and do any other + per-instance setup at this stage. +5. testr will make callouts to execute tests, supplying files that should be + copied into the execution environment. Note that instances may be used for + more than one command execution. +6. testr will callout to dispose of the instances after the test run completes. + +Instances may be expensive to create and dispose of. testr does not perform +any caching, but the callout pattern is intended to facilitate external +caching - the provisioning callout can be used to pull environments out of +a cache, and the dispose to just return it to the cache. + +## Configuring environment support + +There are three callouts that testrepository depends on - configured in +.testr.conf as usual. For instance + +```ini + instance_provision=foo -c $INSTANCE_COUNT + instance_dispose=bar $INSTANCE_IDS + instance_execute=quux $INSTANCE_ID $FILES -- $COMMAND +``` + +These should operate as follows: + +* instance_provision should start up the number of instances provided in the + `$INSTANCE_COUNT` parameter. It should print out on stdout the instance ids + that testr should supply to the dispose and execute commands. There should + be no other output on stdout (stderr is entirely up for grabs). An exit code + of non-zero will cause testr to consider the command to have failed. A + provisioned instance should be able to execute the list tests command and + execute tests commands that testr will run via the instance_execute callout. + Its possible to lazy-provision things if you desire - testr doesn't care - + but to reduce latency we suggest performing any rsync or other code + synchronisation steps during the provision step, as testr may make multiple + calls to one environment, and re-doing costly operations on each command + execution would impair performance. + +* instance_dispose should take a list of instance ids and get rid of them + this might mean putting them back in a pool of instances, or powering them + off, or terminating them - whatever makes sense for your project. + +* instance_execute should accept an instance id, a list of files that need to + be copied into the instance and a command to run within the instance. It + needs to copy those files into the instance (it may adjust their paths if + desired). If the paths are adjusted, the same paths within `$COMMAND` should be + adjusted to match. Execution that takes place with a shared filesystem can + obviously skip file copying or adjusting (and the $FILES parameter). When the + instance_execute terminates, it should use the exit code that the command + used within the instance. Stdout and stderr from instance_execute are + presumed to be that of `$COMMAND`. In particular, stdout is where the subunit + test output, and subunit test listing output, are expected, and putting other + output into stdout can lead to surprising results - such as corrupting the + subunit stream. + instance_execute is invoked for both test listing and test executing + callouts. diff --git a/doc/src/repositories.md b/doc/src/repositories.md new file mode 100644 index 0000000..23431c5 --- /dev/null +++ b/doc/src/repositories.md @@ -0,0 +1,16 @@ +# Repositories + +A testr repository is a very simple disk structure. It contains the following +files (for a format 1 repository - the only current format): + +* `format`: This file identifies the precise layout of the repository, in case future changes are needed. + +* `next-stream`: This file contains the serial number to be used when adding another stream to the repository. + +* `failing`: This file is a stream containing just the known failing tests. + It is updated whenever a new stream is added to the repository, so that it only references known failing tests. + +* `#N` - all the streams inserted in the repository are given a serial number. + +* `repo.conf`: This file contains user configuration settings for the repository. + `testr repo-config` will dump a repo configration and `test help repo-config` has online help for all the repository settings. diff --git a/doc/src/running-tests.md b/doc/src/running-tests.md new file mode 100644 index 0000000..d5dd16c --- /dev/null +++ b/doc/src/running-tests.md @@ -0,0 +1,74 @@ +# Running Tests + +testr is taught how to run your tests by interepreting your .testr.conf file. +For instance: + +```ini + [DEFAULT] + test_command=foo $IDOPTION + test_id_option=--bar $IDFILE +``` + +will cause `testr run` to run `foo` and process it as `testr load` would. +Likewise `testr run --failing` will automatically create a list file listing +just the failing tests, and then run `foo --bar failing.list` and process it as +`testr load` would. failing.list will be a newline separated list of the +test ids that your test runner outputs. If there are no failing tests, no test +execution will happen at all. + +Arguments passed to `testr run` are used to filter test ids that will be run - +testr will query the runner for test ids and then apply each argument as a +regex filter. Tests that match any of the given filters will be run. Arguments +passed to run after a `--` are passed through to your test runner command +line. For instance, using the above config example `testr run quux -- bar +--no-plugins` would query for test ids, filter for those that match `quux` and +then run `foo bar --load-list tempfile.list --no-plugins`. Shell variables +are expanded in these commands on platforms that have a shell. + +Having setup a .testr.conf, a common workflow then becomes: + +```sh + # Fix currently broken tests - repeat until there are no failures. + $ testr run --failing + # Do a full run to find anything that regressed during the reduction process. + $ testr run + # And either commit or loop around this again depending on whether errors + # were found. +``` + +The `--failing option` turns on `--partial` automatically (so that if the +partial test run were to be interrupted, the failing tests that aren't run are +not lost). + +Another common use case is repeating a failure that occured on a remote +machine (e.g. during a jenkins test run). There are two common ways to do +approach this. + +Firstly, if you have a subunit stream from the run you can just load it: + +```sh + $ testr load < failing-stream + # Run the failed tests + $ testr run --failing +``` + +The streams generated by test runs are in `.testrepository/` named for their test +id - e.g. `.testrepository/0` is the first stream. + +If you do not have a stream (because the test runner didn't output subunit or +you don't have access to the .testrepository) you may be able to use a list +file. If you can get a file that contains one test id per line, you can run +the named tests like this: + +```sh + $ testr run --load-list FILENAME +``` + +This can also be useful when dealing with sporadically failing tests, or tests +that only fail in combination with some other test - you can bisect the tests +that were run to get smaller and smaller (or larger and larger) test subsets +until the error is pinpointed. + +`testr run --until-failure`` will run your test suite again and again and +again stopping only when interrupted or a failure occurs. This is useful +for repeating timing-related test failures. \ No newline at end of file diff --git a/doc/src/setuptools-integration.md b/doc/src/setuptools-integration.md new file mode 100644 index 0000000..3727c1a --- /dev/null +++ b/doc/src/setuptools-integration.md @@ -0,0 +1,16 @@ +# Setuptools integration + +testrepository provides a setuptools commands for ease of integration with +setuptools-based workflows: + +* testr: + `python setup.py testr` will run testr in parallel mode + Options that would normally be passed to testr run can be added to the + testr-options argument. + `python setup.py testr --testr-options="--failing"` will append `--failing` + to the test run. +* testr --coverage: + `python setup.py testr --coverage` will run testr in code coverage mode. This + assumes the installation of the python coverage module. +* `python testr --coverage --omit=ModuleThatSucks.py` will append + --omit=ModuleThatSucks.py to the coverage report command. diff --git a/doc/example-failing-subunit-stream b/examples/example-failing-subunit-stream similarity index 100% rename from doc/example-failing-subunit-stream rename to examples/example-failing-subunit-stream diff --git a/doc/example-passing-subunit-stream b/examples/example-passing-subunit-stream similarity index 100% rename from doc/example-passing-subunit-stream rename to examples/example-passing-subunit-stream