Writing a unit test

This how-to will go through the process of writing a unit test, using a real example from pyani: the get_version() function provided in each subcommand's API. Note: for each subcommand, the specific checks have had to be modified slightly; this is an unavoidable side effect of wrapping third-party software.

Here is an example of the get_version() function from fastani.py:

def get_version(fastani_exe: Path = pyani_config.FASTANI_DEFAULT) -> str:
    """Return FastANI package version as a string.

    :param fastani_exe: path to FastANI executable

    We expect fastANI to return a string on STDOUT as

    .. code-block:: bash

        $ ./fastANI -v
        version 1.32

    we concatenate this with the OS name.

    The following circumstances are explicitly reported as strings:

    - no executable at passed path
    - non-executable file at passed path (this includes cases where the user doesn't have execute permissions on the file)
    - no version info returned
    """
    try:
        fastani_path = Path(shutil.which(fastani_exe))  # type:ignore
    except TypeError:
        return f"{fastani_exe} is not found in $PATH"

    if fastani_path is None:
        return f"{fastani_exe} is not found in $PATH"

    if not fastani_path.is_file():  # no executable
        return f"No fastANI executable at {fastani_path}"

    # This should catch cases when the file can't be executed by the user
    if not os.access(fastani_path, os.X_OK):  # file exists but not executable
        return f"fastANI exists at {fastani_path} but not executable"

    cmdline = [fastani_exe, "-v"]  # type: List
    result = subprocess.run(
        cmdline,
        shell=False,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        check=True,
    )  # type CompletedProcess

    match = re.search(
        r"(?<=version\s)[0-9\.]*", str(result.stderr + result.stdout, "utf-8")
    )
    version = match.group()  # type: ignore

    if 0 == len(version.strip()):
        return f"fastANI exists at {fastani_path} but could not retrieve version"

    return f"{platform.system()}_{version} ({fastani_path})"

There are several unit tests in the pyani test suite that have been written to address different parts of this one function. (They can be found in pyani/tests/test_fastani.py below the comment # Test get_version().)

Comments in the test file tell the user what situations the different test cases are designed to cover:

• no executable location is specified
• there is no executable
• there is a file, but it is not executable
• there is an executable file, but the version can't be retrieved

A straightforward test

The first test case tests this part of the function above:

    try:
        fastani_path = Path(shutil.which(fastani_exe))  # type:ignore
    except TypeError:
        return f"{fastani_exe} is not found in $PATH"

The test itself is shown below:

# Test case 0: no executable location is specified
def test_get_version_nonetype():
    """Test behaviour when no location for the executable is given."""
    test_file_0 = None

    assert (
        fastani.get_version(test_file_0) == f"{test_file_0} is not found in $PATH"
    )

This test is fairly straightforward. At the start, a value of None is assigned to the variable test_file_0. There is then an assertion that providing this variable to fastani.get_version() will produce the string None is not found in $PATH—the evaluated f-string.

Fixtures and monkeypatching

The second case, intended to test the lines:

    if not fastani_path.is_file():  # no executable
        return f"No fastANI executable at {fastani_path}"

becomes more complex, and requires the introduction of two new concepts: pytest fixtures and monkeypatching.

# Test case 1: there is no executable
def test_get_version_no_exe(executable_missing, monkeypatch):
    """Test behaviour when there is no file at the specified executable location."""
    test_file_1 = Path("/non/existent/fastani")
    assert (
        fastani.get_version(test_file_1) == f"No fastANI executable at {test_file_1}"
    )

Fixtures

Fixtures are functions whose main purpose is to return an object used as a fake input in code to be tested. They have the general form of:

@pytest.fixture
def fixture_name():
    •••
    return fixture

executable_missing is an example of a fixture in use. It is passed to the test as an argument, without the parentheses associated with function calls.

The actual definition of executable_missing is more complex than that shown above. The reason for this is that the fake input it gives to the code being tested needs to "pass" some hurdles (the code tested in the previous test) before reaching the lines it is intended to test. In order to bypass these, some aspects of the fake input must look real. This is achieved by 'monkeypatching' the code.

Monkeypatching

monkeypatch is, itself, a fixture provided with pytest. It provides several methods that can be used to specify behaviour of parts of your code during testing—for instance, monkeypatch.setattr(), which allows the user to provide an alternate (mock) function with a specified return value, that will be run instead of one called in the original code.

Found in conftest.py, executable_missing is defined as:

@pytest.fixture
def executable_missing(monkeypatch):
    """Mocks an executable path that does not point to a file."""

    def mock_which(*args, **kwargs):
        """Mock a call to `shutil.which()`, which produces an absolute file path."""
        return args[0]

    def mock_isfile(*args, **kwargs):
        """Mock a call to `Path.is_file()`."""
        return False

    monkeypatch.setattr(shutil, "which", mock_which)  # Path(test_file_1))
    monkeypatch.setattr(Path, "is_file", mock_isfile)

Note that it is passed monkeypatch as an argument; this is also passed as an argument to the test itself.

There are two mock versions of functions defined inside the fixture: mock_which() and mock_isfile(). These are written to accept any number of arguments and keyword arguments (*args, **kwargs), allowing them to 'swallow' anything they are passed by the actual code. They then return a set value.

In the case of mock_which(), this is the argument to fastani.get_version(test_file_1)), accessed by index because it hasn't been assigned a name in the mock function. shutil.which() checks the executable $PATH and returns the absolute location of its argument, assuming its argument is: a valid file, is executable, and is on the executable $PATH. Returning the first argument, which is known to be a Path object because it is defined as such in the test, provides the correct type of object for the variable fastani_path.

For mock_isfile(), the return value is just False because Path.is_file() tests whether a file exists, and the purpose of the test is to make sure it correctly flags input that is not the name of a file. (Mocking the function eliminates the possibility of accidentally passing a string that is a valid file name.)

Once the mock functions have been defined, monkeypatch must be told to use these, instead of the actual functions used in the code being tested. This is done with the monkeypatch.setattr() method, which takes three arguments: the module where the original function is found; the name of the function, as a string; and the name of the mock function to replace it with.

Subsequent tests

In the case of this get_version() function, in which several things need to be tested in sequence, the fixtures used by subsequent tests sometimes involve only minor tweaks from previous ones. This is the case with the third test case, which tests the lines:

    # This should catch cases when the file can't be executed by the user
    if not os.access(fastani_path, os.X_OK):  # file exists but not executable
        return f"fastANI exists at {fastani_path} but not executable"

The test itself is:

# Test case 2: there is a file, but it is not executable
def test_get_version_exe_not_executable(executable_not_executable, monkeypatch):
    """Test behaviour when the file at the executable location is not executable."""
    test_file_2 = Path("/non/executable/fastani")
    assert (
        fastani.get_version(test_file_2)
        == f"fastANI exists at {test_file_2} but not executable"
    )

The fixture, executable_not_executable is defined in conftest.py as:

@pytest.fixture
def executable_not_executable(monkeypatch):
    """
    Mocks an executable path that does not point to an executable file,
    but does point to a file.
    """

    def mock_which(*args, **kwargs):
        """Mock an absolute file path."""
        return args[0]

    def mock_isfile(*args, **kwargs):
        """Mock a call to `os.path.isfile()`."""
        return True

    def mock_access(*args, **kwargs):
        """Mock a call to `os.access()`."""
        return False

    monkeypatch.setattr(shutil, "which", mock_which)
    monkeypatch.setattr(Path, "is_file", mock_isfile)
    monkeypatch.setattr(os, "access", mock_access)

There are two difference between this, and the aforementioned executable_missing fixture: here, mock_isfile() returns True; and there's a new function definition and monkeypatch.setattr() call for mock_access.

The different return value for mock_isfile() reflects that this test needs to bypass code tested previously.

os.access() checks whether the current user has the correct permissions to access the file that is its argument; this includes the ability to execute a file that can otherwise be accessed.

Mocking other objects

Sometimes it may be necessary to create mock objects that can be used as return values for mock functions inside a fixture used for monkeypatching. These have whatever attributes and methods are necessary to make them appear like valid objects for the context where they are needed—but are otherwise unlike the objects that would normally be passed.

This third technique can be seen in the test case for fastani.get_version(), which covers these lines:

    cmdline = [fastani_exe, "-v"]  # type: List
    result = subprocess.run(
        cmdline,
        shell=False,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        check=True,
    )  # type CompletedProcess

    match = re.search(
        r"(?<=version\s)[0-9\.]*", str(result.stderr + result.stdout, "utf-8")
    )
    version = match.group()  # type: ignore

    if 0 == len(version.strip()):
        return f"fastANI exists at {fastani_path} but could not retrieve version"

In order to test this, it is necessary to ensure the value of version.strip() is an empty string. The test is shown here:

# Test case 3: there is an executable file, but the version can't be retrieved
def test_get_version_exe_no_version(executable_without_version, monkeypatch):
    """Test behaviour when the version for the executable can not be retrieved."""
    test_file_3 = Path("/missing/version/fastani")
    assert (
        fastani.get_version(test_file_3)
        == f"fastANI exists at {test_file_3} but could not retrieve version"
    )

Mock objects

The fixture, executable_without_version builds on executable_not_executable:

@pytest.fixture
def executable_without_version(monkeypatch):
    """
    Mocks an executable file for which the version can't be obtained, but
    which runs without incident.
    """

    def mock_which(*args, **kwargs):
        """Mock an absolute file path."""
        return args[0]

    def mock_isfile(*args, **kwargs):
        """Mock a call to `os.path.isfile()`."""
        return True

    def mock_access(*args, **kwargs):
        """Mock a call to `os.access()`."""
        return True

    def mock_subprocess(*args, **kwargs):
        """Mock a call to `subprocess.run()`."""
        return MockProcess(b"mock bytes", b"mock bytes")

    def mock_search(*args, **kwargs):
        """Mock a call to `re.search()`."""
        return MockMatch()

    monkeypatch.setattr(shutil, "which", mock_which)
    monkeypatch.setattr(Path, "is_file", mock_isfile)
    monkeypatch.setattr(os.path, "isfile", mock_isfile)
    monkeypatch.setattr(os, "access", mock_access)
    monkeypatch.setattr(subprocess, "run", mock_subprocess)
    monkeypatch.setattr(re, "search", mock_search)

The return value for mock_access() is once again changed from False to True, for the same reason that of mock_isfile() changed in the previous test. There are also two more mock functions and related calls to monkeypatch.setattr(), for mock_subprocess() and mock_search(). In this case, mock_subprocess() is something needed to move past the call to subprocess.run(), which is not what is really being tested, to the call to re.search(), which is.

These mock functions return some custom classes that are defined earlier in conftest.py:

class MockProcess(NamedTuple):
    """Mock process object."""

    stdout: str
    stderr: str


class MockMatch(NamedTuple):
    """Mock match object."""

    def group(self):
        return ""

The first of these, MockProcess, is intended to emulate the output of the subprocess.run() method. The only parts of this method's return value that are needed are attributes named stdout and stderr, so the class is defined as a NamedTuple with just these attributes.

The return value from mock_subprocess() is then an instance of the MockProcess class, that has been given the byte string: b"mock bytes", twice. This is because these values are expected to be in the form of byte strings; using 'mock bytes' as the value is for the purpose of making the code self-commenting; the byte strings could also be empty, or a specific value could be specified, if needed for the test.

mock_search() returns an instance of MockMatch, which is assigned to match. This object has one method, group(), which returns an empty string.

Summary

With these three techniques, it is possible to test many different scenarios, but there also many cases that introduce complexity not addressed here. The other attributes of monkeypatch may be necessary in other cases; if the point of a test is to check for exceptions that are raised the unittest.TestCase class may be needed. There are also situations—such as testing code that uses a pool of workers to do things asynchronously via multiprocessing—which are very difficult to test (or even debug) because they run in different subprocesses, and thereore operate a bit like black boxes, making logging difficult.