WritingNewTests

Writing your own virt test

In this article, we'll talk about:

Where the test files are located
Example of a simple, real test located in the test tree
Write a simple test file
Add the appropriate configuration for your test
Try out your new test

Where the virt test files are located ?

The subtests can be located in 2 subdirs:

tests - These tests are written in a fairly virt technology agnostic way, so they can be used by other virt technologies testing. More specifically, they do not use the vm monitor.

[lmr@freedom virt-test.git]$ ls tests
autotest_control.py  fail.py              __init__.py        lvm.py                   nfs_corrupt.py       rv_connect.py       skip.py                     vlan.py
boot.py              file_transfer.py     iofuzz.py          mac_change.py            nicdriver_unload.py  rv_copyandpaste.py  softlockup.py               warning.py
boot_savevm.py       fillup_disk.py       ioquit.py          module_probe.py          nic_promisc.py       rv_disconnect.py    stress_boot.py              watchdog.py
build.py             fullscreen_setup.py  iozone_windows.py  multicast.py             ntttcp.py            rv_fullscreen.py    trans_hugepage_defrag.py    whql_client_install.py
clock_getres.py      guest_s4.py          jumbo.py           negative_create.py       ping.py              rv_input.py         trans_hugepage.py           whql_submission.py
dd_test.py           guest_test.py        kdump.py           netperf.py               pxe.py               save_restore.py     trans_hugepage_swapping.py  yum_update.py
ethtool.py           image_copy.py        linux_s3.py        netstress_kill_guest.py  remove_guest.py      shutdown.py         unattended_install.py

kvm/tests - These are tests that do use specific kvm infrastructure, specifically the qemu monitor. other virt technologies can't use it so they go here.

[lmr@freedom virt-test.git]$ ls kvm/tests
9p.py             __init__.py                                     migration_with_speed_measurement.py  qemu_io_blkdebug.py       timedrift_with_migration.py
balloon_check.py  kernel_install.py                               multi_disk.py                        qemu_iotests.py           timedrift_with_reboot.py
block_mirror.py   ksm_overcommit.py                               nic_bonding.py                       qmp_basic.py              timedrift_with_stop.py
block_stream.py   migration_multi_host_cancel.py                  nic_hotplug.py                       qmp_basic_rhel6.py        time_manage.py
cdrom.py          migration_multi_host_downtime_and_speed.py      nmi_watchdog.py                      seabios.py                unittest_kvmctl.py
cgroup.py         migration_multi_host_ping_pong.py               pci_hotplug.py                       set_link.py               unittest.py
cpuflags.py       migration_multi_host.py                         perf_kvm.py                          smbios_table.py           usb.py
cpu_hotplug.py    migration_multi_host_with_file_transfer.py      performance.py                       stepmaker.py              virtio_console.py
enospc.py         migration_multi_host_with_speed_measurement.py  physical_resources_check.py          steps.py                  vmstop.py
floppy.py         migration.py                                    qemu_guest_agent.py                  stop_continue.py
getfd.py          migration_with_file_transfer.py                 qemu_guest_agent_snapshot.py         system_reset_bootable.py
hdparm.py         migration_with_reboot.py                        qemu_img.py                          timedrift.py

In the source tree, you can see a simple test that touches some interesting functionality, tests/clock_getres.py:

import logging, os
from autotest.client.shared import error


def run_clock_getres(test, params, env):
    """
    Verify if guests using kvm-clock as the time source have a sane clock
    resolution.

    @param test: kvm test object.
    @param params: Dictionary with test parameters.
    @param env: Dictionary with the test environment.
    """
    source_name = "test_clock_getres/test_clock_getres.c"
    source_name = os.path.join(test.virtdir, "deps", source_name)
    dest_name = "/tmp/test_clock_getres.c"
    bin_name = "/tmp/test_clock_getres"

    if not os.path.isfile(source_name):
        raise error.TestError("Could not find %s" % source_name)

    vm = env.get_vm(params["main_vm"])
    vm.verify_alive()
    timeout = int(params.get("login_timeout", 360))
    session = vm.wait_for_login(timeout=timeout)

    vm.copy_files_to(source_name, dest_name)
    session.cmd("gcc -lrt -o %s %s" % (bin_name, dest_name))
    session.cmd(bin_name)
    logging.info("PASS: Guest reported appropriate clock resolution")
    logging.info("Guest's dmesg:\n%s", session.cmd_output("dmesg").strip())

This is a simple test that:

Picks a vm from the main env file, if there's one
Verify whether it its process is alive
Logs into it
Copies the source code of the test resolution test to the vm
Compiles and executes it

#. Assuming it ended successfully, it will return 0, then session.cmd won't # raise an exception, then the test passed. Otherwise, the test will fail.

If you check the base tests cfg file, you can identify the test variant for the boot test. We'll explain more about the meaning of the stuff on this variant in a minute. From the file:

shared/cfg/subtests.cfg.sample

You can see the following snippet:

- clock_getres: install setup image_copy unattended_install.cdrom
    virt_test_type = kvm libvirt
    no JeOS
    only Linux
    type = clock_getres

So, the rules for a new test in kvm autotest are:

Choosing a name. On the case above, it is clock_getres.
The test code has to be located in one of the 2 locations specified above, and named after, well, the test name. Example: clock_getres.py
The file clock_getres.py has to implement at least one python function named run_[test_name], which takes exactly 3 args, the same for every single test. Example: run_clock_getres(test, params, env)
- test: virt test object
- param: Dict with test configuration
- env: The env file that stores vms and installers, used by the kvm test to pick and store objects.
A variant in tests_base.cfg.sample, with a variant with your test name. This variant has to be located in the tests session of tests_base.cfg. Just look for the boot variant and you'll find out where it is. Note that the variant name/label does not need to follow the name of your test, it can be something different, such as ponies. Although most of the time you want them both to be the same to simplify things. You normally break this assumption if some namespace clash is going to happen (Example: the cdrom test had a namespace clash with unattended_install.cdrom, so we named the variant cdrom_test). The only important thing is that type must have the name of your test. The variable type is effectively what allows the kvm test to locate your test code and run it, the name of the variant itself is just an arbitrary label.
virt_test_type indicates the types of virt tests that can run it. If your test uses the qemu monitor facility, it's a kvm specific test, so you would indicate only 'kvm'. 'no JeOS' means that, since the test does require a working toolchain in the guest, it won't work on the JeOS so it is left out of JeOS test sets. 'only Linux' means the test is specific to Linux guest OS. There are tests that work for both Windows and Linux, therefore this restriction doesn't apply to them.

Write our own 'uptime' test - Step by Step procedure

Now, let's go and write our uptime test, which only purpose in life is to pick up a living guest, connect to it via ssh, and return its uptime.

Git clone virt_test.git to a convenient location, say $HOME/Code/virt-test. See the download source documentation. Please do use git and clone the repo to the location mentioned.
Our uptime test won't need any kvm specific feature. Thinking about it, we only need a vm object and stablish an ssh session to it, so we can run the command. So we can store our brand new test under tests. At the autotest root location:
```
[lmr@freedom virt-test.git]$ touch tests/uptime.py
[lmr@freedom virt-test.git]$ git add tests/uptime.py
```
Ok, so that's a start. So, we have at least to implement a function run_uptime. Let's start with it and just put the keyword pass, which is a no op. Our test will be like:
```
def run_uptime(test, params, env):
    """
    Docstring describing uptime.
    """
    pass
```
Now, what is the API we need to grab a VM from our test environment? Our env object has a method, get_vm, that will pick up a given vm name stored in our environment. Some of them have aliases. main_vm contains the name of the main vm present in the environment, which is, most of the time, vm1. env.get_vm returns a vm object, which we'll store on the variable vm. It'll be like this:
```
def run_uptime(test, params, env):
    """
    Docstring describing uptime.
    """
    vm = env.get_vm(params["main_vm"])
```
A vm object has lots of interesting methods, which we plan on documenting them more thoroughly, but for now, we want to ensure that this VM is alive and functional, at least from a qemu process standpoint. So, we'll call the method verify_alive(), which will verify whether the qemu process is functional and if the monitors, if any exist, are functional. If any of these conditions are not satisfied due to any problem, an exception will be thrown and the test will fail. This requirement is because sometimes due to a bug the vm process might be dead on the water, or the monitors are not responding.
```
def run_uptime(test, params, env):
    """
    Docstring describing uptime.
    """
    vm = env.get_vm(params["main_vm"])
    vm.verify_alive()
```
Next step, we want to log into the vm. the vm method that does return a remote session object is called wait_for_login(), and as one of the parameters, it allows you to adjust the timeout, that is, the time we want to wait to see if we can grab an ssh prompt. We have top level variable login_timeout, and it is a good practice to retrieve it and pass its value to wait_for_login(), so if for some reason we're running on a slower host, the increase in one variable will affect all tests. Note that it is completely OK to just override this value, or pass nothing to wait_for_login(), since this method does have a default timeout value. Back to business, picking up login timeout from our dict of parameters:
```
def run_uptime(test, params, env):
    """
    Docstring describing uptime.
    """
    vm = env.get_vm(params["main_vm"])
    vm.verify_alive()
    timeout = float(params.get("login_timeout", 240))
```

Now we'll call wait_for_login() and pass the timeout to it, storing the resulting session object on a variable named session.

def run_uptime(test, params, env):
    """
    Docstring describing uptime.
    """
    vm = env.get_vm(params["main_vm"])
    vm.verify_alive()
    timeout = float(params.get("login_timeout", 240))
    session = vm.wait_for_login(timeout=timeout)

The kvm test will do its best to grab this session, if it can't due to a timeout or other reason it'll throw a failure, failing the test. Assuming that things went well, now you have a session object, that allows you to type in commands on your guest and retrieve the outputs. So most of the time, we can get the output of these commands throught the method cmd(). It will type in the command, grab the stdin and stdout, return them so you can store it in a variable, and if the exit code of the command is != 0, it'll throw a aexpect.ShellError?. So getting the output of the unix command uptime is as simple as calling cmd() with 'uptime' as a parameter and storing the result in a variable called uptime:
```
def run_uptime(test, params, env):
    """
    Docstring describing uptime.
    """
    vm = env.get_vm(params["main_vm"])
    vm.verify_alive()
    timeout = float(params.get("login_timeout", 240))
    session = vm.wait_for_login(timeout=timeout)
    uptime = session.cmd('uptime')
```
If you want to just print this value so it can be seen on the test logs, just log the value of uptime using the logging library. Since that is all we want to do, we may close the remote connection, to avoid ssh/rss sessions lying around your test machine, with the method close(). Now, note that all failures that might happen here are implicitly handled by the methods called. If a test went from its beginning to its end without unhandled exceptions, autotest assumes the test automatically as PASSed, no need to mark a test as explicitly passed. If you have explicit points of failure, for more complex tests, you might want to add some exception raising.
```
def run_uptime(test, params, env):
    """
    Docstring describing uptime.
    """
    vm = env.get_vm(params["main_vm"])
    vm.verify_alive()
    timeout = float(params.get("login_timeout", 240))
    session = vm.wait_for_login(timeout=timeout)
    uptime = session.cmd("uptime")
    logging.info("Guest uptime result is: %s", uptime)
    session.close()
```
Now, I deliberately introduced a bug on this code just to show you guys how to use some tools to find and remove trivial bugs on your code. I strongly encourage you guys to check your code with the script called run_pylint.py, located at the utils directory at the top of your $AUTOTEST_ROOT. This tool calls internally the other python tool called pylint to catch bugs on autotest code. I use it so much the utils dir of my devel autotest tree is on my $PATH. So, to check our new uptime code, we can call (important, if you don't have pylint install it with yum install pylint or equivalent for your distro):
```
[lmr@freedom virt-test.git]$ tools/run_pylint.py tests/uptime.py -q
************* Module virt-test.git.tests.uptime
E0602: 10,4:run_uptime: Undefined variable 'logging'
```

Ouch. So there's this undefined variable called logging on line 10 of the code. It's because I forgot to import the logging library, which is a python library to handle info, debug, warning messages. Let's Fix it and the code becomes:

import logging

def run_uptime(test, params, env):
    """
    Docstring describing uptime.
    """
    vm = env.get_vm(params["main_vm"])
    vm.verify_alive()
    timeout = float(params.get("login_timeout", 240))
    session = vm.wait_for_login(timeout=timeout)
    uptime = session.cmd("uptime")
    logging.info("Guest uptime result is: %s", uptime)
    session.close()

Let's re-run run_pylint.py to see if it's happy with the code generated:

[lmr@freedom virt-test.git]$ tools/run_pylint.py tests/uptime.py -q
[lmr@freedom virt-test.git]$

So we're good. Nice! Now, as good indentation does matter to python, we have another small utility called reindent.py, that will fix indentation problems, and cut trailing whitespaces on your code. Very nice for tidying up your test before submission.
```
[lmr@freedom virt-test.git]$ tools/reindent.py tests/uptime.py
```
I also use run_pylint with no -q catch small things such as wrong spacing around operators and other subtle issues that go against PEP 8 and the coding style document. Please take pylint's output with a handful of salt, you don't need to work each and every issue that pylint finds, I use it to find unused imports and other minor things.
```
[lmr@freedom virt-test.git]$ tools/run_pylint.py tests/uptime.py
************* Module virt-test.git.tests.uptime
C0111:  1,0: Missing docstring
C0103:  7,4:run_uptime: Invalid name "vm" (should match [a-z_][a-z0-9_]{2,30}$)
W0613:  3,15:run_uptime: Unused argument 'test'
```
These other complaints you don't really need to fix. Due to the tests design, they all use 3 arguments, 'vm' is a shorthand that we have been using for a long time as a variable name to hold a VM object, and the only docstring we'd like you to fill is the one in the run_uptime function.
Now, we need a variant on shared/cfg/subtests.cfg to inform kvm autotest where the fancy new test is located. In actuality, to contribute it upstream, the variant needs to be written to subtests.cfg.sample. Just remember to mirror the file to subtests.cfg to make sure your instance of kvm autotest will be able to run it. Editing subtests.cfg.sample and adding the new variant (I chose to put it after the trans_hugepage variant, which is immediately before the destructive tests):
```
- uptime: install setup image_copy unattended_install.cdrom
    virt_test_type = kvm libvirt
    only Linux
    type = uptime
```

All this means:

The uptime variant depends on having a functional, installed guest, which makes it dependent of either one of install setup image_copy unattended_install.cdrom if they ran previously. We'll use JeOS so don't worry about it.
The new test runs OK in both kvm and libvirt tests.
As we're going to use 'uptime' explicitly, this only works for Linux guests.
The test type is uptime.
This will yield a functional git commit so you can send it to the mailing list.

Important: The test config file is sensitive to indentation. So please make sure the new block is aligned with the other test variants, and please use only spaces, to respect the virt test conventions. If you miss the indentation, your new test won't be listed in the test list.

Now, to run it, you need to copy the updated sample file to the actual config file. We use this habit of samples/actual config files copied to avoid making users to keep maintaining the config files on other branches and such.

[lmr@freedom virt-test.git]$ cp shared/cfg/subtests.cfg.sample kvm/cfg/subtests.cfg

Now, you can test your code. When listing the kvm tests your new test should appear in the list:
```
./run -t kvm --list-tests
```

Now, you can run your test to see if everything went good.

[lmr@freedom virt-test.git]$ ./run -t kvm --tests uptime
SETUP: PASS (1.10 s)
DATA DIR: /home/lmr/virt_test
DEBUG LOG: /home/lmr/Code/virt-test.git/logs/run-2012-11-28-13.13.29/debug.log
TESTS: 1
kvm.virtio_blk.smp2.virtio_net.JeOS.17.64.uptime: PASS (23.30 s)

Ok, so now, we have something that can be git commited and sent to the mailing list

diff --git a/shared/cfg/subtests.cfg.sample b/shared/cfg/subtests.cfg.sample
index 9ffb4c9..ba8febd 100644
--- a/shared/cfg/subtests.cfg.sample
+++ b/shared/cfg/subtests.cfg.sample
@@ -2900,6 +2900,11 @@ variants:
         only Linux
         type = clock_getres

+    - uptime: install setup image_copy unattended_install.cdrom
+        virt_test_type = kvm libvirt
+        only Linux
+        type = uptime
+
     - yum_update: install setup image_copy unattended_install.cdrom
         virt_test_type = kvm libvirt
         only Fedora, RHEL
diff --git a/tests/uptime.py b/tests/uptime.py
index e69de29..65d46fa 100644
--- a/tests/uptime.py
+++ b/tests/uptime.py
@@ -0,0 +1,13 @@
+import logging
+
+def run_uptime(test, params, env):
+    """
+    Docstring describing uptime.
+    """
+    vm = env.get_vm(params["main_vm"])
+    vm.verify_alive()
+    timeout = float(params.get("login_timeout", 240))
+    session = vm.wait_for_login(timeout=timeout)
+    uptime = session.cmd("uptime")
+    logging.info("Guest uptime result is: %s", uptime)
+    session.close()

Oh, we forgot to add a decent docstring description. So doing it:

import logging

def run_uptime(test, params, env):
    """
    Uptime test for virt guests:

    1) Boot up a VM.
    2) Stablish a remote connection to it.
    3) Run the 'uptime' command and log its results.

    @param test: KVM test object.
    @param params: Dictionary with the test parameters.
    @param env: Dictionary with test environment.
    """
    vm = env.get_vm(params["main_vm"])
    vm.verify_alive()
    timeout = float(params.get("login_timeout", 240))
    session = vm.wait_for_login(timeout=timeout)
    uptime = session.cmd("uptime")
    logging.info("Guest uptime result is: %s", uptime)
    session.close()

git commit signing it, put a proper description, then send it with git send-email. Profit!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WritingNewTests

Writing your own virt test

Where the virt test files are located ?

Write our own 'uptime' test - Step by Step procedure

Clone this wiki locally