Skip to content

Running Acceptance Tests at PR

Jump to bottom
Jordi Massaguer Pla edited this page Apr 3, 2023 · 25 revisions

This documentation will make sense once this https://github.com/uyuni-project/uyuni/pull/6755 is merged.

🔨 This is a beta preview of running Acceptance Tests at PR. Expect this to be imperfect :)

What acceptance tests at PR are

"Acceptance tests" are tests that are being run on master branch. They are located under testsuite directory. Based on the result of these tests, new versions of uyuni are released.

These tests usually take a lot of time and resources. Between two subsequent runs there can be multiple code merges from different developers, which makes the probability of regressions higher, and more complicated to figure out what line of code broke the tests. This means that when tests fail, the team needs to dedicate some time and energy to figure out the cause and whether some code changes are needed. This usually happens under time constraints, which can increase the stress level of the team.

We aim to lower the chances code merges introduce bugs or regressions, by allowing developers to run the acceptance tests on their Pull Request prior to merging.

However, running at the Pull Request is not the same as running at the master branch. At the master branch, we need to reproduce what a user would do and use the infrastructure a user would. At the Pull Request, we overall need reproducibility and stability: two subsequent runs of the testsuite on the same code should produce the same result. Also, we need them to be usable, which means they should take much less time to produce the results.

Thus, at the Pull Request we are taking some "shortcuts"/"fakes":

  1. We do not package the code as RPMs, but build the code and copy it into the target directory
  2. We do not use VMs, but containers
  3. We do not test SCC integration, nor we have SCC channels configured
  4. We do not test PXE
  5. We do not test reboots

Still, running "Acceptance tests" at PR is useful to filter out code that would break the product before it lands into master.

Enabling it

By default only "beta users" will have acceptance tests run on their Pull Request. If you want to be a beta user, just create a Pull Request that adds your name at the file .github/workflows/test_all_in_one.yml .

Something like

if: github.actor == 'jordimassaguerpla' || github.actor == 'YOUR GITHUB NAME'

Running locally

You can run the same tests locally with your code. Just go do testsuite/podman_runner and run ./run . This will run the same set of commands that are run inside the github action. Then, you can use podman -ti exec CONTAINER bash , podman ps, podman logs CONTAINER, etc. to debug it. You can as well connect to the server using a web browser. Just run podman ps and note which is the port you have to connect to.

Note run will call 00_setup_env.sh . This script is only used when running locally to do a clean up, i.e. to kill the containers. This script has a sleep 10 at the end to wait for all to settle. This usually is enough but if you get some errors that the containers already exist, try increasing that value.

Skipped tests

Tests that are skipped have the label @skip_if_container . Some of them are skipped because do not make sense or are too difficult to test: cobbler, reboot vms, test xen or kvm ... others are skipped because they fail with the current infrastructure, and we need a stable testsuite to start with: i.e. bootstrapping using the web interface.

The plan is work on those that make sense but fail, so that eventually can be enabled, and so the coverage increased.

Clone this wiki locally