Skip to content

Commit

Permalink
Cleanup testing reference manual section
Browse files Browse the repository at this point in the history
Applied style guide recommendations and edited to increase readability.

QA Steps: linter and spellcheck used in editor. Checked rendered output
for correctness. No issues found.

This commit addresses FFTK-2796
This commit applies to FFTK-1702

Signed-off-by: Katrina Prosise <katrina.prosise@foundries.io>
  • Loading branch information
kprosise committed Feb 4, 2024
1 parent 057a487 commit 9729a52
Show file tree
Hide file tree
Showing 4 changed files with 99 additions and 120 deletions.
23 changes: 11 additions & 12 deletions source/reference-manual/testing/fiotest.rst
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
.. _ref-fiotest:

fiotest
Fiotest
=======

The Factory includes a simple, but effective way to manage automated testing.
It's managed by a feature called the Device Gateway Testing API. This API
allows registered devices to report the results of tests. The Factory makes
the raw results of these tests available through api.foundries.io. Fioctl®
also includes the ability to summarize test results.
Your Factory includes a simple but effective way to manage automated testing.
It is managed by the Device Gateway Testing API feature.
This API allows registered devices to report the results of tests.
Your Factory makes the raw results of these tests available through `<api.foundries.io>`_.
The Fioctl® Factory management tool also includes the ability to summarize test results.

The Model
---------
Expand Down Expand Up @@ -36,8 +36,8 @@ The API is built around a generic model for recording test results::
The API
-------

POST /tests/
~~~~~~~~~~~~
``POST /tests/``
~~~~~~~~~~~~~~~~

Create a new test (``status=RUNNING``)::

Expand All @@ -53,8 +53,8 @@ POST /tests/
HTTP 201
LOCATION: /tests/12345

PUT /tests/<test id>
~~~~~~~~~~~~~~~~~~~~
``PUT /tests/<test id>``
~~~~~~~~~~~~~~~~~~~~~~~~

Complete a test with data::

Expand All @@ -79,8 +79,7 @@ PUT /tests/<test id>
Creating Custom Tests
---------------------

The fiotest_ project provides a good starting point and example of how to
do device testing.
The fiotest_ project provides a good starting point and an example of how to do device testing.

.. _fiotest:
https://github.com/foundriesio/fiotest
155 changes: 66 additions & 89 deletions source/reference-manual/testing/lmp-testplan.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,37 +4,34 @@
Test Plan
#########

What to Test
Below you will find the test plan Foundries.io uses for FoundriesFactory®.

What To Test
============

In the context of this test plan ``mandatory`` means
the testing must be performed to release the software.
Moreover all tests must pass before the release is announced.
``Optional`` means that the tests may be performed if the time allows.
If ``optional`` test results are missing release can be announced.
In the context of this test plan, **mandatory** means the testing that must be performed to release the software.
Moreover, all tests must pass before the release is announced.
**Optional** means that the tests may be performed if time allows.
If optional test results are missing, release can be announced.

Testing must focus on the mandatory features of the FoundriesFactory image:

* Linux Microplatform
* Linux microPlatform
* Base OS running with OSTree based root filesystem
* Aktualizr-lite daemon
* Docker engine running docker compose apps
* Docker engine running Docker compose apps
* Fioconfig

Remaining elements of the system may be tested as optional
but are not as important.
For example, testing of various I/O interfaces can be done case-by-case
depending on the customer requirements and hardware capabilities

A complete test list for all devices can be found in the `qa-tools git
repository`_.
Remaining elements of the system may be tested as optional but are not as important.
For example, testing of various I/O interfaces can be done case-by-case depending on the customer requirements and hardware capabilities

A complete test list for all devices can be found in the `qa-tools git repository`_.
Most of the test are stored in the `test-definitions repository`_.

LmP test plan
LmP Test Plan
-------------

Linux microplatform testing must at least cover:
Linux® microPlatform testing must at least cover:

* Boot testing

Expand All @@ -55,13 +52,13 @@ Linux microplatform testing must at least cover:
* Ensure that fioconfig daemon runs
* Ensure that NetworkManager runs

* Ensure that docker engine runs
* Ensure that example docker app runs
* Ensure that Docker engine runs
* Ensure that example Docker app runs

OS features
OS Features
-----------

Boot test and smoke test
Boot Test and Smoke Test
~~~~~~~~~~~~~~~~~~~~~~~~

* Mandatory:
Expand All @@ -70,7 +67,7 @@ Boot test and smoke test
* Check block devices
* Check network devices
* Ping test
* Optee test - xtest (binaries built into the OS image)
* Optee test xtest (binaries built into the OS image)
* Timesyncd

* Ensure NTP is synchronized correctly
Expand All @@ -86,9 +83,9 @@ OSTree

* Optional

OSTree has a runtime test suite. It’s used for validating after
building libostree but it probably can also be used to test the
live system. This might be destructive to the FS though.
OSTree has a runtime test suite.
It is used for validation after building libostree, but can potentially be used to test a live system.
However, this may be destructive to the filesystem.

Docker
~~~~~~
Expand All @@ -107,7 +104,7 @@ Docker
* User-defined bridge
* Host

Networking on host
Networking on Host
~~~~~~~~~~~~~~~~~~

* Mandatory
Expand All @@ -126,15 +123,15 @@ Networking on host
* Check if name resolution works with default settings
(default DNS must be validated before running tests)

Interface testing (optional)
Interface Testing (Optional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Various interfaces are tested depending on the hardware and customer
requirements. Current plan is to execute the following tests:
Various interfaces are tested depending on the hardware and customer requirements.
Current plan is to execute the following tests:

* HDMI (HDMI capture device)

Device update
Device Update
-------------

Aktualizr (OTA API)
Expand All @@ -144,37 +141,36 @@ Aktualizr (OTA API)

* Update

* Update of docker compose apps (new target)
* Update of Docker compose apps (new Target)

* From previous target
* *From* previous Target

* Update of base OS

* From previous platform’ target
* From previous releaseplatform’ target
* From previous platform Target
* From previous *release* platform Target

* Rollback

* Rollback of base OS

Device config (fioconfig)
Device Config (Fioconfig)
~~~~~~~~~~~~~~~~~~~~~~~~~

* Mandatory
* Mandatory, to test whether:

* Test whether factory specific configs are applied properly
* Test whether group specific configs are applied properly
* Test whether device specific configs are applied properly
* Test whether both encrypted and non-encrypted configs are
available on the device
* Factory specific configs are applied properly
* Group specific configs are applied properly
* Device specific configs are applied properly
* Both encrypted and non-encrypted configs are available on the device

How to test
How To Test
===========

LmP tests
LmP Tests
---------

Boot testing
Boot Testing
~~~~~~~~~~~~

There are several kinds of tests involved.
Expand All @@ -187,80 +183,61 @@ There are 2 scenarios for boot testing:
This happens when the software is delivered to the board for the first time.
Since the aktualizr is not yet running on the board,
provisioning has to be done in some other way.
It strongly depends on the hardware limitations and boot source.
For example RaspberryPi can boot from SD card
and it works well with available SDMux devices.
On the other hand iMX8MM should boot from eMMC
and requires UUU for initial flashing.
This is strongly dependent on hardware limitations and boot source.
For example, RaspberryPi can boot from an SD card, and works well with available SDMux devices.
Conversely, iMX8MM should boot from eMMC, and requires UUU for initial flashing.
Both of these provisioning methods are supported by LAVA.
Therefore it is proposed to use LAVA for initial provisioning,
boot and reboot testing in this scenario.
Therefore, it is proposed to use LAVA for initial provisioning, boot, and reboot testing in this scenario.

* Software update (OS update)

Booting after software update can be checked in 2 ways:
with aktualizr-lite or container running on the board
or with an external tool.
Booting after a software update can be checked in 2 ways:
with either aktualizr-lite or a container running on the board, or with an external tool.

When checking reboot after update testing rig needs to know:

* When the test starts (on old target)
* What are the starting (old)
and ending (new) targets
and OSTree hashes
* When the test is finished
(aktualizr performs update,
system is rebooted)
* When the test starts (on old Target)
* What are the starting (old) and ending (new) Targets and OSTree hashes
* When the test is finished (aktualizr performs update, system is rebooted)

Basic tests
Basic Tests
~~~~~~~~~~~

Basic tests are executed on the target either
using the fiotest container (running commands on host)
or LAVA.
Basic tests are executed on the target either using the fiotest container (running commands on host) or LAVA.
Which tool depends on the tested scenario.
We’re currently testing 2 scenarios:
We are currently testing 2 scenarios:

* *Manufacturing* scenario

LAVA can execute tests in Linux shell on the target
and parse results from the serial console.
LAVA can execute tests in Linux shell on the Target and parse results from the serial console.
Tests are executed after flashing an image to the board.
DUT always starts fresh without any previously running software.

* *Rolling update* scenario

``Test-runner.py`` is a script from test-definitions repository.
It’s able to run tests on the remote OS
using SSH as a connection medium.
It is able to run tests on the remote OS using SSH as a connection medium.
It is used to execute tests in the ‘rolling update’ scenario.
Test results are reported to both
qa-reports
and FIO backend.
Test results are reported to both qa-reports and FIO backend.
Reporting to FIO backend is done with fiotest.
Fiotest is also responsible for
starting a test round following an OTA update.
Test plan executed in the “Rolling update” scenario is limited.
Tests disabling networking
and potentially corrupting the OS
are disabled.
Fiotest is also responsible for starting a test round following an OTA update.
Test plan executed in the “rolling update” scenario is limited.
Tests disabling networking and potentially corrupting the OS are disabled.

* Docker apps update

Testing of Docker apps update should be done
using a container registered for aktualizr-lite callbacks.
This way we’re as close as possible to testing production setup.
Testing of Docker apps update should be done using a container registered for aktualizr-lite callbacks.
This way we are as close as possible to testing a production setup.

When to Test
When To Test
============

A testing round is started after every merge to ``lmp-manifest``.
If the build is successful
all testing factories pull latest source from ``lmp-manifest`` project
and merge to their working branches.
Successful build in the testing factory triggers tests on the devices.
If the build is successful, all testing Factories pull the latest source from ``lmp-manifest`` and merge to their working branches.
A successful build in the testing Factory triggers tests on the devices.
OTA update is delivered to the *rolling update* devices.
This also triggers a testing round on the new target.
For a release candidate build additional manual tests are performed.
This also triggers a testing round on the new Target.
For a release candidate buil,d additional manual tests are performed.

.. _qa-tools git repository:
https://github.com/foundriesio/qa-tools
Expand Down
34 changes: 15 additions & 19 deletions source/reference-manual/testing/testing-architecture.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,21 @@
Architecture Overview
=====================

The Factory includes a generic :ref:`API and data model <ref-fiotest>`
to allow customers the freedom to test how they choose. In most cases,
you'll be able to use fiotest as-is for your testing needs.

All things in the Factory revolve around Targets and the testing
mechanism is no different. Devices run Targets. The results of any
tests they run are correlated with its Target. This allows you to get
a test reporting of Targets in your Factory.

The fiotest container can be set up in your Factory as a
:ref:`Docker Compose application <ref-compose-apps>`. The fiotest
container has the ability to interact with the host OS and configure
aktualizr-lite to notify it when a new Target has been installed.
fiotest is then able to run tests and report the results back through
the :ref:`device gateway <ref-ota-architecture>`.

fiotest includes a `test specification`_ that allows you to define
how and what to test. The container can also be extended/customized
for your specific needs.
Your Factory includes a generic :ref:`API and data model <ref-fiotest>` that allows you the freedom to test how you choose.
In most cases, you'll be able to use fiotest as-is for your testing needs.

All things in a Factory revolve around Targets, and the testing mechanism is no different.
Devices run Targets.
The results of any tests they run are correlated with its Target.
This allows you to get a test reporting of Targets in your Factory.

The fiotest container can be set up in your Factory as a :ref:`Docker Compose application <ref-compose-apps>`.
The fiotest container has the ability to interact with the host OS.
It can configure aktualizr-lite to notify it when a new Target has been installed.
Then, fiotest is able to run tests and report the results back through the :ref:`device gateway <ref-ota-architecture>`.

fiotest includes a `test specification`_ that allows you to define how and what to test.
The container can also be extended/customized for your specific needs.

Workflow
~~~~~~~~
Expand Down
7 changes: 7 additions & 0 deletions source/reference-manual/testing/testing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@
Testing
=======

This section provides an overview of FoundriesFactory testing.

:ref:`ref-lmp-testplan` describes the *internal* testing process we follow to help ensure the quality of our releases.

For testing that you may wish to perform, :ref:`ref-testing-architecture` introduces the workflow and general approach.
:ref:`ref-fiotest` introduces the automated testing capabilities available to your Factory using a testing API.

.. toctree::
:maxdepth: 1

Expand Down

0 comments on commit 9729a52

Please sign in to comment.