This is an example test framework for testing embedded applications on the nRF52 and nRF53 platforms. It is not officially supported by Nordic Semiconductor.
Tests are written using the pytest framework, leveraging its filtering and reporting capabilities. The tests can call (C) functions running on the device and get data back, using nRF RPC.
├── build.sh # test FW build script
├── conftest.py # configuration script automatically run by pytest
├── pytest.ini # pytest configuration options
├── README.md
├── sample_devconf.yml # example static device configuration
├── targettest # testing framework (as python library)
│ ├── setup.cfg
│ ├── setup.py
│ └── targettest
│ ├── cbor.py # CBOR encoding/decoding
│ ├── devkit.py # Hardware device representation and operations
│ ├── provision.py # Preparation of the device for a test
│ ├── abstract_transport.py # interface for the nRF RPC packet transport
│ ├── rpc_packet.py # nRF RPC packet formatting
│ ├── rpc_channel.py # nRF RPC packet dispatch (into the test)
│ ├── uart_packet.py # UART packet encapsulation
│ └── uart_channel.py # Serial port handling & UART packet re-assembly
│
└── tests # Top-level test suite folder
└── bt_notify # example test suite
├── fw # firmware for that test suite
│ ├── boards # DeviceTree overlays for specific boards (set UART speed etc)
│ │ ├── nrf52840dk_nrf52840.overlay
│ │ ├── nrf5340dk_nrf5340_cpuapp.conf
│ │ └── nrf5340dk_nrf5340_cpuapp.overlay
│ ├── child_image # Configuration overrides for network core images
│ │ └── rpc_host.conf
│ ├── CMakeLists.txt
│ ├── prj.conf # Default prj.conf
│ ├── prj_nrf5340dk_nrf5340_cpuapp.conf # prj.conf used when compiling for that board
│ ├── src
│ │ ├── main.c
│ │ └── rpc_handler.c # target commands and events
│ └── test_rpc_opcodes.h # command and event IDs
└── test_bt_notify.py # contains the test logic
Install the targettest package as local. This will install the required dependencies.
pip3 install -e targettest/Source zephyr-env.sh (just like when building a stand-alone zephyr project).
Call build.sh, it will build all the test FW images for the 'nrf5340dk_nrf5340_cpuapp' and 'nrf52840dk_nrf52840' platforms.
The build output is located in build/, with the path of the test suite folder.
Firmware images are defined as standard zephyr applications, so the Zephyr and NCS documentation applies.
Use the pytest --collect-only option along with any filtering (-k) necessary.
To get a list of all the test suites (Class) and cases (Function) in the repo:
pytest --collect-onlySample output:
platform linux -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/john/repos/pytest-hw-ncs, configfile: pytest.ini
collected 4 items
<Module tests/bt_notify/test_bt_notify.py>
<Class TestBluetooth>
<Function test_boot>
<Function test_trigger_oops>
<Function test_scan>
<Function test_conn>Pytest's documentation applies.
Call pytest (no arguments) at the repo root.
Logs will only be printed if a testcase fails.
Use the --dut-family and --tester-family options.
E.g. run all tests with the DUT an nRF53 board and the tester an nRF52840 board:
pytest --dut-family=nrf53 --tester-family=nrf52By default, the test system will discover any nordic DKs connected to the computer and allocate the correct ones depending on the family. This behavior can be changed by using a static configuration. When using a static configuration, pytest will not 'waste' time by querying all DKs connected, and will only communicate with the devices in the static configuration.
Call pytest with the --devconf option:
pytest --devconf=./sample_devconf.ymlSee sample_devconf.yml for the format.
If quickly iterating on a testcase, it can be annoying to wait for the devices to be flashed (with the same FW image no less) on each run.
Use the --no-flash switch to skip flashing.
Pytest is quiet by default, only printing the python logger's output when a test fails.
But when debugging, it might be useful to have immediate feedback.
Use the --log-cli-level switch to set the live log level.
Use the -s switch to print stdout immediately.
pytest -s --log-cli-level=DEBUGPytest will by default run all the tests, even if one fails in the middle.
To stop on the first failure, use the -x switch.
The firmware can be debugged while the test is running.
Use the --no-emu switch to sever the test system's connection to the J-Link emulator.
As the test system uses the J-Link emulator connection for logging (over Segger RTT) and reset/halt, it will:
- not capture any device logs
- prompt the user to reset the boards for each testcase
To that end, a combination of switches have to be selected:
-sto get the reset prompts--no-flashto not connect to the emulator for flashing--no-emuto not connect to the emulator during the test execution
E.g. to debug a testcase called 'test_scan':
pytest -s --no-emu --no-flash -k test_scanPytest tests can be run in a python debugger, like any other script. VSCode is free and pretty nice, with support for threads. See this stackoverflow answer for an example debug configuration.
Pytest can emit JUnit reports as per the documentation:
pytest --junitxml=path/to/report.xmlThey can then be plugged in a CI system (like Jenkins) or easily viewed with junit2html:
# will generate path/to/report.xml.html
junit2html path/to/report.xmlPytest has the concept of Test fixtures.
In a nutshell, they define the environment of the test:
Instead on overloading setUp and tearDown methods, the test can require the presence of ready-to-use objects.
The testcase method simply defines a parameter, and pytest will look for and execute a function decorated with @pytest.fixture() that has the same name. This fixture function can either return a value, or act as a context manager and yield a value, allowing it to do some cleanup when the testcase exits.
E.g. a testcase that only deals with checking the throughput of a BLE connection can request the presence of a connection object, for which it can then call send and receive functions. Then the test code is very straightforward to read and there is no confusion on what it is actually testing.
Fixtures can be scoped (session, class, case) and can also request other fixtures.
When pytest is invoked:
- Parsing starts at
conftest.pypytest_addoption()adds some custom options to the pytest cli- The
devkits()fixture registers development kits connected to the computer
Pytest begins executing a test suite:
- The
flasheddevices()fixture provisions two devices of the correct family from the registered list, and flashes them with the firmware that matches the test suite's folder name. The emulator is also connected to. The unused DKs' CPUs are halted.
Pytest begins executing a test case:
- The
testdevices()fixture is requested by the testcases. It opens the PC's serial port, and initiates nRF RPC communication. Once it has received the READY event (0x01) for both DUT and Tester, and opened the RTT logging channel, it returns the two devices as a dict.
Pytest ends the test case:
- Control is returned to
testdevices(), which then prints the device logs for that test case.
Pytest ends the test suite:
- Control is returned to
flasheddevices(), that in turn releases theFlashedDeviceobjects, closing the emulator connections.
It is possible for the python test script to call test functions defined in the firmware. This happens using the nRF RPC library, communicating over UART (serial port). Similarly, it is possible to receive events from the device.
The parameters (or event data) are encoded in the CBOR serialization format, using the ZCBOR library.
The format is roughly: type, opcode, nRF RPC metadata, cbor-encoded data
The opcode is a byte, and the IDs used in the firmware and in the test script need to match. It is recommended to use enums to that effect.
Due to a limitation in nRF RPC, we cannot use the CMD packet type, and instead have to use the EVT packet type, which is asynchronous (non-blocking). EVT packets will still block until an ACK packet is received from the other side.
RPCChannel.evt(): send an event without any data to the device.RPCChannel.evt_cbor(): send an event with encoded data (or parameters) to the device.
Events that are emitted on target are stored in a python Queue in a FIFO manner.
RPCChannel.get_evt(): get an event from the device. No decoding will be done if it contains a data payload.RPCChannel.get_evt_cbor(): get an event from the device. A tuple is returned, containing the raw event and its decoded payload.