Testrun automates specific test cases to verify network and security functionality in IoT devices. It's an open-source tool that manufacturers use to test their IP-capable devices for the purpose of device qualification within Google's Building Operating System (BOS) program.
Test labs and engineers often need to maintain a large and complex network coupled with dynamic configuration files and constant software updates. Testrun helps address major issues like:
- The complexity of managing a testing network
- The time required to perform testing of network functionality
- The accuracy and consistency of testing network functionality
Testrun creates an isolated and controlled network environment on a Linux machine. This removes the necessity for complex hardware, advanced knowledge, and networking experience while enabling test engineers to validate device behavior against Google's BOS requirements.
Testrun supports two modes: automated testing and lab network.
Automated testing of the device under test (DUT) begins once the device is operational (steady state). Containerized test modules execute against the device one module at a time. Testrun produces a report with the results after all modules are executed.
Testrun provides the network and assistive tools for engineers when manual testing or configuration changes are required, reducing the need to maintain a separate but identical lab network. Testrun handles packet captures and logs for each network service for further debugging.
- PC running Ubuntu LTS 20.04, 22.04, or 24.04 (laptop or desktop)
- 2x USB Ethernet adapter (one may be built-in Ethernet)
- Internet connection
Testrun requires Docker. Refer to the installation guide for more information.
The DUT must be able to obtain an IP address via DHCP.
Once you meet the hardware and software requirements, follow the Testrun Get started guide. Additional guidance is available in the docs directory.
Testrun continually evolves to further support end users by automating device network behavior against industry standards. For information on upcoming features, check out the Roadmap.
We're proud of our tool and strive to provide an enjoyable experience for everyone. Testrun goes through rigorous accessibility testing at each release. Download the Testrun: Accessible features video to learn more.You're welcome to submit a new issue and provide feedback on our implementations. To learn more about Google's Belonging initiative and their approach to accessibility, visit their site.
If you encounter a problem during setup or use, raise an issue under the Issues tab. Issue templates exist for both bug reports and feature requests. If neither of these apply, raise a blank issue instead.
We strongly encourage contributions from the community. Review the requirements on the "How to Contribute" page, then follow the developer guidelines.
You can resolve most issues by reinstalling Testrun using these commands:
sudo docker system prune -asudo apt install ./testrun-*.deb
If this doesn't resolve the problem, raise an issue.
Best practices and requirements for IoT devices change often due to technological advances and discovery of vulnerabilities. You can find the current expectations for IoT devices on Google deployments in the Application Security Requirements for IoT Devices. Testrun aims to automate as much of the Application Security Requirements as possible.
The following network services are containerized and accessible to the DUT:
- DHCP in failover configuration with internet connectivity
- IPv6 SLAAC
- DNS
- NTPv4
Note that this list is likely to change over time.
Testrun can be virtualized if the 2x Ethernet adapters are passed through to a VirtualBox VM as a USB device rather than managed network adapters. Visit the virtual machine guide for additional details.