TODO 2: Tests

[nathanjhood]

• Add Catch2 for unit tests #165
• Create Test Spec class or struct #166
• WIP: Begin test coverage #167
  • instance constructs/deconstructs safely
  • REQUIRE_THAT(box.size.x, WithinRel(::rack::mm2px(5.08F))), etc... #327
  • box.size.y == ... as above #328

[nathanjhood]

Idea 💡 run Rack in headless mode with debugger attached during CI runs

I've added Debug and Release modes to the build system and codebase, which provides hooks for using things like assert() alongside the unit tests.

To facilitate this, I've added a launch.json for launching Rack directly from VSCode, with a C++ debugger attached; this makes it possible to do things like add breakpoints to the code.

Very interesting: when running Rack like this, it actually fires it's own debug assertions at you at runtime, in response to your module code. I've encountered debug assertions that my modules were a number of pixels short of the required module height (128.5mm precisely results in a number of pixels just below the required 380 - Inkscape mutates source files automatically to round mm's to valid pixel integer values...)

I've not quite been able to construct a ModuleWidget within Catch2 - the test case instantly fails with a SIGABRT or a SIGSEV, which is really questionable in itself - if it cannot be constructed, why make a class/struct, which has a constructor?

I was hoping to construct the ModuleWidget inside my unit tests so that I can check box.size has not changed as a result of some mutation of the SVG files by Inkscape which might pass unnoticed... instead, I can at least assert at construction-time everything which I need.

The issue with that is, it requires running Rack, and with a debugger attached, in order to check those assertions.

The current CI pipeline covers unit tests thanks to Catch2 and CTest. But, there is no end to end testing.

I've quietly added a step to each workflow, to install VCV Rack 2 Free from official sources:

• Winget for Windows
• homebrew cask for MacOS
• cURL for Linux 😝

The only platform which actually requires the installation of the Rack executable is Windows (specifically, MinGW!) because the Rack SDK only ships with a libRack.dll.a, which is missing the other half, the libRack.dll itself; that ships with Rack 2 Free. This was a major source of pain for me when I was first attempting the vcpkg/CMake thing, and the realisation came that the Rack docs do actually inform us that the Rack executable is a requirement for development, which I had totally overlooked.

Anyway, Rack can be run from the command line and do lots of cool things, like load a custom project template - very handy, or take screenshots - very very handy indeed. It supports a --headless mode which runs with no GUI... perfect for CI 👍

Solution

My thinking is to add a step within the workflows which runs the built plugin in headless mode.

Gotchas

It will probably only make sense to do this if we can get the program to close itself after x amount of time, by hoping and assuming that all runners have support for something like timeout 60 ${RACK_INSTALL_DIR}/Rack.

I immediately don't like that this adds 60 seconds to every build run. 😢 a reasonable amount of uptime is required... my benchmarking at home suggests that Rack should boot within under one second on any of those runners. And yet, as I write, GitHub is running incredibly slow this evening and each workflow is crawling well below usual speeds by a factor between x4 and x6. The timeout 60 command is a fixed number and does not account for anything like that, meaning the workflow would probably fail "for no reason" on occasions such as tonight...

I'd also like to figure out how to extract Rack's log.txt if anything fails.

More thought required before committing.

[nathanjhood]

I immediately don't like that this adds 60 seconds to every build run

Could be mitigated slightly by only running the, let's call them "runtime assertion tests", on the preview branch, with the option to run them customarily from the actions dispatch button.

Slightly smelly because it allows "bugged" code to pass between branches, up until that code reaches the branch with the "runtime assertion tests" workflow.

[nathanjhood]

Idea 💡 add CTest config for running Rack executable with C++ debugger attached

Yes - it means probably writing more CMake or whatever to make sure the Rack executable is acquired, then doing find_program and stuff... and probably should end up being part of the Rack-SDK CMake stuff.

But, parity is very important. A CTest wrapper around around launching Rack to test the built plugin will ensure the same test conditions across users, devs/contributors, platforms, and CI runners.

Not in a rush to do it since I'm personally happy with the VSCode launch.json for now, but this will become a good one at some point in the near future.