The format of a .dyalogtest file

A .dyalogtest file is a text file that contains declarations describing a test suite (a collection of tests, optionally prepared by executing setup code and optionally completed by running teardown fn(s)). The format is {item}:{value}[,{options}] (casing is irrelevant), options depend on the item. Items of a .dyalogtest file are:

• DyalogTest (optional) - numeric, gives the minimum version of DTest that the suite requires
• ID (optional) - charvec
• Description (optional) - charvec, might be interesting for others and is printed on screen when test is run
• SuccessValue (optional) - charvec, defines the return value of a successful test. (more)
• AlertIfCoverageBelow (optional) - numeric, defines a threshold that will be checked against when running Coverage tests. Useful if you want to make sure that coverage never goes below a certain value that you already achieved.
• CodeCoverage_Subject (optional) - charvec, points to the subject of coverage tests (by default, all namespaces below # will be observed)
• CodeCoverage_Ignore (optional) - charvec, contains names of namespaces to ignore in Coverage-Analysis. (NB: these 3 parameters are only considered when coverage analysis is done. To enable coverage analysis when running tests, use the -coverage modifier.
• Setup (optional) - charvec, name of setup to run before executing test (if multiple entries exist, DTest will run all tests against all setups!
• Test (mandatory) - charvec, name of a test function
• Teardown (optional) - charvec, name of the teardown function that is run when all tests are executed (this can "undo" what Setup did - mostly relevant wrt to external things such as DB Connections, file ties etc.)

Additionally, .dyalogtest files can be commented using APL-Comments (with the "lamp" symbol ⍝)

Reasons for having a .dyalogtest file

Given the flexibility of DTest's commandline, it may seem redundant or even pointless to have an additional file that explicitely defines what tests to run. After all, we have the naming convention test_* and can easily determine what to do. And the commandline gives us 1000s of modifiers to specify all details. Here are some of the advantages of .dyalogtest that we found:

• tests can be run in different contexts

  • nightly, unattended QA tests that should check with meticulous care
  • interactively, by the developer as he works on the code - he may focus on "core" functionality or just the area that he's working on. (Which he's can do with in good conscience as he's aware that "nightly" would catch possible regressions in other areas)

• a suite helps with repeatability and automation of tests (in a CI environment); commandline & modifiers are more suited for interactive use

• TBC (Comments or PRs welcome!)