Writing tests

How to test

Following the philosophy of unit testing, you will normally write one or more tests of a discrete area of functionality that your application provides. While tools such as Selenium are used for testing of user-interaction, these tests are more geared towards backend functionality, ie. a test could call a calculation engine with certain arguments and then test the result against an expected value. Testing isolated parts of the app ("units") will give you confidence that these units are correct.

Using CI-Tools, these tests can be integrated into the development-cycle and can be executed automatically whenever a source-repository is updated and provide immediate notification if the code produces different results.

A test typically contains 3 steps:

• setup: bring in tools etc. to put the workspace into a state where tests can run. It is not mandatory to have a setup - you can alternatively perform the neccessary steps manully - but this will make it more difficult to reproduce reliably. The advantage of the setup is that is a reliable way to create the environment for tests.

• running tests: a "test" is a function that is executed directly (by referencing it on the commandline) or a part of a test-suite. The function is supposed to return a result which can either be a

  • text-vector (or a vtv) containing messages about failed tests

  • an empty vector as an indication for successful completion of the test.

    A discussion of the test-DSL follows in a separate section below.

    NB: the SuccessValue modifier allows setting alternate results to associate with successful tests. (It can also be used as a directive in .dyalogtest files)

• teardown: removal of everything that was setup for the tests.

Note that the exact same "3-step-policy" applies to test-suites.

Writing a test

A test-fn has to be a monadic fn that returns a result (it is recommended to use the variable r for the result as DSL statements such as Because refer to it). So the header needs to be

∇ r←MyTest sink
∇

Usually some initialization for the specific tests may follow (ie. bringing in test-data and initialization of variables). And then tests are executed by calling your functions and comparing their result against the expected result.

In Dyalog, we settled on a convention for our own tests to always branch to a label fail if a test fails. This leads to an almost "literal programming"-style, as the Test-DSL enables code like "→fail Because 'Got wrong result!' - so a typical test looks like the following:

∇ r←MyFirstTest nul
  r←'' ⍝ Initialize
  :If 4 Check {execute fn that normally returns scalar 4}
     →fail Because'2+2≠4!'
  :EndIf
  ...
  more tests
  ...
  →0
  fail:
  ⍝ possibility for some local teardown...
∇

• it is important to notice that the fn Check returns TRUE if its arguments do NOT match. This may feel counter-intuitive, but results in clearer code ;)
• The variable r will be updated by the Because-function.
• tests will always be executed in a separate "fresh" namespace. So your setup should take care to create the environment without relying on fixed paths - or alternatively set up things in #.
• Check (as well as Assert) internally uses ≡ for comparisons

v1.70

v1.70 introduced important changes to testing:

• test fns may now also be dfns
• instead of returning an empty string to indicate "success", a testing fn may also return a boolean or in fact, any value. .dyalogtestfiles have a new specifier which can be used to define the "SuccessValue". It is also available as a modifier when calling ]DTest.
• the Assert fn allows for a simpler structure of test fns

Tests vs. Test-Suite

While a given test-fn will perform tests on one unit of your code (or one area of functionality that a given fn provides), several of these tests can be jointly executed through a test-suite. The suite also provides options to define one or more (alternate) setups that the tests will be run against. Expressed in pseudo-code:

r←''
:For setup :in SETUPS
   :If 0=≢res←⍎setup
      :For test :In TESTS
         r,←test
      :EndFor
   :Else  
   r,←res
:EndFor
⍎¨TEARDOWN

As you see, all tests are executed against all setups (read next section for details about sequence of tests).

A suite is a text-file with the extension .dyalogtest. It contains the "master-design" of tests: setup-routines, selection of tests and finally one or more teardown-functions which make sure that tests will leave an empty sandbox behind: it is important to ensure that all external links, i.e. file-ties, are properly closed.

Order of tests

By default, tests and setups will each be executed in random order (to detect undesired possible side-effects and dependencies between them). The order modifier of ]DTest gives you control over the order of execution:

• order=0 => random order
• order=1 => alphabetical order
• order="_a b c_" => specify desired order directly (note that this can not be used to control the number of tests executed - we will always execute all neccessary tests)

The Test-DSL

We make additional functions available when tests are run to facilitate easier comparison against reference-results. They are mostly defined in the (temporary) namespace the test executes in. You can also access "system-variables" (of the ]DTest-System) by referring to the parent namespace ##. Likely candidates are included in this documentation.

Check

bool←expect Check got

This function can be used to compare the expected result of a function execution to the actual result. If the command-line switch "-halt" is used, it will suspend execution when it finds that expect≢got. (If the values are small (less than 200 bytes), they will be shown in the session to facilitate diagnosis.) NB: the result will be 0 if ⍺≡⍵ and 1 if they differ!

Assert (v1.70)

Assertis the classic fn used in almost all languages when implementing tests. It is also available with DTest. Details on a dedicated page.

IsNotElement

bool←got IsNotElement expected_list

There might be cases in which you do not expect one specific result, but rather would look for one of several possible outcomes. While that could also be emulate with statements like 1 Check(res∊A B C), you'd loose easy access to the "got-part" when examining failures. Therefore we provide this function which makes it easier to check for membership - or better: non-membership.

The left argument is expected to be a scalar and anything with ≢>1 will automatically be nested, so you can use expressions such as 'one' IsNotElement 'one' 'two' 'three' without needing nesting of ⍺.

Because

larg←larg Because text

This function adds the text to the test's log (in and returns the left argument. If you have a label "fail", you can then write →fail Because'explanation'.

Log

[larg]Log txt

Adds text in txt to the error-log. Optional larg is a prefix (we will add a ":" to separate that) or a vector with options in name/value pairs:

• Prefix - a text that is prefixed to the txt you logged (separated by :)

• Type: one of I|W|E to indicate the type of message you're logging:

  • Information
  • Warning
  • Error

halt (boolean flag)

Halt on error? (1=Yes, 0=No). This allows writing :trap (~##.halt)/0 which is recommended to conditionally disable your internal trapping, if you want to allow inspection of the code on error (if executing with the -h-Switch).

verbose (boolean flag)

The value of the verbose-switch can be accessed from a test in the parent-namespace: ##.verbose to see if additional (verbose) output was asked for. Do not use Log (that would indicate failure!) but rather send output to the session using ⎕←'output'.

##.TESTSOURCE

Path of the folder where the test is stored.

##.DYALOG

Path to the interpreter's home-directory. (No trailing backslash)

##.RandomVal

[context]RandomVal arg

Random numbers are useful when you want to test functions against varying input. However, it might be challenging to reproduce errors that only occur with certain input. This function provides a way to generate random numbers that allow for easy reproduction of an error: all results will be stored in a file (named context.rng.txt with context being optional with a default related to ⎕SI and ⎕LC.) All *.rng.txt-files will be deleted when a suite executed successfully - otherwise they will be kept. When the function is called and a corresponding .rng.txt-file is found, the values stored in that file will be used. In other words: as long as your test fails, it will use the same random numbers.

The right argument arg contains the arguments to ? as follows:

• If rarg has one element, it means monadic application of ?rarg[1]
• Alternatively, rarg may have two elements which means rarg[2]?rarg[1]

Common variables

Please refer to this link for more information about variables that are available for DBuild and DTest.

Samples

Dyalog uses ]DTest for our own utilities as well, so feel free to study these tests and use them as inspiration for your testing:

• Conga-Tests
• Isolates