Controlling the pretty-print format for failing unit tests

[larshum]

When a utest fails, it pretty-prints the contents of the left- and right-hand side using an expression printer (boot) or generated pretty-print functions for a particular type (mi). In most cases, this works fine, so it would be annoying if all unit tests required a user-defined pretty-print function. However, I commonly end up in situations where the default pretty printer could be (far) better. The result is that I end up re-running the program without tests but with a manually inserted print of what I'm testing to see what went wrong.

For example, consider the outputs at the bottom of this text (click to expand, they're pretty big), where I compare two almost equivalent ASTs consisting of one let-binding and an application. I find this completely unreadable, and I'm sure you agree. So what I end up doing in situations like this is to add a printLn (expr2str lhs) and then run without --test, which in this case would yield something like

let f =
  lam x.
    addi
      x
      1
in
f
  3

Furthermore, we want the flexibility to control how the utest arguments are pretty-printed. I think the format of expr2str works in most cases, but when testing the type-checker, this format would not be sufficient, as it does not show the underlying types. On the other hand, printing the types would be annoying when doing tests for the symbolizer, which runs before the type-checker. Therefore, I argue that we want the ability to control how the arguments of a utest are pretty-printed when a test fails.

However, I don't have any concrete suggestions for the syntax. The main problem is that we currently allow the left and right arguments to have different types when an equality function is provided. Therefore, we have to be able to specify which pretty-print function to use for which argument, and I cannot think of any syntax for that which I like. To get the discussion started, I'll propose the following syntax

utest a with b using {eq = c, lpp = d, rpp = e} in
...

where all fields of the record are optional.

Boot output

/private/tmp/example.mc: 
 ** Unit test FAILED: FILE "/private/tmp/example.mc" 5:0-15:12 **
    LHS: LetAst_TmLet {tyAnnot = (UnknownTypeAst_TyUnknown {info = (NoInfo ())}),
              tyBody = (UnknownTypeAst_TyUnknown {info = (NoInfo ())}),
              inexpr = (AppAst_TmApp {info = (NoInfo ()),
                                      rhs = (ConstAst_TmConst {info = (NoInfo (
                                                               )),
                                                               val = (IntAst_CInt {
                                                               val = 3}),
                                                               ty = (UnknownTypeAst_TyUnknown {
                                                               info = (NoInfo (
                                                               ))})}),
                                      lhs = (VarAst_TmVar {frozen = false,
                                                           ident = (("f"),
                                                                    symb(95258)),
                                                           info = (NoInfo (
                                                           )),
                                                           ty = (UnknownTypeAst_TyUnknown {
                                                           info = (NoInfo (
                                                           ))})}),
                                      ty = (UnknownTypeAst_TyUnknown {
                                      info = (NoInfo ())})}),
              ident = (("f"),symb(95258)),info = (NoInfo ()),
              body = (LamAst_TmLam {tyIdent = (UnknownTypeAst_TyUnknown {
                                    info = (NoInfo ())}),
                                    tyAnnot = (UnknownTypeAst_TyUnknown {
                                    info = (NoInfo ())}),
                                    ident = (("x"),symb(95258)),
                                    info = (NoInfo ()),
                                    body = (AppAst_TmApp {info = (NoInfo (
                                                          )),
                                                          rhs = (ConstAst_TmConst {
                                                          info = (NoInfo (
                                                          )),
                                                          val = (IntAst_CInt {
                                                          val = 1}),
                                                          ty = (UnknownTypeAst_TyUnknown {
                                                          info = (NoInfo (
                                                          ))})}),
                                                          lhs = (AppAst_TmApp {
                                                          info = (NoInfo (
                                                          )),
                                                          rhs = (VarAst_TmVar {
                                                          frozen = false,
                                                          ident = (("x"),
                                                                   symb(95258)),
                                                          info = (NoInfo (
                                                          )),
                                                          ty = (UnknownTypeAst_TyUnknown {
                                                          info = (NoInfo (
                                                          ))})}),
                                                          lhs = (ConstAst_TmConst {
                                                          info = (NoInfo (
                                                          )),
                                                          val = (ArithIntAst_CAddi (
                                                          )),
                                                          ty = (UnknownTypeAst_TyUnknown {
                                                          info = (NoInfo (
                                                          ))})}),
                                                          ty = (UnknownTypeAst_TyUnknown {
                                                          info = (NoInfo (
                                                          ))})}),
                                                          ty = (UnknownTypeAst_TyUnknown {
                                                          info = (NoInfo (
                                                          ))})}),
                                    ty = (UnknownTypeAst_TyUnknown {info = (NoInfo (
                                                                    ))})}),
              ty = (UnknownTypeAst_TyUnknown {info = (NoInfo ())})}
    RHS: LetAst_TmLet {tyAnnot = (UnknownTypeAst_TyUnknown {info = (NoInfo ())}),
              tyBody = (UnknownTypeAst_TyUnknown {info = (NoInfo ())}),
              inexpr = (AppAst_TmApp {info = (NoInfo ()),
                                      rhs = (ConstAst_TmConst {info = (NoInfo (
                                                               )),
                                                               val = (IntAst_CInt {
                                                               val = 2}),
                                                               ty = (UnknownTypeAst_TyUnknown {
                                                               info = (NoInfo (
                                                               ))})}),
                                      lhs = (VarAst_TmVar {frozen = false,
                                                           ident = (("f"),
                                                                    symb(95258)),
                                                           info = (NoInfo (
                                                           )),
                                                           ty = (UnknownTypeAst_TyUnknown {
                                                           info = (NoInfo (
                                                           ))})}),
                                      ty = (UnknownTypeAst_TyUnknown {
                                      info = (NoInfo ())})}),
              ident = (("f"),symb(95258)),info = (NoInfo ()),
              body = (LamAst_TmLam {tyIdent = (UnknownTypeAst_TyUnknown {
                                    info = (NoInfo ())}),
                                    tyAnnot = (UnknownTypeAst_TyUnknown {
                                    info = (NoInfo ())}),
                                    ident = (("x"),symb(95258)),
                                    info = (NoInfo ()),
                                    body = (AppAst_TmApp {info = (NoInfo (
                                                          )),
                                                          rhs = (ConstAst_TmConst {
                                                          info = (NoInfo (
                                                          )),
                                                          val = (IntAst_CInt {
                                                          val = 1}),
                                                          ty = (UnknownTypeAst_TyUnknown {
                                                          info = (NoInfo (
                                                          ))})}),
                                                          lhs = (AppAst_TmApp {
                                                          info = (NoInfo (
                                                          )),
                                                          rhs = (VarAst_TmVar {
                                                          frozen = false,
                                                          ident = (("x"),
                                                                   symb(95258)),
                                                          info = (NoInfo (
                                                          )),
                                                          ty = (UnknownTypeAst_TyUnknown {
                                                          info = (NoInfo (
                                                          ))})}),
                                                          lhs = (ConstAst_TmConst {
                                                          info = (NoInfo (
                                                          )),
                                                          val = (ArithIntAst_CAddi (
                                                          )),
                                                          ty = (UnknownTypeAst_TyUnknown {
                                                          info = (NoInfo (
                                                          ))})}),
                                                          ty = (UnknownTypeAst_TyUnknown {
                                                          info = (NoInfo (
                                                          ))})}),
                                                          ty = (UnknownTypeAst_TyUnknown {
                                                          info = (NoInfo (
                                                          ))})}),
                                    ty = (UnknownTypeAst_TyUnknown {info = (NoInfo (
                                                                    ))})}),
              ty = (UnknownTypeAst_TyUnknown {info = (NoInfo ())})}
    Using: #var"MExprEq_eqExpr"
ERROR! 0 successful tests and 1 failed tests.

Mi output

 ** Unit test FAILED: </private/tmp/example.mc 5:0-15:12>**
    LHS: LetAst_TmLet {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, ident = (['f'], sym (100339)), body = LamAst_TmLam {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, ident = (['x'], sym (100339)), body = AppAst_TmApp {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, lhs = AppAst_TmApp {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, lhs = ConstAst_TmConst {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, val = ArithIntAst_CAddi {}}, rhs = VarAst_TmVar {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, ident = (['x'], sym (100339)), frozen = false}}, rhs = ConstAst_TmConst {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, val = IntAst_CInt {val = 1}}}, tyAnnot = UnknownTypeAst_TyUnknown {info = NoInfo {}}, tyIdent = UnknownTypeAst_TyUnknown {info = NoInfo {}}}, tyAnnot = UnknownTypeAst_TyUnknown {info = NoInfo {}}, inexpr = AppAst_TmApp {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, lhs = VarAst_TmVar {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, ident = (['f'], sym (100339)), frozen = false}, rhs = ConstAst_TmConst {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, val = IntAst_CInt {val = 3}}}, tyBody = UnknownTypeAst_TyUnknown {info = NoInfo {}}}
    RHS: LetAst_TmLet {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, ident = (['f'], sym (100339)), body = LamAst_TmLam {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, ident = (['x'], sym (100339)), body = AppAst_TmApp {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, lhs = AppAst_TmApp {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, lhs = ConstAst_TmConst {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, val = ArithIntAst_CAddi {}}, rhs = VarAst_TmVar {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, ident = (['x'], sym (100339)), frozen = false}}, rhs = ConstAst_TmConst {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, val = IntAst_CInt {val = 1}}}, tyAnnot = UnknownTypeAst_TyUnknown {info = NoInfo {}}, tyIdent = UnknownTypeAst_TyUnknown {info = NoInfo {}}}, tyAnnot = UnknownTypeAst_TyUnknown {info = NoInfo {}}, inexpr = AppAst_TmApp {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, lhs = VarAst_TmVar {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, ident = (['f'], sym (100339)), frozen = false}, rhs = ConstAst_TmConst {ty = UnknownTypeAst_TyUnknown {info = NoInfo {}}, info = NoInfo {}, val = IntAst_CInt {val = 2}}}, tyBody = UnknownTypeAst_TyUnknown {info = NoInfo {}}}
    Using: #var"MExprEq_eqExpr"

[br4sco]

An alternative is that the user can supply a function all a. all b. a -> b -> () that is applied to the lhs and rhs on a failing test. E.g.,

utest a with b using eqab 
  onfail lam a. lam b. print "lhs:"; printa a; print "\nrhs:"; printb b; print "\n" 
in
...

This would require a new keyword unless someone is creative with the existing ones. The idea is that a failing utest would not pretty print the lhs and rhs if this function is specified.