diff --git a/doc/commands.md b/doc/commands.md index 87ca1404..3eae9a17 100644 --- a/doc/commands.md +++ b/doc/commands.md @@ -22,7 +22,7 @@ This lists all subcommands and their most important options. - Problem validation - [`bt input [-v] [testcases [testcases ...]]`](#input) - [`bt output [-v] [testcases [testcases ...]]`](#output) - - [`bt validate [-v] [--input | --answer] [--remove | --move-to DIR] [testcases [testcases ...]]`](#validate) + - [`bt validate [-v] [--input | --answer | --invalid] [--remove | --move-to DIR] [testcases [testcases ...]]`](#validate) - [`bt constraints [-v]`](#constraints) - Creating new contest/problems - [`bt new_contest [contestname]`](#new_contest) @@ -262,7 +262,9 @@ See `run ` for a description of how to pass testcases. `bt validate --answer ` is similar to `bt validate --input` but validates `.ans` files instead of `.in` files. -`bt validate` validates both input and answer files. +`bt validate --invalid ` checks invalid test cases in `data/invalid_*`. + +`bt validate` validates both input and answer files, and the invalid test cases. It supports the following flags when run for a single problem: diff --git a/doc/validation.md b/doc/validation.md index fe48443e..f2e96c57 100644 --- a/doc/validation.md +++ b/doc/validation.md @@ -26,11 +26,11 @@ These are some things that hold for all types of validation mentioned below. working directory. For `output validation`, you can write e.g. files `judgemessage.txt`, `judgeerror.txt`, `teammessage.txt`, and `score.txt`. (BAPCtools only handles the first 2.) -- The return code should be `42` for success/AC. -- The return code should be `43` for failure/WA. (Note that the spec is +- The return code must be `42` for successful validation. +- The return code must be `43` for failed validation. (Note that the spec is slightly more lenient and allows any non-`42` return code for input format validation. BAPCtools expects a code of exactly `43` when validating - `data/invalid_inputs` testcases (see below).) + invalid testcases (see below).) - For input and answer validation, the out-of-spec `--constraints-file ` flag is set when running `bt constraints`. The validator can write some statistics on the testcase to this file. See the [implementation @@ -76,6 +76,67 @@ output_validator /path/to/testcase.in /path/to/testcase.ans /path/to/feedbackdir In particular, note the flags `case_sensitive` and `space_change_sensitive`, which allows an output validator to be more strict about the format of `.ans` files than about submission output. +## Invalid test cases + +`bt validate --invalid` + +BAPCtools expects deliberately invalid testcases in `data/invalid_{inputs, answers, outputs}`. +These ensure that validators reject under the expected circumstances. + +### Invalid inputs + +Invalid inputs are placed in `data/invalid_inputs` and consist only of an `.in`-file. +An invalid input must be rejected by at least one input validator. + +Examples: + +```yaml +"negative": + in: "-1" +"too_large": + in: "100" +"not_a_number: + in: foo +"extra input": + in: 0 1 +"trailing whitespace": + in: " 0" +``` + +### Invalid answers + +Invalid answers are test cases in `data/invalid_answers`. +Such a test case consist of input and answer files (`.in` and `.ans`), just like a normal test case. +The input file must pass input validation (i.e., all input validators must accept). +The testcase must fail answer validation, i.e., at least one answer validator or the output validator must reject it. +The output validator is run in strict mode, i.e., with the flags `case_sensitive` and `space_change_sensitive`; +to ensure maximum conformity of answer files in the test data. + +Examples: + +```yaml +"negative": + in: "0" + ans: "-1" +"trailing whitespace": + in: "0" + ans: " 0" +``` + +### Invalid outputs + +Invalid outputs are valid test cases in `data/invalid_output` with an additional `.out`-file that must fail output validation. +In particular, the input file must pass input validation, the answer file must pass answer validation. +However, the `.out`-file must fail output validation for the test case. + +Examples: +```yaml +"wrong_answer": + in: "0" + ans: "0" + out: "1" +``` + ## Output validation The output validator checks whether the output of a submission is correct. @@ -112,19 +173,3 @@ validator instead of a file. Similarly, `stdout` is connected to a pipe that forwards to the submission's `stdin`. -## `data/invalid_inputs` validation - -BAPCtools allows testcases in `data/invalid_inputs` to test that validators fail on -specific types of input that do not follow the constraints. In -particular: - -- If `data/invalid_inputs/.in` is present, at least one input format validator must fail (return `43`). - -## `data/invalid_outputs` validation (out-of-spec) - -BAPCtools allows testcases in `data/invalid_outputs` to test that output-format -validators (for problems with default validation) fail on -specific types of output that do not follow the constraints. In -particular: - -- If `data/invalid_outputs.{in,ans}` are both present, at least one output format validator must fail (return `43`).