|
|
|
@ -15,6 +15,12 @@ one the expected output. Tests can be organized into groups for easier
|
|
|
|
|
organization and to facilitate more complex grading scenarios (e.g. "An
|
|
|
|
|
assignment passes only if it passes all tests in group X").
|
|
|
|
|
|
|
|
|
|
Tests are run by a special program called a *judge*, which compares the output
|
|
|
|
|
of the program to the expected output. By setting a judge, we can specify how
|
|
|
|
|
strict the testing is - for example, some assignments require the solution to
|
|
|
|
|
output exactly the same bytes as expected. Others permit any number of
|
|
|
|
|
whitespace characters between words of the output.
|
|
|
|
|
|
|
|
|
|
### Stages
|
|
|
|
|
|
|
|
|
|
A stage is a logical unit of the testing process. It specifies how to do a step
|
|
|
|
@ -30,6 +36,8 @@ contained in its configuration:
|
|
|
|
|
- **Test input policy** - how to pass the test input to the program?
|
|
|
|
|
- redirect it to its standard input
|
|
|
|
|
- pass the name of the input file as an argument
|
|
|
|
|
- **Judge** - which judge should be used to evaluate the solution's output?
|
|
|
|
|
Custom judges can be supplied with the assignment.
|
|
|
|
|
- **Error policy** (optional) - what should we do when a test fails?
|
|
|
|
|
- **interrupt** the stage (default)
|
|
|
|
|
- **continue** with the next test
|
|
|
|
|