a few words about judges

master
Teyras 9 years ago
parent d4f39620e7
commit a44a0c0a49

@ -15,6 +15,12 @@ one the expected output. Tests can be organized into groups for easier
organization and to facilitate more complex grading scenarios (e.g. "An
assignment passes only if it passes all tests in group X").
Tests are run by a special program called a *judge*, which compares the output
of the program to the expected output. By setting a judge, we can specify how
strict the testing is - for example, some assignments require the solution to
output exactly the same bytes as expected. Others permit any number of
whitespace characters between words of the output.
### Stages
A stage is a logical unit of the testing process. It specifies how to do a step
@ -30,6 +36,8 @@ contained in its configuration:
- **Test input policy** - how to pass the test input to the program?
- redirect it to its standard input
- pass the name of the input file as an argument
- **Judge** - which judge should be used to evaluate the solution's output?
Custom judges can be supplied with the assignment.
- **Error policy** (optional) - what should we do when a test fails?
- **interrupt** the stage (default)
- **continue** with the next test

Loading…
Cancel
Save