|
|
@ -10,6 +10,10 @@ build, run and test it. The testing process is divided into stages.
|
|
|
|
|
|
|
|
|
|
|
|
### Tests
|
|
|
|
### Tests
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
A test is a pair of files, where the first one specifies an input and the other
|
|
|
|
|
|
|
|
one the expected output. Tests can be organized into groups for easier
|
|
|
|
|
|
|
|
organization and to facilitate more complex grading scenarios (e.g. "An
|
|
|
|
|
|
|
|
assignment passes only if it passes all tests in group X").
|
|
|
|
|
|
|
|
|
|
|
|
### Stages
|
|
|
|
### Stages
|
|
|
|
|
|
|
|
|
|
|
@ -20,8 +24,12 @@ contained in its configuration:
|
|
|
|
- **Name** - a unique string identifier of the stage
|
|
|
|
- **Name** - a unique string identifier of the stage
|
|
|
|
- **Build command** (optional) - used to prepare the submitted files for this
|
|
|
|
- **Build command** (optional) - used to prepare the submitted files for this
|
|
|
|
test stage
|
|
|
|
test stage
|
|
|
|
- **Test list** - specifies the tests to be run during this stage
|
|
|
|
- **Test list** - specifies the tests (or test groups) to be run during this
|
|
|
|
|
|
|
|
stage
|
|
|
|
- **Test command** - used to run one specific test
|
|
|
|
- **Test command** - used to run one specific test
|
|
|
|
|
|
|
|
- **Test input policy** - how to pass the test input to the program?
|
|
|
|
|
|
|
|
- redirect it to its standard input
|
|
|
|
|
|
|
|
- pass the name of the input file as an argument
|
|
|
|
- **Error policy** (optional) - what should we do when a test fails?
|
|
|
|
- **Error policy** (optional) - what should we do when a test fails?
|
|
|
|
- **interrupt** the stage (default)
|
|
|
|
- **interrupt** the stage (default)
|
|
|
|
- **continue** with the next test
|
|
|
|
- **continue** with the next test
|
|
|
|