|
|
|
@ -24,8 +24,13 @@ whitespace characters between words of the output.
|
|
|
|
|
### Stages
|
|
|
|
|
|
|
|
|
|
A stage is a logical unit of the testing process. It specifies how to do a step
|
|
|
|
|
in the build process and how to test if it behaves correctly. The following is
|
|
|
|
|
contained in its configuration:
|
|
|
|
|
in the build process and how to test if the student's submission behaves
|
|
|
|
|
correctly. After the evaluation, the worker outputs a log for every processed
|
|
|
|
|
stage that contains information such as which tests passed and how many
|
|
|
|
|
resources were used.
|
|
|
|
|
|
|
|
|
|
A stage's configuration contains the following (this doesn't yet map exactly to
|
|
|
|
|
a particular configuration file format):
|
|
|
|
|
|
|
|
|
|
- **Name** - a unique string identifier of the stage
|
|
|
|
|
- **Build command** (optional) - used to prepare the submitted files for this
|
|
|
|
@ -34,18 +39,23 @@ contained in its configuration:
|
|
|
|
|
stage
|
|
|
|
|
- **Test command** - used to run one specific test
|
|
|
|
|
- **Test input policy** - how to pass the test input to the program?
|
|
|
|
|
- redirect it to its standard input
|
|
|
|
|
- pass the name of the input file as an argument
|
|
|
|
|
- **redirect** it to its standard input (default)
|
|
|
|
|
- pass the path to an input file as an **argument**
|
|
|
|
|
- **Judge** - which judge should be used to evaluate the solution's output?
|
|
|
|
|
Custom judges can be supplied with the assignment.
|
|
|
|
|
- **Limits** - how much memory, time, etc. can be used when evaluating a test
|
|
|
|
|
- **Error policy** (optional) - what should we do when a test fails?
|
|
|
|
|
- **interrupt** the stage (default)
|
|
|
|
|
- **continue** with the next test
|
|
|
|
|
- **jump** to another stage (TODO cycle detection?)
|
|
|
|
|
- **interrupt** the evaluation (default)
|
|
|
|
|
- **continue** with another **test**
|
|
|
|
|
- **continue** with another **group**
|
|
|
|
|
- **jump** to another stage
|
|
|
|
|
- **Success policy** (optional) - what to do when all tests pass?
|
|
|
|
|
- **jump** to another stage (the next one by default)
|
|
|
|
|
- **end** the evaluation, even if there are still unprocessed stages
|
|
|
|
|
|
|
|
|
|
When jumping between stages, it's only possible to jump forward, so that no
|
|
|
|
|
stage is evaluated multiple times.
|
|
|
|
|
|
|
|
|
|
## Case study
|
|
|
|
|
|
|
|
|
|
We present some of the courses that might use ReCodEx to evaluate homework
|
|
|
|
@ -54,8 +64,8 @@ of stages.
|
|
|
|
|
|
|
|
|
|
### Simple programming exercises
|
|
|
|
|
|
|
|
|
|
For example introductory programming courses such as Programming I or
|
|
|
|
|
Programming in Java.
|
|
|
|
|
For example introductory programming courses such as Programming I or Java
|
|
|
|
|
programming.
|
|
|
|
|
|
|
|
|
|
In the simplest case we only need one stage that builds the program and passes
|
|
|
|
|
the test inputs to its standard input. We will use the C language for this
|
|
|
|
|