diff --git a/Assignments.md b/Assignments.md index 3254dfd..962238d 100644 --- a/Assignments.md +++ b/Assignments.md @@ -24,8 +24,13 @@ whitespace characters between words of the output. ### Stages A stage is a logical unit of the testing process. It specifies how to do a step -in the build process and how to test if it behaves correctly. The following is -contained in its configuration: +in the build process and how to test if the student's submission behaves +correctly. After the evaluation, the worker outputs a log for every processed +stage that contains information such as which tests passed and how many +resources were used. + +A stage's configuration contains the following (this doesn't yet map exactly to +a particular configuration file format): - **Name** - a unique string identifier of the stage - **Build command** (optional) - used to prepare the submitted files for this @@ -34,18 +39,23 @@ contained in its configuration: stage - **Test command** - used to run one specific test - **Test input policy** - how to pass the test input to the program? - - redirect it to its standard input - - pass the name of the input file as an argument + - **redirect** it to its standard input (default) + - pass the path to an input file as an **argument** - **Judge** - which judge should be used to evaluate the solution's output? Custom judges can be supplied with the assignment. +- **Limits** - how much memory, time, etc. can be used when evaluating a test - **Error policy** (optional) - what should we do when a test fails? - - **interrupt** the stage (default) - - **continue** with the next test - - **jump** to another stage (TODO cycle detection?) + - **interrupt** the evaluation (default) + - **continue** with another **test** + - **continue** with another **group** + - **jump** to another stage - **Success policy** (optional) - what to do when all tests pass? - **jump** to another stage (the next one by default) - **end** the evaluation, even if there are still unprocessed stages +When jumping between stages, it's only possible to jump forward, so that no +stage is evaluated multiple times. + ## Case study We present some of the courses that might use ReCodEx to evaluate homework @@ -54,8 +64,8 @@ of stages. ### Simple programming exercises -For example introductory programming courses such as Programming I or -Programming in Java. +For example introductory programming courses such as Programming I or Java +programming. In the simplest case we only need one stage that builds the program and passes the test inputs to its standard input. We will use the C language for this