|
|
|
|
@ -692,161 +692,6 @@ testWeights:
|
|
|
|
|
d: 100
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Writing Job Configuration
|
|
|
|
|
|
|
|
|
|
To run and evaluate an exercise the backend needs to know the steps how to do
|
|
|
|
|
that. This is different for each environment (operation system, programming
|
|
|
|
|
language, etc.), so each of the environments needs to have separate
|
|
|
|
|
configuration.
|
|
|
|
|
|
|
|
|
|
Backend works with a powerful, but quite low level description of simple
|
|
|
|
|
connected tasks written in YAML syntax. More about the syntax and general task
|
|
|
|
|
overview can be found on [separate
|
|
|
|
|
page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned
|
|
|
|
|
features was user friendly configuration editor, but due to tight deadline and
|
|
|
|
|
team composition it did not make it to the first release. However, writing
|
|
|
|
|
configuration in the basic format will be always available and allows users to
|
|
|
|
|
use the full expressive power of the system.
|
|
|
|
|
|
|
|
|
|
This section walks through creation of job configuration for _hello world_
|
|
|
|
|
exercise. The goal is to compile file _source.c_ and check if it prints `Hello
|
|
|
|
|
World!` to the standard output. This is the only test case named **A**.
|
|
|
|
|
|
|
|
|
|
The problem can be split into several tasks:
|
|
|
|
|
|
|
|
|
|
- compile _source.c_ into _helloworld_ with `/usr/bin/gcc`
|
|
|
|
|
- run _helloworld_ and save standard output into _out.txt_
|
|
|
|
|
- fetch predefined output (suppose it is already uploaded to fileserver) with
|
|
|
|
|
hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_
|
|
|
|
|
- compare _out.txt_ and _reference.txt_ by `/usr/bin/diff`
|
|
|
|
|
|
|
|
|
|
The absolute path of tools can be obtained from system administrator. However,
|
|
|
|
|
`/usr/bin/gcc` is location, where the GCC binary is available almost everywhere,
|
|
|
|
|
so location of some tools can be (professionally) guessed.
|
|
|
|
|
|
|
|
|
|
First, write header of the job to the configuration file.
|
|
|
|
|
|
|
|
|
|
```{.yml}
|
|
|
|
|
submission:
|
|
|
|
|
job-id: hello-word-job
|
|
|
|
|
hw-groups:
|
|
|
|
|
- group1
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Basically it means, that the job _hello-world-job_ needs to be run on workers
|
|
|
|
|
that belong to the `group_1` hardware group . Reference files are downloaded
|
|
|
|
|
from the default location configured in API (such as
|
|
|
|
|
`http://localhost:9999/exercises`) if not stated explicitly otherwise. Job
|
|
|
|
|
execution log will not be saved to result archive.
|
|
|
|
|
|
|
|
|
|
Next the tasks have to be constructed under _tasks_ section. In this demo job,
|
|
|
|
|
every task depends only on previous one. The first task has input file
|
|
|
|
|
_source.c_ (if submitted by user) already available in working directory, so
|
|
|
|
|
just call the GCC. Compilation is run in sandbox as any other external program
|
|
|
|
|
and should have relaxed time and memory limits. In this scenario, worker
|
|
|
|
|
defaults are used. If compilation fails, the whole job is immediately terminated
|
|
|
|
|
(because the _fatal-failure_ bit is set). Because _bound-directories_ option in
|
|
|
|
|
sandbox limits section is mostly shared between all tasks, it can be set in
|
|
|
|
|
worker configuration instead of job configuration (suppose this for following
|
|
|
|
|
tasks). For configuration of workers please contact your administrator.
|
|
|
|
|
|
|
|
|
|
Please note that working directory inside sandbox is automatically bounded to
|
|
|
|
|
the directory with fetched user source codes and therefore you do not have to
|
|
|
|
|
bound it by yourself. Also note that directories inside sandbox can be bound to
|
|
|
|
|
different paths, so inside sandbox you have to use special paths. For working
|
|
|
|
|
directory inside sandbox you can use ${EVAL_DIR} variable.
|
|
|
|
|
|
|
|
|
|
```{.yml}
|
|
|
|
|
- task-id: "compilation"
|
|
|
|
|
type: "initiation"
|
|
|
|
|
fatal-failure: true
|
|
|
|
|
cmd:
|
|
|
|
|
bin: "/usr/bin/gcc"
|
|
|
|
|
args:
|
|
|
|
|
- "source.c"
|
|
|
|
|
- "-o"
|
|
|
|
|
- "helloworld"
|
|
|
|
|
sandbox:
|
|
|
|
|
name: "isolate"
|
|
|
|
|
limits:
|
|
|
|
|
- hw-group-id: group1
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The compiled program is executed with time and memory limit set and the standard
|
|
|
|
|
output is redirected to a file. This task depends on _compilation_ task, because
|
|
|
|
|
the program cannot be executed without being compiled first. It is important to
|
|
|
|
|
mark this task with _execution_ type, so exceeded limits will be reported in
|
|
|
|
|
frontend.
|
|
|
|
|
|
|
|
|
|
Time and memory limits set directly for a task have higher priority than worker
|
|
|
|
|
defaults. One important constraint is, that these limits cannot exceed limits
|
|
|
|
|
set by workers. Worker defaults are present as a safety measure so that a
|
|
|
|
|
malformed job configuration cannot block the worker forever. Worker default
|
|
|
|
|
limits should be reasonably high, like a gigabyte of memory and several hours of
|
|
|
|
|
execution time. For exact numbers please contact your administrator.
|
|
|
|
|
|
|
|
|
|
It is important to know that if the output of a program (both standard and
|
|
|
|
|
error) is redirected to a file, the sandbox disk quotas apply to that file, as
|
|
|
|
|
well as the files created directly by the program. In case the outputs are
|
|
|
|
|
ignored, they are redirected to `/dev/null`, which means there is no limit on
|
|
|
|
|
the output length (as long as the printing fits in the time limit).
|
|
|
|
|
|
|
|
|
|
```{.yml}
|
|
|
|
|
- task-id: "execution_1"
|
|
|
|
|
test-id: "A"
|
|
|
|
|
type: "execution"
|
|
|
|
|
dependencies:
|
|
|
|
|
- compilation
|
|
|
|
|
cmd:
|
|
|
|
|
bin: "helloworld"
|
|
|
|
|
sandbox:
|
|
|
|
|
name: "isolate"
|
|
|
|
|
stdout: ${EVAL_DIR}/out.txt
|
|
|
|
|
limits:
|
|
|
|
|
- hw-group-id: group1
|
|
|
|
|
time: 0.5
|
|
|
|
|
memory: 8192
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Fetch sample solution from file server. Base URL of file server is in the header
|
|
|
|
|
of the job configuration, so only the name of required file (its `sha1sum` in
|
|
|
|
|
our case) is necessary.
|
|
|
|
|
|
|
|
|
|
```{.yml}
|
|
|
|
|
- task-id: "fetch_solution_1"
|
|
|
|
|
test-id: "A"
|
|
|
|
|
dependencies:
|
|
|
|
|
- execution
|
|
|
|
|
cmd:
|
|
|
|
|
bin: "fetch"
|
|
|
|
|
args:
|
|
|
|
|
- "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b"
|
|
|
|
|
- "${SOURCE_DIR}/reference.txt"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Comparison of results is quite straightforward. It is important to set the task
|
|
|
|
|
type to _evaluation_, so that the return code is set to 0 if the program is
|
|
|
|
|
correct and 1 otherwise. We do not set our own limits, so the default limits are
|
|
|
|
|
used.
|
|
|
|
|
|
|
|
|
|
```{.yml}
|
|
|
|
|
- task-id: "judge_1"
|
|
|
|
|
test-id: "A"
|
|
|
|
|
type: "evaluation"
|
|
|
|
|
dependencies:
|
|
|
|
|
- fetch_solution_1
|
|
|
|
|
cmd:
|
|
|
|
|
bin: "/usr/bin/diff"
|
|
|
|
|
args:
|
|
|
|
|
- "out.txt"
|
|
|
|
|
- "reference.txt"
|
|
|
|
|
sandbox:
|
|
|
|
|
name: "isolate"
|
|
|
|
|
limits:
|
|
|
|
|
- hw-group-id: group1
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
<!---
|
|
|
|
|
// vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
|
|
|
|
|
-->
|
|
|
|
|
|
Martin Polanka
7 years ago