You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
342 lines
13 KiB
Markdown
342 lines
13 KiB
Markdown
8 years ago
|
# Job configuration
|
||
|
|
||
|
Following description will cover configuration as seen from API point of view,
|
||
|
worker may have some optional items mandatory (they are filled by API
|
||
|
automatically). Bold items in lists describing the values are mandatory, italic
|
||
|
ones are optional.
|
||
|
|
||
|
## Configuration items
|
||
|
|
||
|
- **submission** -- information about this particular submission
|
||
|
- **job-id** -- textual ID which should be unique in whole recodex
|
||
|
- _file-collector_ -- address from which fetch tasks will download data (API
|
||
|
will fill)
|
||
|
- _log_ -- default is false, can be omitted, determines whether job
|
||
|
execution will be logged into one shared log
|
||
|
- **hw-groups** -- list of hardware groups for which are specified limits in
|
||
|
this configuration
|
||
|
- **tasks** -- list (not map) of individual tasks
|
||
|
- **task-id** -- unique identifier of task in scope of one submission
|
||
|
- _priority_ -- higher number, higher priority, defaults to 1
|
||
|
- _fatal-failure_ -- if true, than execution of whole job will be stopped
|
||
|
after failing of this one, defaults to false
|
||
|
- _dependencies_ -- list of dependencies which have to be fulfilled before
|
||
|
this task, can be omitted if there is no dependencies
|
||
|
- **cmd** -- description of command which will be executed
|
||
|
- **bin** -- the binary itself (full path of external command or name of
|
||
|
internal task)
|
||
|
- _args_ -- list of arguments which will be sent into execution unit
|
||
|
- _test-id_ -- ID of the test this task is part of -- must be specified for
|
||
|
tasks which the particular test's result depends on
|
||
|
- _type_ -- type of the task, can be omitted, default value is _inner_ --
|
||
|
possible values are: _inner_, _initiation_, _execution_, _evaluation_.
|
||
|
Each logical test must contain 0 or more _initiation_ tasks, at least one
|
||
|
task of type _execution_ (time and memory limits exceeded are presentet to
|
||
|
user) and exactly one of type _evaluation_ (typicaly judge). _Inner_ task
|
||
|
type is mainly for internal tasks, but can be used for external tasks,
|
||
|
which are not part of any test.
|
||
|
- _sandbox_ -- wrapper for external tasks which will run in sandbox, if
|
||
|
defined task is automatically external
|
||
|
- **name** -- name of used sandbox
|
||
|
- _stdin_ -- file to which standard input will be redirected, can be
|
||
|
omitted
|
||
|
- _stdout_ -- file to which standard output will be redirected, can be
|
||
|
omitted
|
||
|
- _stderr_ -- file to which error output will be redirected, can be
|
||
|
omitted
|
||
|
- _limits_ -- list of limits which can be passed to sandbox, can be
|
||
|
omitted, in that case defaults will be used
|
||
|
- **hw-group-id** -- determines specific limits for specific
|
||
|
machines
|
||
|
- _time_ -- time of execution in second
|
||
|
- _wall-time_ -- wall time in seconds
|
||
|
- _extra-time_ -- extra time which will be added to execution
|
||
|
- _stack-size_ -- size of stack of executed program in kilobytes
|
||
|
- _memory_ -- overall memory limit for application in kilobytes
|
||
|
- _parallel_ -- integral number of processes which can run
|
||
|
simultaneously, time and memory limits are merged from all
|
||
|
potential processes/threads
|
||
|
- _disk-size_ -- size of all IO operations from/to files in
|
||
|
kilobytes
|
||
|
- _disk-files_ -- number of files which can be opened
|
||
|
- _environ-variable_ -- wrapper for map of environmental variables,
|
||
|
union with default worker configuration
|
||
|
- _chdir_ -- this will be working directory of executed application
|
||
|
- _bound-directories_ -- list of structures representing directories
|
||
|
which will be visible inside sandbox, union with default worker
|
||
|
configuration. Contains 3 suboptions: **src** -- source pointing
|
||
|
to actual system directory, **dst** -- destination inside sandbox
|
||
|
which can have its own filesystem binding and **mode** --
|
||
|
determines connection mode of specified directory, one of values:
|
||
|
RW (allow read-write access), NOEXEC (disallow execution of
|
||
|
binaries), FS (mount device-less filesystem like `/proc`), MAYBE
|
||
|
(silently ignore the rule if the bound directory does not exist),
|
||
|
DEV (allow access to character and block devices).
|
||
|
|
||
|
|
||
|
## Configuration example
|
||
|
|
||
|
This configuration example serves only for demonstration purposes. Some items
|
||
|
can be omitted and defaults from worker configuration will be used.
|
||
|
|
||
|
```{.yml}
|
||
|
---
|
||
|
submission: # happy hippoes fence
|
||
|
job-id: hippoes
|
||
|
file-collector: http://localhost:9999/exercises
|
||
|
log: true
|
||
|
hw-groups:
|
||
|
- group1
|
||
|
tasks:
|
||
|
- task-id: "compilation"
|
||
|
priority: 2
|
||
|
fatal-failure: true
|
||
|
cmd:
|
||
|
bin: "/usr/bin/gcc"
|
||
|
args:
|
||
|
- "solution.c"
|
||
|
- "-o"
|
||
|
- "a.out"
|
||
|
sandbox:
|
||
|
name: "isolate"
|
||
|
limits:
|
||
|
- hw-group-id: group1
|
||
|
parallel: 0
|
||
|
chdir: ${EVAL_DIR}
|
||
|
bound-directories:
|
||
|
- src: ${SOURCE_DIR}
|
||
|
dst: ${EVAL_DIR}
|
||
|
mode: RW
|
||
|
- task-id: "fetch_test_1"
|
||
|
priority: 4
|
||
|
fatal-failure: false
|
||
|
dependencies:
|
||
|
- compilation
|
||
|
cmd:
|
||
|
bin: "fetch"
|
||
|
args:
|
||
|
- "1.in"
|
||
|
- "${SOURCE_DIR}/kuly.in"
|
||
|
- task-id: "evaluation_test_1"
|
||
|
priority: 5
|
||
|
fatal-failure: false
|
||
|
dependencies:
|
||
|
- fetch_test_1
|
||
|
cmd:
|
||
|
bin: "a.out"
|
||
|
sandbox:
|
||
|
name: "isolate"
|
||
|
limits:
|
||
|
- hw-group-id: group1
|
||
|
time: 0.5
|
||
|
memory: 8192
|
||
|
chdir: ${EVAL_DIR}
|
||
|
bound-directories:
|
||
|
- src: ${SOURCE_DIR}
|
||
|
dst: ${EVAL_DIR}
|
||
|
mode: RW
|
||
|
- task-id: "fetch_test_solution_1"
|
||
|
priority: 6
|
||
|
fatal-failure: false
|
||
|
dependencies:
|
||
|
- evaluation_test_1
|
||
|
cmd:
|
||
|
bin: "fetch"
|
||
|
args:
|
||
|
- "1.out"
|
||
|
- "${SOURCE_DIR}/1.out"
|
||
|
- task-id: "judging_test_1"
|
||
|
priority: 7
|
||
|
fatal-failure: false
|
||
|
dependencies:
|
||
|
- fetch_test_solution_1
|
||
|
cmd:
|
||
|
bin: "${JUDGES_DIR}/recodex-judge-normal"
|
||
|
args:
|
||
|
- "1.out"
|
||
|
- "plot.out"
|
||
|
sandbox:
|
||
|
name: "isolate"
|
||
|
limits:
|
||
|
- hw-group-id: group1
|
||
|
parallel: 0
|
||
|
chdir: ${EVAL_DIR}
|
||
|
bound-directories:
|
||
|
- src: ${SOURCE_DIR}
|
||
|
dst: ${EVAL_DIR}
|
||
|
mode: RW
|
||
|
- task-id: "rm_junk_test_1"
|
||
|
priority: 8
|
||
|
fatal-failure: false
|
||
|
dependencies:
|
||
|
- judging_test_1
|
||
|
cmd:
|
||
|
bin: "rm"
|
||
|
args:
|
||
|
- "${SOURCE_DIR}/kuly.in"
|
||
|
- "${SOURCE_DIR}/plot.out"
|
||
|
- "${SOURCE_DIR}/1.out"
|
||
|
...
|
||
|
```
|
||
|
|
||
|
|
||
|
## Job variables
|
||
|
|
||
|
Because frontend does not know which worker gets the job, its necessary to be a
|
||
|
little general in configuration file. This means that some worker specific
|
||
|
things has to be transparent. Good example of this is that some (evaluation)
|
||
|
directories may be placed differently across all workers. To provide a solution,
|
||
|
variables were established. There are of course some restrictions where
|
||
|
variables can be used. Basically whenever filesystem paths can be used,
|
||
|
variables can be used.
|
||
|
|
||
|
Usage of variables in configuration is simple and kind of shell-like. Name of
|
||
|
variable is put inside braces which are preceded with dollar sign. Real usage is
|
||
|
then something like this: ${VAR}. There should be no quotes or apostrophes
|
||
|
around variable name, just simple text in braces. Parsing is simple and whenever
|
||
|
there is dollar sign with braces job execution unit automatically assumes that
|
||
|
this is a variable, so there is no chance to have this kind of substring
|
||
|
anywhere else.
|
||
|
|
||
|
List of usable variables in job configuration:
|
||
|
|
||
|
- **WORKER_ID** -- integral identification of worker, unique on server
|
||
|
- **JOB_ID** -- identification of this job
|
||
|
- **SOURCE_DIR** -- directory where source codes of job are stored
|
||
|
- **EVAL_DIR** -- evaluation directory which should point inside sandbox. Note,
|
||
|
that some existing directory must be bound inside sanbox under **EVAL_DIR**
|
||
|
name using _bound-directories_ directive inside limits section.
|
||
|
- **RESULT_DIR** -- results from job can be copied here, but only with internal
|
||
|
task
|
||
|
- **TEMP_DIR** -- general temp directory which is not dependent on operating
|
||
|
system
|
||
|
- **JUDGES_DIR** -- directory in which judges are stored (outside sandbox)
|
||
|
|
||
|
## Directories and Files
|
||
|
|
||
|
For each job execution unique directory structure is created. Job is not
|
||
|
restricted to use only specified directories (tasks can do whatever is allowed
|
||
|
on system), but it is advised to use them inside a job. Following directories
|
||
|
are created under working directory of the worker for a job execution. This
|
||
|
directory is configurable and can be the same for multiple worker instances.
|
||
|
|
||
|
- `downloads/${WORKER_ID}/${JOB_ID}` -- where the downloaded archive is saved
|
||
|
- `submission/${WORKER_ID}/${JOB_ID}` -- decompressed submission is stored here
|
||
|
- `eval/${WORKER_ID}/${JOB_ID}` -- this directory is accessible in job
|
||
|
configuration using variables and all execution should happen here
|
||
|
- `temp/${WORKER_ID}/${JOB_ID}` -- directory where all sort of temporary files
|
||
|
can be stored
|
||
|
- `results/${WORKER_ID}/${JOB_ID}` -- again accessible directory from job
|
||
|
configuration which is used to store all files which will be upload on
|
||
|
fileserver, usually there will be only yaml result file and optionally log,
|
||
|
every other file has to be copied here explicitly from job
|
||
|
|
||
|
## ReCodEx judges
|
||
|
|
||
|
Below is list of judges which are packed with ReCodEx project.
|
||
|
|
||
|
- **recodex-judge-normal** is base judge used by most of exercises. This judge
|
||
|
compares two text files. It compares only text tokens regardless on amount of
|
||
|
whitespace between them.
|
||
|
```
|
||
|
Usage: recodex-judge-normal [-r | -n | -rn] <file1> <file2>
|
||
|
```
|
||
|
- file1 and file2 are paths to files that will be compared
|
||
|
- switch options `-r` and `-n` can be specified as a 1st optional argument.
|
||
|
- `-n` judge will treat newlines as ordinary whitespace (it will ignore
|
||
|
line breaking)
|
||
|
- `-r` judge will treat tokens as real numbers and compares them
|
||
|
accordingly (with some amount of error)
|
||
|
|
||
|
- **recodex-judge-filter** can be used for preprocess output files before real
|
||
|
judging. This judge filters C-like comments from a text file. The comment
|
||
|
starts with double slash sequence (`//`) and finishes with newline. If the
|
||
|
comment takes whole line, then whole line is filtered.
|
||
|
```
|
||
|
Usage: recodex-judge-filter [inputFile [outputFile]]
|
||
|
```
|
||
|
- if `outputFile` is ommited, std. output is used instead.
|
||
|
- if both files are ommited, application uses std. input and output.
|
||
|
|
||
|
- **recodex-judge-shuffle** is for judging results with semantics of set, where
|
||
|
ordering is not important. This judge compares two text files and returns 0
|
||
|
if they matches (and 1 otherwise). Two files are compared with no regards for
|
||
|
whitespace (whitespace acts just like token delimiter).
|
||
|
```
|
||
|
Usage: recodex-judge-shuffle [-[n][i][r]] <file1> <file2>
|
||
|
```
|
||
|
- `-n` ignore newlines (newline is considered only a whitespace)
|
||
|
- `-i` ignore items order on the row (tokens on each row may be permutated)
|
||
|
- `-r` ignore order of rows (rows may be permutated); this option has no
|
||
|
effect when `-n` is used
|
||
|
|
||
|
|
||
|
## Results
|
||
|
|
||
|
Results of tasks are sent back in YAML format compressed into archive. This
|
||
|
archive can contain further files, such as job logging information and files
|
||
|
which were explicitly copied into results directory. Results file contains job
|
||
|
identification and results of individual tasks.
|
||
|
|
||
|
### Results items
|
||
|
|
||
|
- **job-id** -- identification of job to which this results belongs
|
||
|
- **hw-group** -- Hardware group identifier of worker which performed the
|
||
|
evaluation
|
||
|
- _error_message_ -- present only if whole execution failed and none of tasks
|
||
|
were executed
|
||
|
- **results** -- list of tasks results
|
||
|
- **task-id** -- unique identification of task in scope of this job
|
||
|
- **status** -- three states: OK (execution of task was successful;
|
||
|
sandboxed program could be killed, but sandbox exited normally), FAILED
|
||
|
(error while executing task), SKIPPED (execution of task was skipped)
|
||
|
- _error_message_ -- defined only in internal tasks on failure
|
||
|
- _sandbox_results_ -- if defined than this task was external and was run in
|
||
|
sandbox
|
||
|
- **exitcode** -- integer which executed program gave on exit
|
||
|
- **time** -- time in seconds in which program exited
|
||
|
- **wall-time** -- wall time in seconds
|
||
|
- **memory** -- how much memory program used in kilobytes
|
||
|
- **max-rss** -- maximum resident set size used in kilobytes
|
||
|
- **status** -- two letter status code: OK (success), RE (runtime
|
||
|
error), SG (program died on signal), TO (timed out), XX (internal
|
||
|
error of the sandbox)
|
||
|
- **exitsig** -- description of exit signal
|
||
|
- **killed** -- boolean determining if program exited correctly or was
|
||
|
killed
|
||
|
- **message** -- status message on failure
|
||
|
|
||
|
### Example result file
|
||
|
|
||
|
```{.yml}
|
||
|
---
|
||
|
job-id: 5
|
||
|
hw-group: "group1"
|
||
|
results:
|
||
|
- task-id: compile1
|
||
|
status: OK
|
||
|
sandbox_results:
|
||
|
exitcode: 0
|
||
|
time: 5
|
||
|
wall-time: 5
|
||
|
memory: 50000
|
||
|
max-rss: 50000
|
||
|
status: RE
|
||
|
exitsig: 1
|
||
|
killed: true
|
||
|
message: "Time limit exceeded"
|
||
|
- task-id: eval1
|
||
|
status: FAILED
|
||
|
error_message: "Task failed, something very bad happend!"
|
||
|
.
|
||
|
.
|
||
|
.
|
||
|
...
|
||
|
```
|
||
|
|
||
|
|
||
|
<!---
|
||
|
// vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
|
||
|
-->
|
||
|
|