From a8b31f0b612eeccf82f92ee7b4bdebd71d9d3d86 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Sat, 24 Feb 2018 14:27:40 +0100 Subject: [PATCH] move writing job config from user doc --- Job-configuration.md | 154 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) diff --git a/Job-configuration.md b/Job-configuration.md index 4e0aeeb..6eabf58 100644 --- a/Job-configuration.md +++ b/Job-configuration.md @@ -406,6 +406,160 @@ results: ... ``` +## Writing Job Configuration + +To run and evaluate an exercise the backend needs to know the steps how to do +that. This is different for each environment (operation system, programming +language, etc.), so each of the environments needs to have separate +configuration. + +Backend works with a powerful, but quite low level description of simple +connected tasks written in YAML syntax. More about the syntax and general task +overview can be found on [separate +page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned +features was user friendly configuration editor, but due to tight deadline and +team composition it did not make it to the first release. However, writing +configuration in the basic format will be always available and allows users to +use the full expressive power of the system. + +This section walks through creation of job configuration for _hello world_ +exercise. The goal is to compile file _source.c_ and check if it prints `Hello +World!` to the standard output. This is the only test case named **A**. + +The problem can be split into several tasks: + +- compile _source.c_ into _helloworld_ with `/usr/bin/gcc` +- run _helloworld_ and save standard output into _out.txt_ +- fetch predefined output (suppose it is already uploaded to fileserver) with + hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_ +- compare _out.txt_ and _reference.txt_ by `/usr/bin/diff` + +The absolute path of tools can be obtained from system administrator. However, +`/usr/bin/gcc` is location, where the GCC binary is available almost everywhere, +so location of some tools can be (professionally) guessed. + +First, write header of the job to the configuration file. + +```{.yml} +submission: + job-id: hello-word-job + hw-groups: + - group1 +``` + +Basically it means, that the job _hello-world-job_ needs to be run on workers +that belong to the `group_1` hardware group . Reference files are downloaded +from the default location configured in API (such as +`http://localhost:9999/exercises`) if not stated explicitly otherwise. Job +execution log will not be saved to result archive. + +Next the tasks have to be constructed under _tasks_ section. In this demo job, +every task depends only on previous one. The first task has input file +_source.c_ (if submitted by user) already available in working directory, so +just call the GCC. Compilation is run in sandbox as any other external program +and should have relaxed time and memory limits. In this scenario, worker +defaults are used. If compilation fails, the whole job is immediately terminated +(because the _fatal-failure_ bit is set). Because _bound-directories_ option in +sandbox limits section is mostly shared between all tasks, it can be set in +worker configuration instead of job configuration (suppose this for following +tasks). For configuration of workers please contact your administrator. + +Please note that working directory inside sandbox is automatically bounded to +the directory with fetched user source codes and therefore you do not have to +bound it by yourself. Also note that directories inside sandbox can be bound to +different paths, so inside sandbox you have to use special paths. For working +directory inside sandbox you can use ${EVAL_DIR} variable. + +```{.yml} +- task-id: "compilation" + type: "initiation" + fatal-failure: true + cmd: + bin: "/usr/bin/gcc" + args: + - "source.c" + - "-o" + - "helloworld" + sandbox: + name: "isolate" + limits: + - hw-group-id: group1 +``` + +The compiled program is executed with time and memory limit set and the standard +output is redirected to a file. This task depends on _compilation_ task, because +the program cannot be executed without being compiled first. It is important to +mark this task with _execution_ type, so exceeded limits will be reported in +frontend. + +Time and memory limits set directly for a task have higher priority than worker +defaults. One important constraint is, that these limits cannot exceed limits +set by workers. Worker defaults are present as a safety measure so that a +malformed job configuration cannot block the worker forever. Worker default +limits should be reasonably high, like a gigabyte of memory and several hours of +execution time. For exact numbers please contact your administrator. + +It is important to know that if the output of a program (both standard and +error) is redirected to a file, the sandbox disk quotas apply to that file, as +well as the files created directly by the program. In case the outputs are +ignored, they are redirected to `/dev/null`, which means there is no limit on +the output length (as long as the printing fits in the time limit). + +```{.yml} +- task-id: "execution_1" + test-id: "A" + type: "execution" + dependencies: + - compilation + cmd: + bin: "helloworld" + sandbox: + name: "isolate" + stdout: ${EVAL_DIR}/out.txt + limits: + - hw-group-id: group1 + time: 0.5 + memory: 8192 +``` + +Fetch sample solution from file server. Base URL of file server is in the header +of the job configuration, so only the name of required file (its `sha1sum` in +our case) is necessary. + +```{.yml} +- task-id: "fetch_solution_1" + test-id: "A" + dependencies: + - execution + cmd: + bin: "fetch" + args: + - "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b" + - "${SOURCE_DIR}/reference.txt" +``` + +Comparison of results is quite straightforward. It is important to set the task +type to _evaluation_, so that the return code is set to 0 if the program is +correct and 1 otherwise. We do not set our own limits, so the default limits are +used. + +```{.yml} +- task-id: "judge_1" + test-id: "A" + type: "evaluation" + dependencies: + - fetch_solution_1 + cmd: + bin: "/usr/bin/diff" + args: + - "out.txt" + - "reference.txt" + sandbox: + name: "isolate" + limits: + - hw-group-id: group1 +``` +