From 152d8640fb3ebd7889511a0c03f9a9fcc4c7c2e8 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Sun, 1 Jan 2017 20:17:04 +0100 Subject: [PATCH] "Evaluation unit executed on backend" completed --- Rewritten-docs.md | 47 +++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 39 insertions(+), 8 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index ede64e9..f50473b 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -487,11 +487,11 @@ described as well. ### Evaluation unit executed on backend -One of the bigger requests for the new system is to support complex +One of the bigger requests for the new system is to support a complex configuration of execution pipeline. The idea comes from lecturers of Compiler principles class who want to migrate their semi-manual evaluation process to CodEx. Unfortunately, CodEx is not capable of such compilicated exercise setup. -None of evaluation systems we found is capable of such task, so design from +None of evaluation systems we found is can handle such task, so design from scratch is needed. There are two main approaches to design a complex execution configuration. It @@ -528,11 +528,40 @@ is a small example. ![Task serialization](https://github.com/ReCodEx/wiki/raw/master/images/Assignment_overview.png) -@todo: why we need priority and exact order of tasks? - -@todo: division to initiation, execution and evaluation why and what for - -@todo: how to solve problem with specific worker environment, mention internal job variables +The _job root_ task is imaginary single starting point of each job. When the +_CompileA_ task is finished, the _RunAA_ task is started (or _RunAB_, but should +be deterministic by position in configuration file -- tasks stated earlier +should be executed earlier). The task priorities guaranties, that after +_CompileA_ task all dependent tasks are executed before _CompileB_ task (they +have higher priority number). For example this is useful to control which files +are present in a working directory at every moment. To sum up, there are 3 +ordering criteria: dependencies, then priorities and finally position of task in +configuration. Together, they define a unambiguous linear ordering of all tasks. + +For grading there are several important tasks. First, tasks executing submitted +code need to be checked for time and memory limits. Second, outputs of judging +tasks need to be checked for correctness (represented by return value or by data +on standard output) and should not fail on time or memory limits. This division +is transparent for backend, each task is executed the same way. But frontend +must know which tasks from whole job are important and what is their kind. It is +reasonable, to keep this piece of information alongside the tasks in job +configuration, so each task can have a label about its purpose. There are three +categories of tasks: + +- _initiation_ -- setting up the environment, compilling +code, etc.; for users failure means error in their sources which are not +compatible with running it with examination data +- _execution_ -- running the user code with examination data, must not exceed + time and memory limits; for users failure means wrong design, slow data + structures, etc. +- _evaluation_ -- comparing user and examination outputs; for user failure means + that the program does not compute the right results + +Each job is composed of multiple tasks of these types which are semanticaly +grupped into tests. A test can represent one set of examination data for user +code. To mark the groupping, another task label can be used. Each test must have +exactly one _evaluation_ task (to show success or failure to users) and +arbitraty number of tasks with other types. ## Implementation analysis @@ -614,10 +643,12 @@ synchronization and such. At this point we have worker with two internal parts listening one and execution one. Implementation of first one is quite straighforward and clear. So lets discuss what should be happening in execution subsystem. Jobs as work units can quite vary and do completely different things, that means configuration and worker has to be prepared for this kind of generality. Configuration and its solution was already discussed above, implementation in worker is then quite straightforward. Worker has internal structures to which loads and which stores metadata given in configuration. Whole job is mapped to job metadata structure and tasks are mapped to either external ones or internal ones (internal commands has to be defined within worker), both are different whether they are executed in sandbox or as internal worker commands. -@todo: task types: initiation, execution, evaluation and inner... mainly describe what is happening if inner task fails +@todo: task types: initiation, execution, evaluation and inner... mainly describe what is happening if inner task fails; tasks types are mentioned in "Evaluation unit executed on backend" section above, do not duplicate content here @todo: maybe describe folders within execution and what they can be used for? +@todo: how to solve problem with specific worker environment, mention internal job variables + After successful arrival of job, worker has to prepare new execution environment, then solution archive has to be downloaded from fileserver and extracted. Job configuration is located within these files and loaded into internal structures and executed. After that results are uploaded back to fileserver. These steps are the basic ones which are really necessary for whole execution and have to be executed in this precise order. Interesting problem is with supplementary files (inputs, sample outputs). There are two approaches which can be observed. Supplementary files can be downloaded either on the start of the execution or during execution. If the files are downloaded at the beginning execution does not really started at this point and if there are problems with network worker find it right away and can abort execution without executing single task. Slight problems can arise if some of the files needs to have same name (e.g. solution assumes that input is `input.txt`), in this scenario downloaded files cannot be renamed at the beginning but during execution which is somehow impractical and not easily observed. Second solution of this problem when files are downloaded on the fly has quite opposite problem, if there are problems with network worker will find it during execution when for instance almost whole execution is done, this is also not ideal solution if we care about burnt hardware resources. On the other hand using this approach users have quite advanced control of execution flow and know what files exactly are available during execution which is from users perspective probably more appealing then the first solution. Based on that downloading of supplementary files using 'fetch' tasks during execution was chosen and implemented.