restruct

Rewritten-docs.md

@@ -972,6 +972,52 @@ HTTP(S).
 
 ![Communication schema](https://github.com/ReCodEx/wiki/raw/master/images/Backend_Connections.png)
 
+### Job Configuration
+
+@todo: link to execution unit executed by recodex and lets give this chapter a bit of petite description
+
+@todo: discuss what should be in configuration: limits, dependencies, priorities... whatever
+
+#### Supplementary Files
+
+Interesting problem arise with supplementary files (e.g., inputs, sample
+outputs). There are two approaches which can be observed. Supplementary files
+can be downloaded either on the start of the execution or during execution.
+
+If the files are downloaded at the beginning, execution does not really started
+at this point and thus if there are problems with network, worker will find it
+right away and can abort execution without executing single task. Slight
+problems can arise if some of the files needs to have same name (e.g. solution
+assumes that input is `input.txt`), in this scenario downloaded files cannot be
+renamed at the beginning but during execution which is somehow impractical and
+not easily observed by the authors of job configurations.
+
+Second solution of this problem when files are downloaded on the fly has quite
+opposite problem, if there are problems with network, worker will find it during
+execution when for instance almost whole execution is done, this is also not
+ideal solution if we care about burnt hardware resources. On the other hand
+using this approach users have quite advanced control of execution flow and know
+what files exactly are available during execution which is from users
+perspective probably more appealing then the first solution. Based on that,
+downloading of supplementary files using 'fetch' tasks during execution was
+chosen and implemented.
+
+#### Job Variables
+
+As mentioned above worker has job directories but users who are writing and
+managing job configurations do not know where they are (on some particular
+worker) and how they can be accessed and written into configuration. For this
+kind of task we have to introduce some kind of marks or signs which will
+represent particular folders. Marks or signs can have form broadly used
+variables.
+
+Variables can be used everywhere where filesystem paths are used within
+configuration file. This will solve problem with specific worker environment and
+specific hierarchy of directories. Final form of variables is `${...}` where
+triple dot is textual description. This format was used because of special
+dollar sign character which cannot be used within filesystem path, braces are
+there only to border textual description of variable.
+
 ### Broker
 
 The broker is responsible for keeping track of available workers and
@@ -1078,44 +1124,48 @@ kind of in-process messages. The ZeroMQ library which we already use provides
 in-process messages that work on the same principles as network communication,
 which is convenient and solves problems with thread synchronization.
 
-#### Evaluation
+#### Execution of Jobs
 
 At this point we have worker with two internal parts listening one and execution
 one. Implementation of first one is quite straightforward and clear. So let us
 discuss what should be happening in execution subsystem.
 
-After successful arrival of job, worker has to prepare new execution
+After successful arrival of the job from broker to the listening thread, the job
-environment, then solution archive has to be downloaded from fileserver and
+is immediatelly redirected to execution thread. In there worker has to prepare
-extracted. Job configuration is located within these files and loaded into
+new execution environment, solution archive has to be downloaded from fileserver
+and extracted. Job configuration is located within these files and loaded into
 internal structures and executed. After that, results are uploaded back to
 fileserver. These steps are the basic ones which are really necessary for whole
 execution and have to be executed in this precise order.
 
-#### Job Configuration
+The evaluation unit executed by ReCodEx and job configuration were already
-
+discussed above. The conclusion was that jobs containing small tasks will be
-Jobs as work units can quite vary and do completely different things, that means
+used. Particular format of the actual job configuration can be found in 'Job
-configuration and worker has to be prepared for this kind of generality.
+configuration' appendix.  Implementation of parsing and storing these data in
-Configuration and its solution was already discussed above, implementation in
+worker is then quite straightforward.
-worker is then quite also quite straightforward.
 
 Worker has internal structures to which loads and which stores metadata given in
 configuration. Whole job is mapped to job metadata structure and tasks are
 mapped to either external ones or internal ones (internal commands has to be
 defined within worker), both are different whether they are executed in sandbox
-or as internal worker commands.
+or as an internal worker commands.
+
+#### Task Execution Failure
 
 Another division of tasks is by task-type field in configuration. This field can
 have four values: initiation, execution, evaluation and inner. All was discussed
-and described above in configuration analysis. What is important to worker is
+and described above in evaluation unit analysis. What is important to worker is
 how to behave if execution of task with some particular type fails.
 
 There are two possible situations execution fails due to bad user solution or
 due to some internal error. If execution fails on internal error solution cannot
 be declared overly as failed. User should not be punished for bad configuration
-or some network error. This is where task types are useful. Generally
+or some network error. This is where task types are useful.
-initiation, execution and evaluation are tasks which are somehow executing code
+
+Initiation, execution and evaluation are tasks which are usually executing code
 which was given by users who submitted solution of exercise. If this kinds of
 tasks fail it is probably connected with bad user solution and can be evaluated.
+
 But if some inner task fails solution should be re-executed, in best case
 scenario on different worker. That is why if inner task fails it is sent back to
 broker which will reassign job to another worker. More on this subject should be
@@ -1151,45 +1201,6 @@ searching through this system should be easy. In addition if solutions of users
 have access only to evaluation directory then they do not have access to
 unnecessary files which is better for overall security of whole ReCodEx.
 
-#### Job Variables
-
-As mentioned above worker has job directories but users who are writing and
-managing job configurations do not know where they are (on some particular
-worker) and how they can be accessed and written into configuration. For this
-kind of task we have to introduce some kind of marks or signs which will
-represent particular folders. Marks or signs can have form broadly used
-variables.
-
-Variables can be used everywhere where filesystem paths are used within
-configuration file. This will solve problem with specific worker environment and
-specific hierarchy of directories. Final form of variables is `${...}` where
-triple dot is textual description. This format was used because of special
-dollar sign character which cannot be used within filesystem path, braces are
-there only to border textual description of variable.
-
-#### Supplementary Files
-
-Interesting problem is with supplementary files (inputs, sample outputs). There
-are two approaches which can be observed. Supplementary files can be downloaded
-either on the start of the execution or during execution. If the files are
-downloaded at the beginning, execution does not really started at this point and
-if there are problems with network worker will find it right away and can abort
-execution without executing single task. Slight problems can arise if some of
-the files needs to have same name (e.g. solution assumes that input is
-`input.txt`), in this scenario downloaded files cannot be renamed at the
-beginning but during execution which is somehow impractical and not easily
-observed.
-
-Second solution of this problem when files are downloaded on the fly has quite
-opposite problem, if there are problems with network, worker will find it during
-execution when for instance almost whole execution is done, this is also not
-ideal solution if we care about burnt hardware resources. On the other hand
-using this approach users have quite advanced control of execution flow and know
-what files exactly are available during execution which is from users
-perspective probably more appealing then the first solution. Based on that,
-downloading of supplementary files using 'fetch' tasks during execution was
-chosen and implemented.
-
 ### Sandboxing
 
 There are numerous ways how to approach sandboxing on different platforms,