Format

Rewritten-docs.md

@@ -699,11 +699,60 @@ ZeroMQ is possible to provide in-process messages working on the same principles
 as network communication which is quite handy and solves problems with threads
 synchronization and such.
 
-At this point we have worker with two internal parts listening one and execution one. Implementation of first one is quite straighforward and clear. So lets discuss what should be happening in execution subsystem. Jobs as work units can quite vary and do completely different things, that means configuration and worker has to be prepared for this kind of generality. Configuration and its solution was already discussed above, implementation in worker is then quite also quite straightforward. Worker has internal structures to which loads and which stores metadata given in configuration. Whole job is mapped to job metadata structure and tasks are mapped to either external ones or internal ones (internal commands has to be defined within worker), both are different whether they are executed in sandbox or as internal worker commands.
+At this point we have worker with two internal parts listening one and execution
-
+one. Implementation of first one is quite straighforward and clear. So lets
-Another division of tasks is by task-type field in configuration. This field can have four values: initiation, execution, evaluation and inner. All was discussed and described above in configuration analysis. What is important to worker is how to behave if execution of task with some particular type fails. There are two possible situations execution fails due to bad user solution or due to some internal error. If execution fails on internal error solution cannot be declared overly as failed. User should not be punished for bad configuration or some network error. This is where task types are useful. Generally initiation, execution and evaluation are tasks which are somehow executing code which was given by users who submitted solution of exercise. If this kinds of tasks fail it is probably connected with bad user solution and can be evaluated. But if some inner task fails solution should be re-executed, in best case scenario on different worker. That is why if inner task fails it is sent back to broker which will reassign job to another worker. More on this subject should be discussed in broker assigning algorithms section.
+discuss what should be happening in execution subsystem. Jobs as work units can
-
+quite vary and do completely different things, that means configuration and
-There is also question about working directory or directories of job, which directories should be used and what for. There is one simple answer on this every job will have only one specified directory which will contain every file with which worker will work in the scope of whole job execution. This is of course nonsense there has to be some logical division. The least which must be done are two folders one for internal temporary files and second one for evaluation. The directory for temporary files is enough to comprehend all kind of internal work with filesystem but only one directory for whole evaluation is somehow not enough. Users solutions are downloaded in form of zip archives so why these should be present during execution or why the results and files which should be uploaded back to fileserver should be cherry picked from the one big directory? The answer is of course another logical division into subfolders. The solution which was chosen at the end is to have folders for downloaded archive, decompressed solution, evaluation directory in which user solution is executed and then folders for temporary files and for results and generally files which should be uploaded back to fileserver with solution results. Of course there has to be hierarchy which separate folders from different workers on the same machines. That is why paths to directories are in format: ${DEFAULT}/${FOLDER}/${WORKER_ID}/${JOB_ID} where default means default working directory of whole worker, folder is particular directory for some purpose (archives, evaluation...). Mentioned division of job directories proved to be flexible and detailed enough, everything is in logical units and where it is supposed to be which means that searching through this system should be easy. In addition if solutions of users have access only to evaluation directory then they do not have access to unnecessary files which is better for overall security of whole ReCodEx.
+worker has to be prepared for this kind of generality. Configuration and its
+solution was already discussed above, implementation in worker is then quite
+also quite straightforward. Worker has internal structures to which loads and
+which stores metadata given in configuration. Whole job is mapped to job
+metadata structure and tasks are mapped to either external ones or internal ones
+(internal commands has to be defined within worker), both are different whether
+they are executed in sandbox or as internal worker commands.
+
+Another division of tasks is by task-type field in configuration. This field can
+have four values: initiation, execution, evaluation and inner. All was discussed
+and described above in configuration analysis. What is important to worker is
+how to behave if execution of task with some particular type fails. There are
+two possible situations execution fails due to bad user solution or due to some
+internal error. If execution fails on internal error solution cannot be declared
+overly as failed. User should not be punished for bad configuration or some
+network error. This is where task types are useful. Generally initiation,
+execution and evaluation are tasks which are somehow executing code which was
+given by users who submitted solution of exercise. If this kinds of tasks fail
+it is probably connected with bad user solution and can be evaluated. But if
+some inner task fails solution should be re-executed, in best case scenario on
+different worker. That is why if inner task fails it is sent back to broker
+which will reassign job to another worker. More on this subject should be
+discussed in broker assigning algorithms section.
+
+There is also question about working directory or directories of job, which
+directories should be used and what for. There is one simple answer on this
+every job will have only one specified directory which will contain every file
+with which worker will work in the scope of whole job execution. This is of
+course nonsense there has to be some logical division. The least which must be
+done are two folders one for internal temporary files and second one for
+evaluation. The directory for temporary files is enough to comprehend all kind
+of internal work with filesystem but only one directory for whole evaluation is
+somehow not enough. Users solutions are downloaded in form of zip archives so
+why these should be present during execution or why the results and files which
+should be uploaded back to fileserver should be cherry picked from the one big
+directory? The answer is of course another logical division into subfolders. The
+solution which was chosen at the end is to have folders for downloaded archive,
+decompressed solution, evaluation directory in which user solution is executed
+and then folders for temporary files and for results and generally files which
+should be uploaded back to fileserver with solution results. Of course there has
+to be hierarchy which separate folders from different workers on the same
+machines. That is why paths to directories are in format:
+${DEFAULT}/${FOLDER}/${WORKER_ID}/${JOB_ID} where default means default working
+directory of whole worker, folder is particular directory for some purpose
+(archives, evaluation...). Mentioned division of job directories proved to be
+flexible and detailed enough, everything is in logical units and where it is
+supposed to be which means that searching through this system should be easy. In
+addition if solutions of users have access only to evaluation directory then
+they do not have access to unnecessary files which is better for overall
+security of whole ReCodEx.
 
 As we discovered above worker has job directories but users who are writing and
 managing job configurations do not know where they are (on some particular
@@ -720,9 +769,31 @@ description of variable.
 
 #### Evaluation
 
-After successful arrival of job, worker has to prepare new execution environment, then solution archive has to be downloaded from fileserver and extracted. Job configuration is located within these files and loaded into internal structures and executed. After that results are uploaded back to fileserver. These steps are the basic ones which are really necessary for whole execution and have to be executed in this precise order.
+After successful arrival of job, worker has to prepare new execution
-
+environment, then solution archive has to be downloaded from fileserver and
-Interesting problem is with supplementary files (inputs, sample outputs). There are two approaches which can be observed. Supplementary files can be downloaded either on the start of the execution or during execution. If the files are downloaded at the beginning execution does not really started at this point and if there are problems with network worker find it right away and can abort execution without executing single task. Slight problems can arise if some of the files needs to have same name (e.g. solution assumes that input is `input.txt`), in this scenario downloaded files cannot be renamed at the beginning but during execution which is somehow impractical and not easily observed. Second solution of this problem when files are downloaded on the fly has quite opposite problem, if there are problems with network worker will find it during execution when for instance almost whole execution is done, this is also not ideal solution if we care about burnt hardware resources. On the other hand using this approach users have quite advanced control of execution flow and know what files exactly are available during execution which is from users perspective probably more appealing then the first solution. Based on that downloading of supplementary files using 'fetch' tasks during execution was chosen and implemented.
+extracted. Job configuration is located within these files and loaded into
+internal structures and executed. After that results are uploaded back to
+fileserver. These steps are the basic ones which are really necessary for whole
+execution and have to be executed in this precise order.
+
+Interesting problem is with supplementary files (inputs, sample outputs). There
+are two approaches which can be observed. Supplementary files can be downloaded
+either on the start of the execution or during execution. If the files are
+downloaded at the beginning execution does not really started at this point and
+if there are problems with network worker find it right away and can abort
+execution without executing single task. Slight problems can arise if some of
+the files needs to have same name (e.g. solution assumes that input is
+`input.txt`), in this scenario downloaded files cannot be renamed at the
+beginning but during execution which is somehow impractical and not easily
+observed. Second solution of this problem when files are downloaded on the fly
+has quite opposite problem, if there are problems with network worker will find
+it during execution when for instance almost whole execution is done, this is
+also not ideal solution if we care about burnt hardware resources. On the other
+hand using this approach users have quite advanced control of execution flow and
+know what files exactly are available during execution which is from users
+perspective probably more appealing then the first solution. Based on that
+downloading of supplementary files using 'fetch' tasks during execution was
+chosen and implemented.
 
 #### Caching mechanism
 
@@ -748,7 +819,19 @@ within cron which is able to delete files which were unused for some time.
 Together with worker fetching feature cleaner completes machine specific caching
 system.
 
-Cleaner as mentioned is simple script which is executed regularly as cron job. If there is caching system like it was introduced in paragraph above there are little possibilities how cleaner should be implemented. On various filesystems there is usually support for two  particular timestamps, `last access time` and `last modification time`. Files in cache are once downloaded and then just copied, this means that last modification time is set only once on creation of file and last access time should be set every time on copy. This imply last access time is what is needed here. But last modification time is widely used by operating systems, on the other hand last access time is not by default. More on this subject can be found [here](https://en.wikipedia.org/wiki/Stat_%28system_call%29#Criticism_of_atime). For proper cleaner functionality filesystem which is used by worker for caching has to have last access time for files enabled.
+Cleaner as mentioned is simple script which is executed regularly as cron job.
+If there is caching system like it was introduced in paragraph above there are
+little possibilities how cleaner should be implemented. On various filesystems
+there is usually support for two  particular timestamps, `last access time` and
+`last modification time`. Files in cache are once downloaded and then just
+copied, this means that last modification time is set only once on creation of
+file and last access time should be set every time on copy. This imply last
+access time is what is needed here. But last modification time is widely used by
+operating systems, on the other hand last access time is not by default. More on
+this subject can be found
+[here](https://en.wikipedia.org/wiki/Stat_%28system_call%29#Criticism_of_atime).
+For proper cleaner functionality filesystem which is used by worker for caching
+has to have last access time for files enabled.
 
 Having cleaner as separated component and caching itself handled in worker is
 kind of blurry and is not clearly observable that it works without any race
@@ -775,11 +858,21 @@ delete files. Implementation in cleaner follows:
 - there is a loop going through all files and even directories in specified
   cache folder
 	- last access time of file or folder is detected
-	- last access time is subtracted from reference timestamp into difference
+	- last access time is subtracted from reference timestamp into
-	- difference is compared against specified maximal file age, if difference
+	  difference
-	  is greater, file or folder is deleted
+	- difference is compared against specified maximal file age, if
-
+	  difference is greater, file or folder is deleted
-Previous description implies that there is gap between detection of last access time and deleting file within cleaner. In the gap there can be worker which will access file and the file is anyway deleted but this is fine, file is deleted but worker has it copied. Another problem can be with two workers downloading the same file, but this is also not a problem file is firstly downloaded to working folder and after that copied to cache. And even if something else unexpectedly fails and because of that fetch task will fail during execution even that should be fine. Because fetch tasks should have 'inner' task type which implies that fail in this task will stop all execution and job will be reassigned to another worker. It should be like the last salvation in case everything else goes wrong.
+
+Previous description implies that there is gap between detection of last access
+time and deleting file within cleaner. In the gap there can be worker which will
+access file and the file is anyway deleted but this is fine, file is deleted but
+worker has it copied. Another problem can be with two workers downloading the
+same file, but this is also not a problem file is firstly downloaded to working
+folder and after that copied to cache. And even if something else unexpectedly
+fails and because of that fetch task will fail during execution even that should
+be fine. Because fetch tasks should have 'inner' task type which implies that
+fail in this task will stop all execution and job will be reassigned to another
+worker. It should be like the last salvation in case everything else goes wrong.
 
 #### Sandboxing
 
@@ -788,15 +881,53 @@ describing all possible approaches is out of scope of this document. Instead of
 that have a look at some of the features which are certainly needed for ReCodEx
 and propose some particular sandboxes implementations on linux or Windows.
 
-General purpose of sandbox is safely execute software in any form, from scripts to binaries. Various sandboxes differ in how safely are they and what limiting features they have. Ideal situation is that sandbox will have numerous options and corresponding features which will allow administrators to setup environment as they like and which will not allow user programs to somehow damage executing machine in any way possible.
+General purpose of sandbox is safely execute software in any form, from scripts
-
+to binaries. Various sandboxes differ in how safely are they and what limiting
-For ReCodEx and its evaluation there is need for at least these features: execution time and memory limitation, disk operations limit, disk accessibility restrictions and network restrictions. All these features if combined and implemented well are giving pretty safe sandbox which can be used for all kinds of users solutions and should be able to restrict and stop any standard way of attacks or errors.
+features they have. Ideal situation is that sandbox will have numerous options
-
+and corresponding features which will allow administrators to setup environment
-Linux systems have quite extent support of sandboxing in kernel, there were introduced and implemented kernel namespaces and cgroups which combined can limit hardware resources (cpu, memory) and separate executing program into its own namespace (pid, network). These two features comply sandbox requirement for ReCodEx so there were two options, either find existing solution or implement new one. Luckily existing solution was found and its name is **isolate**. Isolate does not use all possible kernel features but only subset which is still enough to be used by ReCodEx.
+as they like and which will not allow user programs to somehow damage executing
-
+machine in any way possible.
-The opposite situation is in Windows world, there is limited support in its kernel which makes sandboxing a bit trickier. Windows kernel only has ways how to restrict privileges of a process through restriction of internal access tokens. Monitoring of hardware resources is not possible but used resources can be obtained through newly created job objects. But find sandbox which can do all things needed for ReCodEx seems to be impossible. There are numerous sandboxes for Windows but they all are focused on different things in a lot of cases they serves as safe environment for malicious programs, viruses in particular. Or they are designed as a separate filesystem namespace for installing a lot of temporarily used programs. From all these we can mention Sandboxie, Comodo Internet Security, Cuckoo sandbox and many others. None of these is fitted as sandbox solution for ReCodEx. With this being said we can safely state that designing and implementing new general sandbox for Windows is out of scope of this project.
+
-
+For ReCodEx and its evaluation there is need for at least these features:
-New general sandbox for Windows is out of bussiness but what about more specialized solution used for instance only for C#. CLR as a virtual machine and runtime environment has a pretty good security support for restrictions and separation which is also transfered to C#. This makes it quite easy to implement simple sandbox within C# but suprisingly there cannot be found some well known general purpose implementations. As said in previous paragraph implementing our own solution is out of scope of project there is simple not enough time. But C# sandbox is quite good topic for another project for example semestral project for C# course so it might be written and integrated in future.
+execution time and memory limitation, disk operations limit, disk accessibility
+restrictions and network restrictions. All these features if combined and
+implemented well are giving pretty safe sandbox which can be used for all kinds
+of users solutions and should be able to restrict and stop any standard way of
+attacks or errors.
+
+Linux systems have quite extent support of sandboxing in kernel, there were
+introduced and implemented kernel namespaces and cgroups which combined can
+limit hardware resources (cpu, memory) and separate executing program into its
+own namespace (pid, network). These two features comply sandbox requirement for
+ReCodEx so there were two options, either find existing solution or implement
+new one. Luckily existing solution was found and its name is **isolate**.
+Isolate does not use all possible kernel features but only subset which is still
+enough to be used by ReCodEx.
+
+The opposite situation is in Windows world, there is limited support in its
+kernel which makes sandboxing a bit trickier. Windows kernel only has ways how
+to restrict privileges of a process through restriction of internal access
+tokens. Monitoring of hardware resources is not possible but used resources can
+be obtained through newly created job objects. But find sandbox which can do all
+things needed for ReCodEx seems to be impossible. There are numerous sandboxes
+for Windows but they all are focused on different things in a lot of cases they
+serves as safe environment for malicious programs, viruses in particular. Or
+they are designed as a separate filesystem namespace for installing a lot of
+temporarily used programs. From all these we can mention Sandboxie, Comodo
+Internet Security, Cuckoo sandbox and many others. None of these is fitted as
+sandbox solution for ReCodEx. With this being said we can safely state that
+designing and implementing new general sandbox for Windows is out of scope of
+this project.
+
+New general sandbox for Windows is out of bussiness but what about more
+specialized solution used for instance only for C#. CLR as a virtual machine and
+runtime environment has a pretty good security support for restrictions and
+separation which is also transfered to C#. This makes it quite easy to implement
+simple sandbox within C# but suprisingly there cannot be found some well known
+general purpose implementations. As said in previous paragraph implementing our
+own solution is out of scope of project there is simple not enough time. But C#
+sandbox is quite good topic for another project for example semestral project
+for C# course so it might be written and integrated in future.
 
 ### Fileserver