@ -699,11 +699,60 @@ ZeroMQ is possible to provide in-process messages working on the same principles
as network communication which is quite handy and solves problems with threads
synchronization and such.
At this point we have worker with two internal parts listening one and execution one. Implementation of first one is quite straighforward and clear. So lets discuss what should be happening in execution subsystem. Jobs as work units can quite vary and do completely different things, that means configuration and worker has to be prepared for this kind of generality. Configuration and its solution was already discussed above, implementation in worker is then quite also quite straightforward. Worker has internal structures to which loads and which stores metadata given in configuration. Whole job is mapped to job metadata structure and tasks are mapped to either external ones or internal ones (internal commands has to be defined within worker), both are different whether they are executed in sandbox or as internal worker commands.
Another division of tasks is by task-type field in configuration. This field can have four values: initiation, execution, evaluation and inner. All was discussed and described above in configuration analysis. What is important to worker is how to behave if execution of task with some particular type fails. There are two possible situations execution fails due to bad user solution or due to some internal error. If execution fails on internal error solution cannot be declared overly as failed. User should not be punished for bad configuration or some network error. This is where task types are useful. Generally initiation, execution and evaluation are tasks which are somehow executing code which was given by users who submitted solution of exercise. If this kinds of tasks fail it is probably connected with bad user solution and can be evaluated. But if some inner task fails solution should be re-executed, in best case scenario on different worker. That is why if inner task fails it is sent back to broker which will reassign job to another worker. More on this subject should be discussed in broker assigning algorithms section.
There is also question about working directory or directories of job, which directories should be used and what for. There is one simple answer on this every job will have only one specified directory which will contain every file with which worker will work in the scope of whole job execution. This is of course nonsense there has to be some logical division. The least which must be done are two folders one for internal temporary files and second one for evaluation. The directory for temporary files is enough to comprehend all kind of internal work with filesystem but only one directory for whole evaluation is somehow not enough. Users solutions are downloaded in form of zip archives so why these should be present during execution or why the results and files which should be uploaded back to fileserver should be cherry picked from the one big directory? The answer is of course another logical division into subfolders. The solution which was chosen at the end is to have folders for downloaded archive, decompressed solution, evaluation directory in which user solution is executed and then folders for temporary files and for results and generally files which should be uploaded back to fileserver with solution results. Of course there has to be hierarchy which separate folders from different workers on the same machines. That is why paths to directories are in format: ${DEFAULT}/${FOLDER}/${WORKER_ID}/${JOB_ID} where default means default working directory of whole worker, folder is particular directory for some purpose (archives, evaluation...). Mentioned division of job directories proved to be flexible and detailed enough, everything is in logical units and where it is supposed to be which means that searching through this system should be easy. In addition if solutions of users have access only to evaluation directory then they do not have access to unnecessary files which is better for overall security of whole ReCodEx.
At this point we have worker with two internal parts listening one and execution
one. Implementation of first one is quite straighforward and clear. So lets
discuss what should be happening in execution subsystem. Jobs as work units can
quite vary and do completely different things, that means configuration and
worker has to be prepared for this kind of generality. Configuration and its
solution was already discussed above, implementation in worker is then quite
also quite straightforward. Worker has internal structures to which loads and
which stores metadata given in configuration. Whole job is mapped to job
metadata structure and tasks are mapped to either external ones or internal ones
(internal commands has to be defined within worker), both are different whether
they are executed in sandbox or as internal worker commands.
Another division of tasks is by task-type field in configuration. This field can
have four values: initiation, execution, evaluation and inner. All was discussed
and described above in configuration analysis. What is important to worker is
how to behave if execution of task with some particular type fails. There are
two possible situations execution fails due to bad user solution or due to some
internal error. If execution fails on internal error solution cannot be declared
overly as failed. User should not be punished for bad configuration or some
network error. This is where task types are useful. Generally initiation,
execution and evaluation are tasks which are somehow executing code which was
given by users who submitted solution of exercise. If this kinds of tasks fail
it is probably connected with bad user solution and can be evaluated. But if
some inner task fails solution should be re-executed, in best case scenario on
different worker. That is why if inner task fails it is sent back to broker
which will reassign job to another worker. More on this subject should be
discussed in broker assigning algorithms section.
There is also question about working directory or directories of job, which
directories should be used and what for. There is one simple answer on this
every job will have only one specified directory which will contain every file
with which worker will work in the scope of whole job execution. This is of
course nonsense there has to be some logical division. The least which must be
done are two folders one for internal temporary files and second one for
evaluation. The directory for temporary files is enough to comprehend all kind
of internal work with filesystem but only one directory for whole evaluation is
somehow not enough. Users solutions are downloaded in form of zip archives so
why these should be present during execution or why the results and files which
should be uploaded back to fileserver should be cherry picked from the one big
directory? The answer is of course another logical division into subfolders. The
solution which was chosen at the end is to have folders for downloaded archive,
decompressed solution, evaluation directory in which user solution is executed
and then folders for temporary files and for results and generally files which
should be uploaded back to fileserver with solution results. Of course there has
to be hierarchy which separate folders from different workers on the same
machines. That is why paths to directories are in format:
${DEFAULT}/${FOLDER}/${WORKER_ID}/${JOB_ID} where default means default working
directory of whole worker, folder is particular directory for some purpose
(archives, evaluation...). Mentioned division of job directories proved to be
flexible and detailed enough, everything is in logical units and where it is
supposed to be which means that searching through this system should be easy. In
addition if solutions of users have access only to evaluation directory then
they do not have access to unnecessary files which is better for overall
security of whole ReCodEx.
As we discovered above worker has job directories but users who are writing and
managing job configurations do not know where they are (on some particular
@ -720,9 +769,31 @@ description of variable.
#### Evaluation
After successful arrival of job, worker has to prepare new execution environment, then solution archive has to be downloaded from fileserver and extracted. Job configuration is located within these files and loaded into internal structures and executed. After that results are uploaded back to fileserver. These steps are the basic ones which are really necessary for whole execution and have to be executed in this precise order.
Interesting problem is with supplementary files (inputs, sample outputs). There are two approaches which can be observed. Supplementary files can be downloaded either on the start of the execution or during execution. If the files are downloaded at the beginning execution does not really started at this point and if there are problems with network worker find it right away and can abort execution without executing single task. Slight problems can arise if some of the files needs to have same name (e.g. solution assumes that input is `input.txt`), in this scenario downloaded files cannot be renamed at the beginning but during execution which is somehow impractical and not easily observed. Second solution of this problem when files are downloaded on the fly has quite opposite problem, if there are problems with network worker will find it during execution when for instance almost whole execution is done, this is also not ideal solution if we care about burnt hardware resources. On the other hand using this approach users have quite advanced control of execution flow and know what files exactly are available during execution which is from users perspective probably more appealing then the first solution. Based on that downloading of supplementary files using 'fetch' tasks during execution was chosen and implemented.
After successful arrival of job, worker has to prepare new execution
environment, then solution archive has to be downloaded from fileserver and
extracted. Job configuration is located within these files and loaded into
internal structures and executed. After that results are uploaded back to
fileserver. These steps are the basic ones which are really necessary for whole
execution and have to be executed in this precise order.
Interesting problem is with supplementary files (inputs, sample outputs). There
are two approaches which can be observed. Supplementary files can be downloaded
either on the start of the execution or during execution. If the files are
downloaded at the beginning execution does not really started at this point and
if there are problems with network worker find it right away and can abort
execution without executing single task. Slight problems can arise if some of
the files needs to have same name (e.g. solution assumes that input is
`input.txt`), in this scenario downloaded files cannot be renamed at the
beginning but during execution which is somehow impractical and not easily
observed. Second solution of this problem when files are downloaded on the fly
has quite opposite problem, if there are problems with network worker will find
it during execution when for instance almost whole execution is done, this is
also not ideal solution if we care about burnt hardware resources. On the other
hand using this approach users have quite advanced control of execution flow and
know what files exactly are available during execution which is from users
perspective probably more appealing then the first solution. Based on that
downloading of supplementary files using 'fetch' tasks during execution was
chosen and implemented.
#### Caching mechanism
@ -748,7 +819,19 @@ within cron which is able to delete files which were unused for some time.
Together with worker fetching feature cleaner completes machine specific caching
system.
Cleaner as mentioned is simple script which is executed regularly as cron job. If there is caching system like it was introduced in paragraph above there are little possibilities how cleaner should be implemented. On various filesystems there is usually support for two particular timestamps, `last access time` and `last modification time`. Files in cache are once downloaded and then just copied, this means that last modification time is set only once on creation of file and last access time should be set every time on copy. This imply last access time is what is needed here. But last modification time is widely used by operating systems, on the other hand last access time is not by default. More on this subject can be found [here](https://en.wikipedia.org/wiki/Stat_%28system_call%29#Criticism_of_atime). For proper cleaner functionality filesystem which is used by worker for caching has to have last access time for files enabled.
Cleaner as mentioned is simple script which is executed regularly as cron job.
If there is caching system like it was introduced in paragraph above there are
little possibilities how cleaner should be implemented. On various filesystems
there is usually support for two particular timestamps, `last access time` and
`last modification time`. Files in cache are once downloaded and then just
copied, this means that last modification time is set only once on creation of
file and last access time should be set every time on copy. This imply last
access time is what is needed here. But last modification time is widely used by
operating systems, on the other hand last access time is not by default. More on
For proper cleaner functionality filesystem which is used by worker for caching
has to have last access time for files enabled.
Having cleaner as separated component and caching itself handled in worker is
kind of blurry and is not clearly observable that it works without any race
@ -775,11 +858,21 @@ delete files. Implementation in cleaner follows:
- there is a loop going through all files and even directories in specified
cache folder
- last access time of file or folder is detected
- last access time is subtracted from reference timestamp into difference
- difference is compared against specified maximal file age, if difference
is greater, file or folder is deleted
Previous description implies that there is gap between detection of last access time and deleting file within cleaner. In the gap there can be worker which will access file and the file is anyway deleted but this is fine, file is deleted but worker has it copied. Another problem can be with two workers downloading the same file, but this is also not a problem file is firstly downloaded to working folder and after that copied to cache. And even if something else unexpectedly fails and because of that fetch task will fail during execution even that should be fine. Because fetch tasks should have 'inner' task type which implies that fail in this task will stop all execution and job will be reassigned to another worker. It should be like the last salvation in case everything else goes wrong.
- last access time is subtracted from reference timestamp into
difference
- difference is compared against specified maximal file age, if
difference is greater, file or folder is deleted
Previous description implies that there is gap between detection of last access
time and deleting file within cleaner. In the gap there can be worker which will
access file and the file is anyway deleted but this is fine, file is deleted but
worker has it copied. Another problem can be with two workers downloading the
same file, but this is also not a problem file is firstly downloaded to working
folder and after that copied to cache. And even if something else unexpectedly
fails and because of that fetch task will fail during execution even that should
be fine. Because fetch tasks should have 'inner' task type which implies that
fail in this task will stop all execution and job will be reassigned to another
worker. It should be like the last salvation in case everything else goes wrong.
#### Sandboxing
@ -788,15 +881,53 @@ describing all possible approaches is out of scope of this document. Instead of
that have a look at some of the features which are certainly needed for ReCodEx
and propose some particular sandboxes implementations on linux or Windows.
General purpose of sandbox is safely execute software in any form, from scripts to binaries. Various sandboxes differ in how safely are they and what limiting features they have. Ideal situation is that sandbox will have numerous options and corresponding features which will allow administrators to setup environment as they like and which will not allow user programs to somehow damage executing machine in any way possible.
For ReCodEx and its evaluation there is need for at least these features: execution time and memory limitation, disk operations limit, disk accessibility restrictions and network restrictions. All these features if combined and implemented well are giving pretty safe sandbox which can be used for all kinds of users solutions and should be able to restrict and stop any standard way of attacks or errors.
Linux systems have quite extent support of sandboxing in kernel, there were introduced and implemented kernel namespaces and cgroups which combined can limit hardware resources (cpu, memory) and separate executing program into its own namespace (pid, network). These two features comply sandbox requirement for ReCodEx so there were two options, either find existing solution or implement new one. Luckily existing solution was found and its name is **isolate**. Isolate does not use all possible kernel features but only subset which is still enough to be used by ReCodEx.
The opposite situation is in Windows world, there is limited support in its kernel which makes sandboxing a bit trickier. Windows kernel only has ways how to restrict privileges of a process through restriction of internal access tokens. Monitoring of hardware resources is not possible but used resources can be obtained through newly created job objects. But find sandbox which can do all things needed for ReCodEx seems to be impossible. There are numerous sandboxes for Windows but they all are focused on different things in a lot of cases they serves as safe environment for malicious programs, viruses in particular. Or they are designed as a separate filesystem namespace for installing a lot of temporarily used programs. From all these we can mention Sandboxie, Comodo Internet Security, Cuckoo sandbox and many others. None of these is fitted as sandbox solution for ReCodEx. With this being said we can safely state that designing and implementing new general sandbox for Windows is out of scope of this project.
New general sandbox for Windows is out of bussiness but what about more specialized solution used for instance only for C#. CLR as a virtual machine and runtime environment has a pretty good security support for restrictions and separation which is also transfered to C#. This makes it quite easy to implement simple sandbox within C# but suprisingly there cannot be found some well known general purpose implementations. As said in previous paragraph implementing our own solution is out of scope of project there is simple not enough time. But C# sandbox is quite good topic for another project for example semestral project for C# course so it might be written and integrated in future.
General purpose of sandbox is safely execute software in any form, from scripts
to binaries. Various sandboxes differ in how safely are they and what limiting
features they have. Ideal situation is that sandbox will have numerous options
and corresponding features which will allow administrators to setup environment
as they like and which will not allow user programs to somehow damage executing
machine in any way possible.
For ReCodEx and its evaluation there is need for at least these features:
execution time and memory limitation, disk operations limit, disk accessibility
restrictions and network restrictions. All these features if combined and
implemented well are giving pretty safe sandbox which can be used for all kinds
of users solutions and should be able to restrict and stop any standard way of
attacks or errors.
Linux systems have quite extent support of sandboxing in kernel, there were
introduced and implemented kernel namespaces and cgroups which combined can
limit hardware resources (cpu, memory) and separate executing program into its
own namespace (pid, network). These two features comply sandbox requirement for
ReCodEx so there were two options, either find existing solution or implement
new one. Luckily existing solution was found and its name is **isolate**.
Isolate does not use all possible kernel features but only subset which is still
enough to be used by ReCodEx.
The opposite situation is in Windows world, there is limited support in its
kernel which makes sandboxing a bit trickier. Windows kernel only has ways how
to restrict privileges of a process through restriction of internal access
tokens. Monitoring of hardware resources is not possible but used resources can
be obtained through newly created job objects. But find sandbox which can do all
things needed for ReCodEx seems to be impossible. There are numerous sandboxes
for Windows but they all are focused on different things in a lot of cases they
serves as safe environment for malicious programs, viruses in particular. Or
they are designed as a separate filesystem namespace for installing a lot of
temporarily used programs. From all these we can mention Sandboxie, Comodo
Internet Security, Cuckoo sandbox and many others. None of these is fitted as
sandbox solution for ReCodEx. With this being said we can safely state that
designing and implementing new general sandbox for Windows is out of scope of
this project.
New general sandbox for Windows is out of bussiness but what about more
specialized solution used for instance only for C#. CLR as a virtual machine and
runtime environment has a pretty good security support for restrictions and
separation which is also transfered to C#. This makes it quite easy to implement
simple sandbox within C# but suprisingly there cannot be found some well known
general purpose implementations. As said in previous paragraph implementing our
own solution is out of scope of project there is simple not enough time. But C#
sandbox is quite good topic for another project for example semestral project
for C# course so it might be written and integrated in future.
Martin Polanka
8 years ago