@todo: why we need priority and exact order of tasks?
@todo: division to initiation, execution and evaluation why and what for
@todo: how to solve problem with specific worker environment, mention internal job variables
The _job root_ task is imaginary single starting point of each job. When the
_CompileA_ task is finished, the _RunAA_ task is started (or _RunAB_, but should
be deterministic by position in configuration file -- tasks stated earlier
should be executed earlier). The task priorities guaranties, that after
_CompileA_ task all dependent tasks are executed before _CompileB_ task (they
have higher priority number). For example this is useful to control which files
are present in a working directory at every moment. To sum up, there are 3
ordering criteria: dependencies, then priorities and finally position of task in
configuration. Together, they define a unambiguous linear ordering of all tasks.
For grading there are several important tasks. First, tasks executing submitted
code need to be checked for time and memory limits. Second, outputs of judging
tasks need to be checked for correctness (represented by return value or by data
on standard output) and should not fail on time or memory limits. This division
is transparent for backend, each task is executed the same way. But frontend
must know which tasks from whole job are important and what is their kind. It is
reasonable, to keep this piece of information alongside the tasks in job
configuration, so each task can have a label about its purpose. There are three
categories of tasks:
- _initiation_ -- setting up the environment, compilling
code, etc.; for users failure means error in their sources which are not
compatible with running it with examination data
- _execution_ -- running the user code with examination data, must not exceed
time and memory limits; for users failure means wrong design, slow data
structures, etc.
- _evaluation_ -- comparing user and examination outputs; for user failure means
that the program does not compute the right results
Each job is composed of multiple tasks of these types which are semanticaly
grupped into tests. A test can represent one set of examination data for user
code. To mark the groupping, another task label can be used. Each test must have
exactly one _evaluation_ task (to show success or failure to users) and
arbitraty number of tasks with other types.
## Implementation analysis
@ -614,10 +643,12 @@ synchronization and such.
At this point we have worker with two internal parts listening one and execution one. Implementation of first one is quite straighforward and clear. So lets discuss what should be happening in execution subsystem. Jobs as work units can quite vary and do completely different things, that means configuration and worker has to be prepared for this kind of generality. Configuration and its solution was already discussed above, implementation in worker is then quite straightforward. Worker has internal structures to which loads and which stores metadata given in configuration. Whole job is mapped to job metadata structure and tasks are mapped to either external ones or internal ones (internal commands has to be defined within worker), both are different whether they are executed in sandbox or as internal worker commands.
@todo: task types: initiation, execution, evaluation and inner... mainly describe what is happening if inner task fails
@todo: task types: initiation, execution, evaluation and inner... mainly describe what is happening if inner task fails; tasks types are mentioned in "Evaluation unit executed on backend" section above, do not duplicate content here
@todo: maybe describe folders within execution and what they can be used for?
@todo: how to solve problem with specific worker environment, mention internal job variables
After successful arrival of job, worker has to prepare new execution environment, then solution archive has to be downloaded from fileserver and extracted. Job configuration is located within these files and loaded into internal structures and executed. After that results are uploaded back to fileserver. These steps are the basic ones which are really necessary for whole execution and have to be executed in this precise order.
Interesting problem is with supplementary files (inputs, sample outputs). There are two approaches which can be observed. Supplementary files can be downloaded either on the start of the execution or during execution. If the files are downloaded at the beginning execution does not really started at this point and if there are problems with network worker find it right away and can abort execution without executing single task. Slight problems can arise if some of the files needs to have same name (e.g. solution assumes that input is `input.txt`), in this scenario downloaded files cannot be renamed at the beginning but during execution which is somehow impractical and not easily observed. Second solution of this problem when files are downloaded on the fly has quite opposite problem, if there are problems with network worker will find it during execution when for instance almost whole execution is done, this is also not ideal solution if we care about burnt hardware resources. On the other hand using this approach users have quite advanced control of execution flow and know what files exactly are available during execution which is from users perspective probably more appealing then the first solution. Based on that downloading of supplementary files using 'fetch' tasks during execution was chosen and implemented.
Petr Stefan
8 years ago