|
|
@ -487,14 +487,51 @@ described as well.
|
|
|
|
|
|
|
|
|
|
|
|
### Evaluation unit executed on backend
|
|
|
|
### Evaluation unit executed on backend
|
|
|
|
|
|
|
|
|
|
|
|
@todo: describe possibilities of "piece of work" which can backend execute, how they can look like, describe our job and its tasks
|
|
|
|
One of the bigger requests for the new system is to support complex
|
|
|
|
|
|
|
|
configuration of execution pipeline. The idea comes from lecturers of Compiler
|
|
|
|
@todo: why is there division to internal and external tasks and why it is needed
|
|
|
|
principles class who want to migrate their semi-manual evaluation process to
|
|
|
|
|
|
|
|
CodEx. Unfortunately, CodEx is not capable of such compilicated exercise setup.
|
|
|
|
|
|
|
|
None of evaluation systems we found is capable of such task, so design from
|
|
|
|
|
|
|
|
scratch is needed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
There are two main approaches to design a complex execution configuration. It
|
|
|
|
|
|
|
|
can be composed of small amount of relatively big components or much more small
|
|
|
|
|
|
|
|
tasks. Big components are easy to write and whole configuration is reasonably
|
|
|
|
|
|
|
|
small. The components are designed for current problems, so it is not scalable
|
|
|
|
|
|
|
|
enough for pleasant future usage. This can be solved by introducing small set of
|
|
|
|
|
|
|
|
single-purposed tasks which can be composed together. The whole configuration is
|
|
|
|
|
|
|
|
then quite bigger, but with great adaptation ability for new conditions and also
|
|
|
|
|
|
|
|
less amount of work programming them. For better user experience, configuration
|
|
|
|
|
|
|
|
generators for some common cases can be introduced.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ReCodEx target is to be continuously developed and used for many years, so the
|
|
|
|
|
|
|
|
smaller tasks are the right choice. Observation of CodEx system shows that
|
|
|
|
|
|
|
|
only a few tasks are needed. In extreme case, only one task is enough -- execute
|
|
|
|
|
|
|
|
a binary. However, for better portability of configurations along different
|
|
|
|
|
|
|
|
systems it is better to implement reasonable subset of operations directly
|
|
|
|
|
|
|
|
without calling system provided binaries. These operations are copy file, create
|
|
|
|
|
|
|
|
new directory, extract archive and so on, altogether called internal tasks.
|
|
|
|
|
|
|
|
Another benefit from custom implementation of these tasks is guarantied safety,
|
|
|
|
|
|
|
|
so no sandbox needs to be used as in exernal tasks case.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
For a job evaluation, the tasks needs to be executed sequentialy in a specified
|
|
|
|
|
|
|
|
order. The idea of running independent tasks in parallel is bad because exact
|
|
|
|
|
|
|
|
time measurement needs controled environment on target computer with
|
|
|
|
|
|
|
|
minimalization of interrupts by other processes. It seems that connecting tasks
|
|
|
|
|
|
|
|
into directed acyclic graph (DAG) can handle all possible problem cases. None of
|
|
|
|
|
|
|
|
the authors, supervisors and involved faculty staff can think of a problem that
|
|
|
|
|
|
|
|
cannot be decomposed into tasks connected in a DAG. The goal of evaluation is
|
|
|
|
|
|
|
|
to satisfy as many tasks as possible. During execution there are sometimes
|
|
|
|
|
|
|
|
multiple choices of next task. To control that, each task can have a priority,
|
|
|
|
|
|
|
|
which is used as a secondary ordering criterium. For better understanding, here
|
|
|
|
|
|
|
|
is a small example.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
![Task serialization](https://github.com/ReCodEx/wiki/raw/master/images/Assignment_overview.png)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@todo: why we need priority and exact order of tasks?
|
|
|
|
|
|
|
|
|
|
|
|
@todo: division to initiation, execution and evaluation why and what for
|
|
|
|
@todo: division to initiation, execution and evaluation why and what for
|
|
|
|
|
|
|
|
|
|
|
|
@todo: in what order should tasks be executed, how to sort them
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@todo: how to solve problem with specific worker environment, mention internal job variables
|
|
|
|
@todo: how to solve problem with specific worker environment, mention internal job variables
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|