of Codility is [opensource](https://github.com/Codility/cui), the rest of
@ -242,12 +240,12 @@ captured progress of writing code for each user.
[CMS](http://cms-dev.github.io/index.html) is an opensource distributed system
for running and organizing programming contests. It is written in Python and
contain several modules. CMS supports C/C++, Pascal, Python, PHP and Java.
PostgreSQL is a single point of failure, all modules heavily depend on database
connection. Task evaluation can be only three step pipeline -- compilation,
execution, evaluation. Execution is performed in
[Isolate](https://github.com/ioi/isolate), sandbox written by consultant of our
project, Mgr. Martin Mareš, Ph.D.
contains several modules. CMS supports C/C++, Pascal, Python, PHP, and Java
programming languages. PostgreSQL is a single point of failure, all modules
heavily depend on the database connection. Task evaluation can be only a three
step pipeline -- compilation, execution, evaluation. Execution is performed in
[Isolate](https://github.com/ioi/isolate), sandbox written by the consultant
of our project, Mgr. Martin Mareš, Ph.D.
### MOE
@ -267,53 +265,50 @@ format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for
exercises. Kattis is primarily used by programming contest organizators, company
recruiters and also some universities.
## ReCodEx goals
From the survey above it is clear, that none of the existing systems is capable
of all the features collected for the new system. No grading system is designed
to support complicated evaluation pipeline, so this part is unexplored field and
has to be designed with caution. Also, no project is modern and extendable in a
way that it can be used as a base for ReCodEx. After considering all these
facts, it is clear that the new system has to be written from scratch. This
implies, that only subset of all features will be implemented in the first
version, the others following later.
The new project is **ReCodEx -- ReCodEx Code Examiner**. The name should point
to CodEx, previous evaluation solution, but also reflect new approach to solve
issues. **Re** as part of the name means redesigned, rewritten, renewed or
restarted.
At this point there is a clear idea how the new system will be used and what are
None of the existing systems we came across is capable of all the required features
of the new system. There is no grading system which is designed to support a complicated
evaluation pipeline, so this part is an unexplored field and has to be designed with caution.
Also, no project is modern and extensible so it could be used as a base for ReCodEx.
After considering all these facts, it was clear that a new system has to be written
from scratch. This implies, that only a subset of all the features will be implemented
in the first version, the other in the following ones.
We named the project as **ReCodEx -- ReCodEx Code Examiner**. The name should point
to the old CodEx, but also reflect the new approach to solve issues.
**Re** as part of the name means redesigned, rewritten, renewed, or restarted.
At this point there is a clear idea how the new system will be used and what are the
major enhancements for future releases. With this in mind, the overall
architecture can be sketched. From the previous research, we set up several
goals, which a new system should have. They mostly reflect drawbacks of current
version of CodEx and reasonable wishes of university users. Most notable
goals, which the new system should have. They mostly reflect drawbacks of the current
version of CodEx and some reasonable wishes of university users. Most notable
features are following:
- modern HTML5 web frontend written in Javascript using a suitable framework
- REST API implemented in PHP, communicating with database, backend and file
- modern HTML5 web frontend written in JavaScript using a suitable framework
- REST API implemented in PHP, communicating with database, evaluation backend and a file
server
- backend is implemented as distributed system on top of message queue framework
- evaluation backend implemented as a distributed system on top of a message queue framework
(ZeroMQ) with master-worker architecture
- worker with basic support of Windows environment (without sandbox, no general
<!-- @todo: WTF is worker??? The concept has not been introduced yet! -->
- worker with basic support of the Windows environment (without sandbox, no general
purpose suitable tool available yet)
- evaluation procedure configured in YAML file, compound of small tasks
connected into arbitrary oriented acyclic graph
- evaluation procedure configured in a YAML file, compound of small tasks
connected into an arbitrary oriented acyclic graph
### Intended usage
Whole system is intended to help both supervisors and students. To achieve this,
it is crucial to keep in mind typical usage scenarios of the system and try to
make these typical tasks as simple as possible. To synchronize visions of
readers, basic concepts are recapitulated.
The whole system is intended to help both teachers (supervisors) and students.
To achieve this, it is crucial to keep in mind typical usage scenarios of the
system and try to make these typical tasks as simple as possible.
First of all, the system has database of users. Each user has assigned a role,
which correspond to his/her privileges. User can be logged in via local
authentication service or university system. There are groups of users, which
corresponds to lectured courses. Groups can be hierarchically ordered to reflect
additional metadata like academic year. For example, reasonable group hierarchy
is like this:
The system has a database of users. Each user has a role assigned,
which correspond to his/her privileges. User can be logged in via
email and password or using the university system. There are groups of users, which
corresponds to the lectured courses. Groups can be hierarchically ordered to reflect
additional metadata such as the academic year. For example, a reasonable group hierarchy
can look like this:
```
Summer term 2016
@ -326,32 +321,34 @@ Summer term 2016
```
In this example, student users are part of the leaf groups, higher groups are
just for keeping related groups together. The hierarchy tree can be modified and
altered to fit specific needs for each organization, even the flat structure is
possible.
In this example, students are members of the leaf groups, the higher level groups
are just for keeping the related groups together. The hierarchy tree can be modified and
altered to fit specific needs of the university or any other organization, even the
flat structure (i.e., no hierarchy) is possible.
One user can be part of multiple groups and also one group can have multiple
users. Each user in a group has a role which defines its capabilities.
Priviledged user can assign a new exercise in his/her group, change assignment
details, view results of other users and manually change them. Normal user can
One user can be part of multiple groups and also one group can of course have multiple
users. Each user in a group has also a specific role for the given group.
Priviledged user (supervisor) can assign a new exercise in his/her group, change assignment
details, view results of other users and manually change them. Normal user (student) can
join a group, get list of assigned exercises, view assignment detail, submit
his/her solution and of course view the results.
his/her solution and view the results of the evaluation.
Database of exercises (algorithmic problems) is another part of the project.
Each exercise consists of text in multiple language variants, evaluation
configuration and set of inputs and reference outputs. Exercises are created by
instructed priviledged users. Assigning exercise to a group means choose one of
the exercises in the list and specify additional data. Assignment has a
deadline, maximum amount of points and configuration for calculating the final
amount, number of tries and supported runtimes (programming languages) including
specific time and memory limits for sandboxed tasks.
Each exercise consists of a text in multiple language variants, an evaluation
configuration and a set of inputs and reference outputs. Exercises are created by
instructed priviledged users. Assigning an exercise to a group means to choose
one of the available exercises and specifying additional properties. An assignment
has a deadline (optionally a second deadline), a maximum amount of points,
a configuration for calculating the final score, a maximum number of submissions,
and a list of supported runtime environemnts (e.g., programming languages) including
specific time and memory limits for the sandboxed tasks.
#### Exercise evaluation chain
The most important part of the application is evaluating exercises for solutions
submitted by users. For imaginary system architecture _UI_, _API_, _Broker_ and
_Worker_ this goes as follows.
The most important part of the system is the evaluation of the solutions
submitted by the users for their assigned exercises.
~~For imaginary system architecture _UI_, _API_, _Broker_ and _Worker_ this goes as follows.~~
First thing users have to do is to submit their solutions to _UI_ which provides
interface to upload files and then submit them. _UI_ sends a request to _API_
@ -385,8 +382,7 @@ includes overview which part succeeded and which failed (optionally with reason
like "memory limit exceeded") and amount of awarded points.
Analysis
========
# Analysis
## Solution concepts analysis
@ -411,9 +407,9 @@ The ReCodEx project is divided into two logical parts – the *Backend*
and the *Frontend*– which interact which each other and which cover the
whole area of code examination. Both of these logical parts are
independent of each other in the sense of being installed on separate
machines on different locations and that one of the parts can be
replaced with different implementation and as long as the communication
protocols are preserved, the system will continue to work as expected.
machines at different locations and that one of the parts can be
replaced with a different implementation and as long as the communication
protocols are preserved, the system will continue working as expected.
*Backend* is the part which is responsible solely for the process of
evaluation a solution of an exercise. Each evaluation of a solution is
@ -426,7 +422,7 @@ environment, specific version of a compiler or the job must be evaluated
on a processor with a specific number of cores. The backend
infrastructure decides whether it will accept a job or decline it based
on the specified requirements. In case it accepts the job, it will be
placed in a queue and processed as soon as possible. The backend
placed in a queue and it will be processed as soon as possible. The backend
publishes the progress of processing of the queued jobs and the results
of the evaluations can be queried after the job processing is finished.
The backend produces a log of the evaluation and scores the solution
@ -558,13 +554,13 @@ synchronization and such.
At this point we have worker with two internal parts listening one and execution
one. Implementation of first one is quite straighforward and clear. So lets
discuss what should be happening in execution subsystem. Jobs as work units can quite vary and do completely different things, that means configuration and worker has to be prepared for this kind of generality. Configuration and its solution was already discussed above, implementation in worker is then quite straightforward. Worker has internal structures to which loads and which stores metadata given in configuration. Whole job is mapped to job metadata structure and tasks are mapped to either external ones or internal ones (internal commands has to be defined within worker), both are different whether they are executed in sandbox or as internal worker commands.
@todo: maybe describe folders within execution and what they can be used for?
discuss what should be happening in execution subsystem...
After successful arrival of job, worker has to prepare new execution environment, then solution archive has to be downloaded from fileserver and extracted. Job configuration is located within these files and loaded into internal structures and executed. After that results are uploaded back to fileserver. These steps are the basic ones which are really necessary for whole execution and have to be executed in this precise order.
@todo: complete paragraph above... execution of job on worker, how it is done,
what steps are necessary and general for all jobs
Interesting problem is with supplementary files (inputs, sample outputs). There are two approaches which can be observed. Supplementary files can be downloaded either on the start of the execution or during execution. If the files are downloaded at the beginning execution does not really started at this point and if there are problems with network worker find it right away and can abort execution without executing single task. Slight problems can arise if some of the files needs to have same name (e.g. solution assumes that input is `input.txt`), in this scenario downloaded files cannot be renamed at the beginning but during execution which is somehow impractical and not easily observed. Second solution of this problem when files are downloaded on the fly has quite opposite problem, if there are problems with network worker will find it during execution when for instance almost whole execution is done, this is also not ideal solution if we care about burnt hardware resources. On the other hand using this approach users have quite advanced control of execution flow and know what files exactly are available during execution which is from users perspective probably more appealing then the first solution. Based on that downloading of supplementary files using 'fetch' tasks during execution was chosen and implemented.
@todo: how can inputs and outputs (and supplementary files) be handled (they can
be downloaded on start of execution, or during...)
As described in fileserver section stored supplementary files have special
filenames which reflects hashes of their content. As such there are no
@ -634,8 +630,7 @@ cleaner completes machine specific caching system.
The Backend
===========
#The Backend
The backend is the part which is hidden to the user and which has only
one purpose: evaluate user’s solutions of their assignments.
@ -660,7 +655,7 @@ for the technical description of the components)
### Fileserver
@todo: stores particular datas from frontend and backend, hashing, HTTP API
@todo: stores particular data from frontend and backend, hashing, HTTP API
### Worker
@ -981,7 +976,7 @@ of entities and relational database models), describe the logical
Simon Rozsival
8 years ago