of Codility is [opensource](https://github.com/Codility/cui), the rest of
of Codility is [opensource](https://github.com/Codility/cui), the rest of
@ -240,44 +238,43 @@ captured progress of writing code for each user.
### CMS
### CMS
[CMS](http://cms-dev.github.io/index.html) is an opensource distributed system
[CMS](http://cms-dev.github.io/index.html) is an opensource distributed system
for running and organizing programming contests. It is written in Python and
for running and organizing programming contests. It is written in Python and
contain several modules. CMS supports C/C++, Pascal, Python, PHP and Java.
contains several modules. CMS supports C/C++, Pascal, Python, PHP, and Java
PostgreSQL is a single point of failure, all modules heavily depend on database
programming languages. PostgreSQL is a single point of failure, all modules
connection. Task evaluation can be only three step pipeline -- compilation,
heavily depend on the database connection. Task evaluation can be only a three
execution, evaluation. Execution is performed in
step pipeline -- compilation, execution, evaluation. Execution is performed in
[Isolate](https://github.com/ioi/isolate), sandbox written by consultant of our
[Isolate](https://github.com/ioi/isolate), sandbox written by the consultant
project, Mgr. Martin Mareš, Ph.D.
of our project, Mgr. Martin Mareš, Ph.D.
### MOE
### MOE
[MOE](http://www.ucw.cz/moe/) is a grading system written in Shell scripts, C
[MOE](http://www.ucw.cz/moe/) is a grading system written in Shell scripts, C
and Python. It does not provide a default GUI interface, all actions have to be
and Python. It does not provide a default GUI interface, all actions have to be
performed from command line. The system does not evaluate submissions in real
performed from command line. The system does not evaluate submissions in real
time, results are computed in batch mode after exercise deadline, using Isolate
time, results are computed in batch mode after exercise deadline, using Isolate
for sandboxing. Parts of MOE are used in other systems like CodEx or CMS, but
for sandboxing. Parts of MOE are used in other systems like CodEx or CMS, but
the system is generally obsolete.
the system is generally obsolete.
### Kattis
### Kattis
[Kattis](http://www.kattis.com/) is another SaaS solution. It provides a clean
[Kattis](http://www.kattis.com/) is another SaaS solution. It provides a clean
and functional web UI, but the rest of the application is too simple. A nice
and functional web UI, but the rest of the application is too simple. A nice
feature is the usage of a [standardized
feature is the usage of a [standardized
format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for
format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for
exercises. Kattis is primarily used by programming contest organizators, company
exercises. Kattis is primarily used by programming contest organizators, company
recruiters and also some universities.
recruiters and also some universities.
## ReCodEx goals
## ReCodEx goals
From the survey above it is clear, that none of the existing systems is capable
None of the existing systems we came across is capable of all the required features
of all the features collected for the new system. No grading system is designed
of the new system. There is no grading system which is designed to support a complicated
to support complicated evaluation pipeline, so this part is unexplored field and
evaluation pipeline, so this part is an unexplored field and has to be designed with caution.
has to be designed with caution. Also, no project is modern and extendable in a
Also, no project is modern and extensible so it could be used as a base for ReCodEx.
way that it can be used as a base for ReCodEx. After considering all these
After considering all these facts, it was clear that a new system has to be written
facts, it is clear that the new system has to be written from scratch. This
from scratch. This implies, that only a subset of all the features will be implemented
implies, that only subset of all features will be implemented in the first
in the first version, the other in the following ones.
version, the others following later.
Gathered features are categorized based on priorities for the whole system. The
Gathered features are categorized based on priorities for the whole system. The
highest priority has main functionality similar to current CodEx. It is a base
highest priority has main functionality similar to current CodEx. It is a base
@ -294,41 +291,40 @@ side) and command-line submit tool. Plagiarism detection is not likely to be
part of any release in near future unless someone other makes the engine. The
part of any release in near future unless someone other makes the engine. The
detection problem is too hard to be solved as part of this project.
detection problem is too hard to be solved as part of this project.
The new project is **ReCodEx -- ReCodEx Code Examiner**. The name should point
We named the project as **ReCodEx -- ReCodEx Code Examiner**. The name should point
to CodEx, previous evaluation solution, but also reflect new approach to solve
to the old CodEx, but also reflect the new approach to solve issues.
issues. **Re** as part of the name means redesigned, rewritten, renewed or
**Re** as part of the name means redesigned, rewritten, renewed, or restarted.
restarted.
At this point there is a clear idea how the new system will be used and what are
At this point there is a clear idea how the new system will be used and what are the
major enhancements for future releases. With this in mind, the overall
major enhancements for future releases. With this in mind, the overall
architecture can be sketched. From the previous research, we set up several
architecture can be sketched. From the previous research, we set up several
goals, which a new system should have. They mostly reflect drawbacks of current
goals, which the new system should have. They mostly reflect drawbacks of the current
version of CodEx and reasonable wishes of university users. Most notable
version of CodEx and some reasonable wishes of university users. Most notable
features are following:
features are following:
- modern HTML5 web frontend written in Javascript using a suitable framework
- modern HTML5 web frontend written in JavaScript using a suitable framework
- REST API implemented in PHP, communicating with database, backend and file
- REST API implemented in PHP, communicating with database, evaluation backend and a file
server
server
- backend is implemented as distributed system on top of message queue framework
- evaluation backend implemented as a distributed system on top of a message queue framework
(ZeroMQ) with master-worker architecture
(ZeroMQ) with master-worker architecture
- worker with basic support of Windows environment (without sandbox, no general
<!-- @todo: WTF is worker??? The concept has not been introduced yet! -->
- worker with basic support of the Windows environment (without sandbox, no general
purpose suitable tool available yet)
purpose suitable tool available yet)
- evaluation procedure configured in YAML file, compound of small tasks
- evaluation procedure configured in a YAML file, compound of small tasks
connected into arbitrary oriented acyclic graph
connected into an arbitrary oriented acyclic graph
### Intended usage
### Intended usage
Whole system is intended to help both supervisors and students. To achieve this,
The whole system is intended to help both teachers (supervisors) and students.
it is crucial to keep in mind typical usage scenarios of the system and try to
To achieve this, it is crucial to keep in mind typical usage scenarios of the
make these typical tasks as simple as possible. To synchronize visions of
system and try to make these typical tasks as simple as possible.
readers, basic concepts are recapitulated.
First of all, the system has database of users. Each user has assigned a role,
The system has a database of users. Each user has a role assigned,
which correspond to his/her privileges. User can be logged in via local
which correspond to his/her privileges. User can be logged in via
authentication service or university system. There are groups of users, which
email and password or using the university system. There are groups of users, which
corresponds to lectured courses. Groups can be hierarchically ordered to reflect
corresponds to the lectured courses. Groups can be hierarchically ordered to reflect
additional metadata like academic year. For example, reasonable group hierarchy
additional metadata such as the academic year. For example, a reasonable group hierarchy
is like this:
can look like this:
```
```
Summer term 2016
Summer term 2016
@ -341,32 +337,34 @@ Summer term 2016
```
```
In this example, student users are part of the leaf groups, higher groups are
In this example, students are members of the leaf groups, the higher level groups
just for keeping related groups together. The hierarchy tree can be modified and
are just for keeping the related groups together. The hierarchy tree can be modified and
altered to fit specific needs for each organization, even the flat structure is
altered to fit specific needs of the university or any other organization, even the
possible.
flat structure (i.e., no hierarchy) is possible.
One user can be part of multiple groups and also one group can have multiple
One user can be part of multiple groups and also one group can of course have multiple
users. Each user in a group has a role which defines its capabilities.
users. Each user in a group has also a specific role for the given group.
Priviledged user can assign a new exercise in his/her group, change assignment
Priviledged user (supervisor) can assign a new exercise in his/her group, change assignment
details, view results of other users and manually change them. Normal user can
details, view results of other users and manually change them. Normal user (student) can
join a group, get list of assigned exercises, view assignment detail, submit
join a group, get list of assigned exercises, view assignment detail, submit
his/her solution and of course view the results.
his/her solution and view the results of the evaluation.
Database of exercises (algorithmic problems) is another part of the project.
Database of exercises (algorithmic problems) is another part of the project.
Each exercise consists of text in multiple language variants, evaluation
Each exercise consists of a text in multiple language variants, an evaluation
configuration and set of inputs and reference outputs. Exercises are created by
configuration and a set of inputs and reference outputs. Exercises are created by
instructed priviledged users. Assigning exercise to a group means choose one of
instructed priviledged users. Assigning an exercise to a group means to choose
the exercises in the list and specify additional data. Assignment has a
one of the available exercises and specifying additional properties. An assignment
deadline, maximum amount of points and configuration for calculating the final
has a deadline (optionally a second deadline), a maximum amount of points,
amount, number of tries and supported runtimes (programming languages) including
a configuration for calculating the final score, a maximum number of submissions,
specific time and memory limits for sandboxed tasks.
and a list of supported runtime environemnts (e.g., programming languages) including
specific time and memory limits for the sandboxed tasks.
#### Exercise evaluation chain
#### Exercise evaluation chain
The most important part of the application is evaluating exercises for solutions
The most important part of the system is the evaluation of the solutions
submitted by users. For imaginary system architecture _UI_, _API_, _Broker_ and
submitted by the users for their assigned exercises.
_Worker_ this goes as follows.
~~For imaginary system architecture _UI_, _API_, _Broker_ and _Worker_ this goes as follows.~~
First thing users have to do is to submit their solutions to _UI_ which provides
First thing users have to do is to submit their solutions to _UI_ which provides
interface to upload files and then submit them. _UI_ sends a request to _API_
interface to upload files and then submit them. _UI_ sends a request to _API_
@ -400,25 +398,24 @@ includes overview which part succeeded and which failed (optionally with reason
like "memory limit exceeded") and amount of awarded points.
like "memory limit exceeded") and amount of awarded points.
Analysis
# Analysis
========
## Solution concepts analysis
## Solution concepts analysis
@todo: what problems were solved on abstract and high levels, how they can be solved and what was the final solution
@todo: what problems were solved on abstract and high levels, how they can be solved and what was the final solution
- which problems are they? ... these ones ↓
- which problems are they? ... these ones ↓
- what type of users there should be, why they are needed
- what type of users there should be, why they are needed
- explain why there is exercise and assignment division, what means what and how they are used
- explain why there is exercise and assignment division, what means what and how they are used
- explain instances why they are usefull what they solve and also discuss licences concept
- explain instances why they are usefull what they solve and also discuss licences concept
- groups, they can be public and private and why is that, what it solves, explain amd discuss treshold and other group features
- groups, they can be public and private and why is that, what it solves, explain amd discuss treshold and other group features
- extended execution pipeline (not just compilation/execution/evaluation) and why it is needed
- extended execution pipeline (not just compilation/execution/evaluation) and why it is needed
- progress state, how it can be done and displayed to user, why random messages
- progress state, how it can be done and displayed to user, why random messages
- how to display generally all outputs of executed programs to user (supervisor, student), what students can or cannot see and why
- how to display generally all outputs of executed programs to user (supervisor, student), what students can or cannot see and why
- judges, discuss what they possibly can do and what it can be used for (returning for instance 2 numbers instead of 1 and why we return just one)
- judges, discuss what they possibly can do and what it can be used for (returning for instance 2 numbers instead of 1 and why we return just one)
- discuss points assigned to solution, why are there bonus points, explain minimal point threshold
- discuss points assigned to solution, why are there bonus points, explain minimal point threshold
- discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why
- discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why
- and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks
- and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks
### Structure of the project
### Structure of the project
@ -426,9 +423,9 @@ The ReCodEx project is divided into two logical parts – the *Backend*
and the *Frontend*– which interact which each other and which cover the
and the *Frontend*– which interact which each other and which cover the
whole area of code examination. Both of these logical parts are
whole area of code examination. Both of these logical parts are
independent of each other in the sense of being installed on separate
independent of each other in the sense of being installed on separate
machines on different locations and that one of the parts can be
machines at different locations and that one of the parts can be
replaced with different implementation and as long as the communication
replaced with a different implementation and as long as the communication
protocols are preserved, the system will continue to work as expected.
protocols are preserved, the system will continue working as expected.
*Backend* is the part which is responsible solely for the process of
*Backend* is the part which is responsible solely for the process of
evaluation a solution of an exercise. Each evaluation of a solution is
evaluation a solution of an exercise. Each evaluation of a solution is
@ -441,7 +438,7 @@ environment, specific version of a compiler or the job must be evaluated
on a processor with a specific number of cores. The backend
on a processor with a specific number of cores. The backend
infrastructure decides whether it will accept a job or decline it based
infrastructure decides whether it will accept a job or decline it based
on the specified requirements. In case it accepts the job, it will be
on the specified requirements. In case it accepts the job, it will be
placed in a queue and processed as soon as possible. The backend
placed in a queue and it will be processed as soon as possible. The backend
publishes the progress of processing of the queued jobs and the results
publishes the progress of processing of the queued jobs and the results
of the evaluations can be queried after the job processing is finished.
of the evaluations can be queried after the job processing is finished.
The backend produces a log of the evaluation and scores the solution
The backend produces a log of the evaluation and scores the solution
@ -649,8 +646,7 @@ cleaner completes machine specific caching system.
The Backend
#The Backend
===========
The backend is the part which is hidden to the user and which has only
The backend is the part which is hidden to the user and which has only
one purpose: evaluate user’s solutions of their assignments.
one purpose: evaluate user’s solutions of their assignments.
@ -675,7 +671,7 @@ for the technical description of the components)
### Fileserver
### Fileserver
@todo: stores particular datas from frontend and backend, hashing, HTTP API
@todo: stores particular data from frontend and backend, hashing, HTTP API
### Worker
### Worker
@ -996,7 +992,7 @@ of entities and relational database models), describe the logical
Petr Stefan
8 years ago