<!-- -
Notes:
* Dvoustrankovy uvod - co by to melo umet
* Analýza - co se rozhodneme delat, jak by se to dalo delat, pridelit dulezitost
- pak se da odkazat na to, proc jsme co nestihli, zahrnout i advanced featury
- odkazovat se u featur, ze to je v planu v pristich verzi - co je dulezite
a co ne!! Zduvodnit tim, jakou podmnozinu featur nechat, snaze se pak bude
popisovat architektura
* V analyze vysvetlit architekturu
* Related works nechat jako samostatnou kapitolu
* Poradi - pozadavky -> related works -> analyza
* Provazani komponent musi rozumet administrator a tvurce ulohy - obecna
kapitola v analyze - puvodni kapitola o analyze byla povedena, jen se tam
micha seznam zprav nebo co - to nezajima vsechny
* Po obecnym uvodu - rozdelit podle potencialniho ctenare - uzivatel ucitel, pak
uzivatel admin
* Instalacni dokumentace stranou, jako posledni
* Uzivatelaka dokumentace - admin: popis prav, autor uloh: nejobsahlejsi, format
skriptu - ale formulovat tak, ze bude popis na co kde kliknout, jazyk popsat
separatne - v budoucnu to bude irelevantni, je potreba daleko hloubeji - je
treba popsat detailne co eelaji, i treba relativni/absolutni adresy, makra,
kde vidi prekladac knihovny a headery... - kapitola na konci
* Uzivatelska dokumentace pro studenta: vysvetleni
* Jak se boduje uloha - tezko rict, kam to patri - nekde na zacatku? Ale zajima
to vsechny role, ucitel musi vedet, jak to nakonfigurovat - zminit treba i jak
bodovat podle casu a pameti (v analyze nebo v uvodu) - vice vystupu od judge,
interpolace bodu podle vyuziti pameti... je to spis mimo uživatelskou
* Nepsat kde na jake tlacitko kliknout
* Tutorialy - scenare, co udelat kdyz chci neco, vzorove pruchody
* U formularu je nejlepsi kdyz zadna dokumentace neni, doplnit popisky k polim
formularu
* V dokumentaci popsat konfigy nekde separatne - skore, yaml - referencni
dokumentace
* Urcite ne FAQ, vic strukturovane
* Instalaci dohromady na konec
* Programatorska dokumentace - "nejmene ctenaru" - neco uz tam mame, neni to
treba davat do tistene dokumentace - do tistene dokumentace dat odkaz na wiki,
neco v tistene ale byt musi - jaky jazyk, designové rozhodnutí - zdůvodnění
nedávat do úvodní analýzy - k referencnim dokumentacim udelat uvod - "restove
API jsme pojali timto zpusobem, deli se to na tyto skupiny, ..."
* Co zvolena architektura znamena, neco to ma dat i uzivateli, ktery
architekturu nezna, kde je drzenej stav
* Z dokumentace musi byt patrne, co dela knihovna a co se musi udelat rucne -
kolik je to prace - psat to vic pro uzivatele, ktery zna technologie, nezna
knihovny
* Mit soucit s tema, ktery to toho tolik neznaji - jak technologie, tak
architekturu a system CodExu
* Nesedi cisla stranek
* Stazeni ZIPu s vystupy Backendu - roztridit na verejne a tajne, verejne i pro
studenta
-->
#
Generally, there are many different ways and opinions on how to teach people
something new. However, most people agree that a hands-on experience is one of
the best ways to make the human brain remember a new skill. Learning must be
entertaining and interactive, with fast and frequent feedback. Some kinds of
knowledge are more suitable for this practical type of learning than others, and
fortunately, programming is one of them.
University education system is one of the areas where this knowledge can be
applied. In computer programming, there are several requirements such as the
code being syntactically correct, efficient and easy to read, maintain and
extend. Correctness and efficiency can be tested automatically to help teachers
save time for their research, but reviewing bad design, bad coding habits and
logical mistakes is really hard to automate and requires manpower.
Checking programs written by students takes a lot of time and requires a lot of
mechanical, repetitive work. The first idea of an automatic evaluation system
comes from Stanford University profesors in 1965. They implemented a system
which evaluated code in Algol submitted on punch cards. In following years, many
similar products were written.
There are two basic ways of automatically evaluating code -- statically (check
the code without running it; safe, but not very precise) or dynamically (run the
code on testing inputs with checking the outputs against reference ones; needs
sandboxing, but provides good real world experience).
<!-- -
*Simon* : I am not very sure about the formulation of 'our university' - shouldn't
we rather say 'The Charles University in Prague' instead?
-->
This project focuses on the machine-controlled part of source code evaluation.
First, the problems of the software used at our university previously were
discussed and similar projects at other educational institutions were examined.
With acquired knowledge from such projects in production, we set up goals for
the new evaluation system, designed the architecture and implemented a fully
operational solution. The system is now ready for production testing
at our university.
## Assignment
The major goal of this project is to create a grading application that will be
used for programming classes at the Faculty of Mathematics and Physics of the
Charles University in Prague. However, the application should be designed in a
modular fashion so that it can be easily extended or modified to make other ways
of using it possible.
The project has a great starting point -- there is an old grading system
currently used at the university (CodEx), so its flaws and weaknesses can be
addressed. Furthermore, many teachers are willing to use and test the new
system. Following requirements were collected both from our personal experience
with CodEx and from teachers' requests.
###
These are the features which are necessary for any system for evaluation of
programming coding assignments used in any university programming course:
- students can use an intuitive user interface for interaction with the system,
mainly for viewing assigned exercises, uploading their own solutions to the
assignments, and viewing the results of the solutions after an automatic
evaluation is finished
- teachers can create exercises including textual description, sample inputs and
correct reference outputs (for example "sum all numbers from given file and
write the result to the standard output")
- teachers can assigning an existing exercise to their class with some specific
properties set (deadlines, etc.)
- teachers can specify their scale of points which will be awarted to the
students depending on the correctness of his/her solution (expressed in
percentage points)
- teachers can view all of the solutions their students submitted and also the
results of the evaluations and they can override the automatically assigned
points to the solutions manually
- teachers can see the statistics of their classes and individual students of
these claseese
- administrators can depend on a safe environment in which the students'
solutions will be executed
- administrators can manage users with support of roles (at least two --
_student_ and _supervisor_ )
CodEx satisfies all these requirements and a few more that originate from the
way courses are organized at our university -- for example, users have roles
(_student_, _supervisor_ and _administrator_ ) that determine their capabilities
in the system and students are divided into groups that correspond to lab
groups.
However, further requirements arose during the ten year long lifetime of the old
system. There are not many ways to improve it from the perspective of a student,
but a lot of feature requests came from administrators and supervisors. The
ideas were mostly gathered from meetings with faculty staff involved with the
current system.
###
- logging in through a university authentication system (e.g. LDAP)
- support for multiple programming environments at once to avoid unacceptable
workload for administrator (maintain separate installations for many courses)
and high hardware occupation
- localization (both UI and exercises)
- Markdown support for exercise texts
- tagging exercises and search by tags
- comments, comments, comments (exercises, tests, solutions, ...)
- edit student solution and privately resubmit it
- resubmit solution with saving all results (including temporary ones)
- mark one student's solution as accepted (used for grading this assignment)
- web and command-line submission tool
- SIS (university information system) integration for fetching personal user data
- plagiarism detection
- advanced low-level evaluation flow configuration with high-level abstraction
layer for ordinary configuration cases
- use of modern technologies with state-of-the-art compilers
The survey shows that the system is used in many different ways, but the core
functionality is the same for all of them. When the system is ready it is
likely that there will be new ideas of how to use the system and thus the system
must be designed to be easily extendable, so everyone can develop their own feature.
This also means that widely used programming languages and techniques should be used,
so users can quickly understand the code and make changes.
To find out the current state in the field of automatic grading systems we did a
short survey at universities, programming contests, and other available tools.
## Related work
This is not a complete list of available evaluators, but only a few projects
which are used these days and can be an inspiration for our project. Each
project from the list has a brief description and some key features mentioned.
### CodEx
Currently used grading solution at the Faculty of Mathematics and Physics of
the Charles University in Prague which was implemented in 2006 by a group
of students. It is called [CodEx -- The Code Examiner ](http://codex.ms.mff.cuni.cz/project/ )
and it has been used with some improvements since then. The original plan was
to use the system only for basic programming courses, but there was a demand
for adapting it for many different subjects.
CodEx is based on dynamic analysis. It features a web-based interface, where
supervisors can assign exercises to their students and the students have a time
window to submit their solutions. Each solution is compiled and run in sandbox
(MO-Eval). The metrics which are checked are: corectness of the output, time
and memory limits. It supports programs written in C, C++, C#, Java, Pascal,
Python and Haskell.
Current system is old, but robust. There were no major security incidents
during its production usage. However, from today's perspective there are
several drawbacks. The main ones are:
- **web interface** -- The web interface is simple and fully functional. But
rapid development in web technologies opens new horizons of how web interface
can be made.
- **web API** -- CodEx offers a very limited XML API based on outdated
technologies that is not sufficient for users who would like to create custom
interfaces such as a command line tool or mobile application.
- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system
calls and blocking the bad ones. This can be easily done for single-threaded
applications, but proves difficult with multi-threaded ones. In present day,
parallelism is a very important area of computing, so there is requirement to
test multi-threaded applications too.
- **instances** -- Different ways of CodEx usage scenarios requires separate
instances (Programming I and II, Java, C#, etc.). This configuration is not
user friendly (students have to register in each instance separately) and
burdens administrators with unnecessary work. CodEx architecture does not
allow sharing hardware between instances, which results in an inefficient use
of hardware for evaluation.
- **task extensibility** -- There is a need to test and evaluate complicated
programs for classes such as Parallel programming or Compiler principles,
which have a more difficult evaluation chain than simple
compilation/execution/evaluation provided by CodEx.
### Progtest
[Progtest ](https://progtest.fit.cvut.cz/ ) is private project of [FIT
ČVUT](https://fit.cvut.cz) in Prague. As far as we know it is used for C/C++,
Bash programming and knowledge-based quizzes. There are several bonus points
and penalties and also a few hints what is failing in the submitted solution. It
is very strict on source code quality, for example `-pedantic` option of GCC,
Valgrind for memory leaks or array boundaries checks via `mudflap` library.
### Codility
[Codility ](https://codility.com/ ) is a web based solution primary targeted to
company recruiters. It is a commercial product available as a SaaS and it supports 16
programming languages. The
[UI ](http://1.bp.blogspot.com/-_isqWtuEvvY/U8_SbkUMP-I/AAAAAAAAAL0/Hup_amNYU2s/s1600/cui.png )
of Codility is [opensource ](https://github.com/Codility/cui ), the rest of
source code is not available. One interesting feature is 'task timeline' --
captured progress of writing code for each user.
### CMS
[CMS ](http://cms-dev.github.io/index.html ) is an opensource distributed system
for running and organizing programming contests. It is written in Python and
contains several modules. CMS supports C/C++, Pascal, Python, PHP, and Java
programming languages. PostgreSQL is a single point of failure, all modules
heavily depend on the database connection. Task evaluation can be only a three
step pipeline -- compilation, execution, evaluation. Execution is performed in
[Isolate ](https://github.com/ioi/isolate ), sandbox written by the consultant
of our project, Mgr. Martin Mareš, Ph.D.
### MOE
[MOE ](http://www.ucw.cz/moe/ ) is a grading system written in Shell scripts, C
and Python. It does not provide a default GUI interface, all actions have to be
performed from command line. The system does not evaluate submissions in real
time, results are computed in batch mode after exercise deadline, using Isolate
for sandboxing. Parts of MOE are used in other systems like CodEx or CMS, but
the system is generally obsolete.
### Kattis
[Kattis ](http://www.kattis.com/ ) is another SaaS solution. It provides a clean
and functional web UI, but the rest of the application is too simple. A nice
feature is the usage of a [standardized
format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for
exercises. Kattis is primarily used by programming contest organizators, company
recruiters and also some universities.
## ReCodEx goals
None of the existing systems we came across is capable of all the required
features of the new system. There is no grading system which is designed to
support a complicated evaluation pipeline, so this part is an unexplored field
and has to be designed with caution. Also, no project is modern and extensible
so it could be used as a base for ReCodEx. After considering all these facts,
it was clear that a new system has to be written from scratch. This implies,
that only a subset of all the features will be implemented in the first version,
the other in the following ones.
Gathered features are categorized based on priorities for the whole system. The
highest priority has main functionality similar to current CodEx. It is a base
line to be useful in production environment, but a new design allows to easily
develop further. On top of that, most of ideas from faculty staff belongs to
second priority bucket, which will be implemented as part of the project. Most
advanced tasks from this category are advanced low-level evaluation
configuration format, using modern tools, connecting to a university systems and
merging separate system instances into single one. Other tasks are scheduled for
next releases after successful project defense. Namely, these are high-level
exercise evaluation configuration with user-friendly interface for common
exercise types, SIS integration (when some API will be available from their
side) and command-line submit tool. Plagiarism detection is not likely to be
part of any release in near future unless someone other makes the engine. The
detection problem is too hard to be solved as part of this project.
We named the project as **ReCodEx -- ReCodEx Code Examiner** . The name should
point to the old CodEx, but also reflect the new approach to solve issues.
**Re** as part of the name means redesigned, rewritten, renewed, or restarted.
At this point there is a clear idea how the new system will be used and what are
the major enhancements for future releases. With this in mind, the overall
architecture can be sketched. From the previous research, we set up several
goals, which the new system should have. They mostly reflect drawbacks of the
current version of CodEx and some reasonable wishes of university users. Most
notable features are following:
- modern HTML5 web frontend written in JavaScript using a suitable framework
- REST API implemented in PHP, communicating with database, evaluation backend
and a file server
- evaluation backend implemented as a distributed system on top of a message
queue framework (ZeroMQ) with master-worker architecture <!-- @todo: WTF is
worker??? The concept has not been introduced yet! -->
- worker with basic support of the Windows environment (without sandbox, no
general purpose suitable tool available yet)
- evaluation procedure configured in a YAML file, compound of small tasks
connected into an arbitrary oriented acyclic graph
### Intended usage
The whole system is intended to help both teachers (supervisors) and students.
To achieve this, it is crucial to keep in mind typical usage scenarios of the
system and try to make these typical tasks as simple as possible.
The system has a database of users. Each user has a role assigned, which
correspond to his/her privileges. User can be logged in via email and password
or using the university system. There are groups of users, which corresponds to
the lectured courses. Groups can be hierarchically ordered to reflect additional
metadata such as the academic year. For example, a reasonable group hierarchy
can look like this:
```
Summer term 2016
|-- Language C# and .NET platform
|
|
|-- Programming I
|
...
```
In this example, students are members of the leaf groups, the higher level
groups are just for keeping the related groups together. The hierarchy tree can
be modified and altered to fit specific needs of the university or any other
organization, even the flat structure (i.e., no hierarchy) is possible.
One user can be part of multiple groups and also one group can of course have
multiple users. Each user in a group has also a specific role for the given
group. Priviledged user (supervisor) can assign a new exercise in his/her
group, change assignment details, view results of other users and manually
change them. Normal user (student) can join a group, get list of assigned
exercises, view assignment detail, submit his/her solution and view the results
of the evaluation.
Database of exercises (algorithmic problems) is another part of the project.
Each exercise consists of a text in multiple language variants, an evaluation
configuration and a set of inputs and reference outputs. Exercises are created
by instructed priviledged users. Assigning an exercise to a group means to
choose one of the available exercises and specifying additional properties. An
assignment has a deadline (optionally a second deadline), a maximum amount of
points, a configuration for calculating the final score, a maximum number of
submissions, and a list of supported runtime environemnts (e.g., programming
languages) including specific time and memory limits for the sandboxed tasks.
#### Exercise evaluation chain
The most important part of the system is the evaluation of the solutions
submitted by the users for their assigned exercises. Concepts of consecutive
steps from source code of solution to results is described on architecture with
two layer -- presentation (_frontend_) and executive (_backend_).
First thing users have to do is to submit their solutions to _frontend_ which
provides interface to upload files and then submit them. It checks the
assignment invariants (deadlines, count of submissions, ...) and stores
submitted files. The runtime environment is automatically detected based on
input files and suitable exercise configuration variant is chosen (one exercise
can have multiple variants, for example C and Java languages). Matching exercise
configuration is then send to _backend_ alongside solution source files.
_Backend_ can have multiple engines to allow processing more jobs in parallel
and a loadbalancer, which tracks states of incoming jobs and performs scheduling
of them. The decission is made based on capabilities of each engine and also job
requirements. When a match is found, the job is held until the particular engine
is jobless and can receive an evaluation request.
Job processing itself stars with obtaining source files and job configuration.
The configuration is parsed into small tasks with simple piece of work.
Evaluation itself goes in direction of tasks ordering. It is crucial to keep
executive computer secure and stable, so isolated sandboxed environment is used
when dealing with unknown source code. When the execution is finished, results
are uploaded back to _frontend_ .
The _frontend_ is immediately notified about finished job. The outcomes are
parsed and results of important tasks (comparing actual and expected results)
saved into storage. Also, points are calculated depending on solution
correctness and assignment configuration. Data presented back to users includes
overview which part succeeded and which failed (optionally with reason like
"memory limit exceeded") and amount of awarded points.
# Analysis
## Solution concepts analysis
@todo: what problems were solved on abstract and high levels, how they can be solved and what was the final solution
- which problems are they? ... these ones below:
- what type of users there should be, why they are needed
- explain why there is exercise and assignment division, what means what and how they are used
- explain instances why they are usefull what they solve and also discuss licences concept
- groups, they can be public and private and why is that, what it solves, explain amd discuss treshold and other group features
- extended execution pipeline (not just compilation/execution/evaluation) and why it is needed
- progress state, how it can be done and displayed to user, why random messages
- how to display generally all outputs of executed programs to user (supervisor, student), what students can or cannot see and why
- judges, discuss what they possibly can do and what it can be used for (returning for instance 2 numbers instead of 1 and why we return just one)
- discuss points assigned to solution, why are there bonus points, explain minimal point threshold
- discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why
- and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks
### Structure of the project
The ReCodEx project is divided into two logical parts – *Backend*
and the *Frontend* –
whole area of code examination. Both of these logical parts are
independent of each other in the sense of being installed on separate
machines at different locations and that one of the parts can be
replaced with a different implementation and as long as the communication
protocols are preserved, the system will continue working as expected.
*Backend* is the part which is responsible solely for the process of
evaluation a solution of an exercise. Each evaluation of a solution is
referred to as a *job* . For each job, the system expects a configuration
document of the job, supplementary files for the exercise (e.g., test
inputs, expected outputs, predefined header files), and the solution of
the exercise (typically source codes created by a student). There might
be some specific requirements for the job, such as a specific runtime
environment, specific version of a compiler or the job must be evaluated
on a processor with a specific number of cores. The backend
infrastructure decides whether it will accept a job or decline it based
on the specified requirements. In case it accepts the job, it will be
placed in a queue and it will be processed as soon as possible. The backend
publishes the progress of processing of the queued jobs and the results
of the evaluations can be queried after the job processing is finished.
The backend produces a log of the evaluation and scores the solution
based on the job configuration document.
*Frontend* on the other hand is responsible for the communication with
the users and provides them a convenient access to the Backend
infrastructure. The Frontend manages user accounts and gathers them into
units called groups. There is a database of exercises which can be
assigned to the groups and the users of these groups can submit their
solutions for these assignments. The Frontend will initiate evaluation
of these solutions by the Backend and it will store the results
afterwards. The results will be visible to authorized users and the
results will be awarded with points according to the score given by the
Backend in the evaluation process. The supervisors of the groups can
edit the parameters of the assignments, review the solutions and the
evaluations in detail and award the solutions with bonus points (both
positive and negative) and discuss about the solution with the author of
the solution. Some of the users can be entitled to create new exercises
and extend the database of exercises which can be assigned to the groups
later on.
The Frontend developed as part of this project was created with the
needs of the Faculty of Mathematics and Physics of the Charles
university in Prague in mind. The users are the students and their
teachers, groups correspond to the different courses, the teachers are
the supervisors of these groups. We believe that this model is
applicable to the needs of other universities, schools, and IT
companies, which can use the same system for their needs. It is also
possible to develop their own frontend with their own user management
system for their specific needs and use the possibilities of the Backend
without any changes, as was mentioned in the previous paragraphs.
In the latter parts of the documentation, both of the Backend and
Frontend parts will be introduced separately and covered in more detail.
The communication protocol between these two logical parts will be
described as well.
@todo: move "General backend implementation" here
@todo: move "General frontend implementation" here
### Evaluation unit executed on backend
@todo: describe possibilities of "piece of work" which can backend execute, how they can look like, describe our job and its tasks
@todo: why is there division to internal and external tasks and why it is needed
@todo: in what order should tasks be executed, how to sort them
@todo: how to solve problem with specific worker environment, mention internal job variables
## Implementation analysis
When developing a project like ReCodEx there has to be some discussion over
implementation details and how to solve some particular problems properly.
This discussion is a never ending story which is done through the whole
development process. Some of the most important implementation problems or
interesting observations will be discussed in this chapter.
### General backend implementation
There are numerous ways how to divide some sort of system into separated
services, from one single component to many and many single-purpose components.
Having only one big service is not feasible, not scalable enough and mainly it
would be one big blob of code which somehow works and is very complex, so this
is not the way. The quite opposite, having a lot of single-purpose components is
also somehow impractical. It is scalable by default and all services would have
quite simple code but on the other hand communication requirements for such
solution would be insane. So there has to be chosen approach which is somehow in
the middle, that means services have to communicate in manner which will not
bring network down, code basis should be reasonable and the whole system has to
be scalable enough. With this being said there can be discussion over particular
division for ReCodEx system.
From the scalable point of view there are two necessary components, the one
which will execute jobs and component which will distribute jobs to the
instances of the first one. This ensures scalability in manner of parallel
execution of numerous jobs which is exactly what is needed. Implementation of
these services are called 'broker' and 'worker', first one handles distribution,
latter execution. These components should be enough to fulfil all above said,
but for the sake of simplicity and better communication gateways with frontend
two other components were added, 'fileserver' and 'monitor'. Fileserver is
simple component whose purpose is to store files which are exchanged between
frontend and backend. Monitor is also quite simple service which is able to
serve job progress state from worker to web application. These two additional
services are on the edge of frontend and backend (like gateways) but logically
they are more connected with backend, so it is considered they belong there.
@todo: what type of communication within backend could be used, mention some frameworks, queue managers, protocols, which was considered
### Fileserver
@todo: fileserver and why is separated
@todo: mention hashing on fileserver and why this approach was chosen
@todo: what can be stored on fileserver
@todo: how can jobs be stored on fileserver, mainly mention that it is nonsence to store inputs and outputs within job archive
### Broker
@todo: assigning of jobs to workers, which are possible algorithms, queues, which one was chosen
@todo: how can jobs be sent over zeromq, mainly mention that files can be transported, but it is not feasible
@todo: making action and reaction over zeromq more general and easily extensible, mention reactor and why is needed and what it solves
### Worker
Worker is component which is supposed to execute incoming jobs from broker. As
such worker should work and support wide range of different infrastructures and
maybe even platforms/operating systems. Support of at least two main operating
systems is desirable and should be implemented. Worker as a service does not
have to be much complicated, but a bit of complex behaviour is needed. Mentioned
complexity is almost exclusively concerned about robust communication with
broker which has to be regularly checked. Ping mechanism is usually used for
this in all kind of projects. This means that worker should be able to send ping
messages even during execution. So worker has to be divided into two separate
parts, the one which will handle communication with broker and the another which
will execute jobs. The easiest solution is to have these parts in separate
threads which somehow tightly communicates with each other. For inner process
commucation there can be used numerous technologies, from shared memory to
condition variables or some kind of in-process messages. Already used library
ZeroMQ is possible to provide in-process messages working on the same principles
as network communication which is quite handy and solves problems with threads
synchronization and such.
At this point we have worker with two internal parts listening one and execution one. Implementation of first one is quite straighforward and clear. So lets discuss what should be happening in execution subsystem. Jobs as work units can quite vary and do completely different things, that means configuration and worker has to be prepared for this kind of generality. Configuration and its solution was already discussed above, implementation in worker is then quite straightforward. Worker has internal structures to which loads and which stores metadata given in configuration. Whole job is mapped to job metadata structure and tasks are mapped to either external ones or internal ones (internal commands has to be defined within worker), both are different whether they are executed in sandbox or as internal worker commands.
@todo: maybe describe folders within execution and what they can be used for?
After successful arrival of job, worker has to prepare new execution environment, then solution archive has to be downloaded from fileserver and extracted. Job configuration is located within these files and loaded into internal structures and executed. After that results are uploaded back to fileserver. These steps are the basic ones which are really necessary for whole execution and have to be executed in this precise order.
Interesting problem is with supplementary files (inputs, sample outputs). There are two approaches which can be observed. Supplementary files can be downloaded either on the start of the execution or during execution. If the files are downloaded at the beginning execution does not really started at this point and if there are problems with network worker find it right away and can abort execution without executing single task. Slight problems can arise if some of the files needs to have same name (e.g. solution assumes that input is `input.txt` ), in this scenario downloaded files cannot be renamed at the beginning but during execution which is somehow impractical and not easily observed. Second solution of this problem when files are downloaded on the fly has quite opposite problem, if there are problems with network worker will find it during execution when for instance almost whole execution is done, this is also not ideal solution if we care about burnt hardware resources. On the other hand using this approach users have quite advanced control of execution flow and know what files exactly are available during execution which is from users perspective probably more appealing then the first solution. Based on that downloading of supplementary files using 'fetch' tasks during execution was chosen and implemented.
As described in fileserver section stored supplementary files have special
filenames which reflects hashes of their content. As such there are no
duplicates stored in fileserver. Worker can use feature too and caches these
files for some while and saves precious bandwith. This means there has to be
system which can download file, store it in cache and after some time of
inactivity delete it. Because there can be multiple worker instances on some
particular server it is not efficient to have this system in every worker on its
own. So it is feasible to have this feature somehow shared among all workers on
the same machine. Solution would be again having separate service connected
through network with workers which would provide such functionality but this
would component with another communication for the purpose where it is not
exactly needed. Implemented solution assume worker has access to specified cache
folder, to this folder worker can download supplementary files and copy them
from here. This means every worker has the possibility to maintain downloads to
cache, but what is worker not able to properly do is deletion of unused files
after some time. For that single-purpose component is introduced which is called
'cleaner'. It is simple script executed within cron which is able to delete
files which were unused for some time. Together with worker fetching feature
cleaner completes machine specific caching system.
@todo: describe a bit more cleaner functionality and that it is safe and there are no unrecoverable races
@todo: sandboxing, what possibilites are out there (linux, Windows), what are general and really needed features, mention isolate, what are isolate features
### Monitor
@todo: how progress status can be sent, why is there separate component of system (monitor) and why is this feature only optional
@todo: monitor and what is done there, mention caching and why it is needed
### General frontend implementation
The frontend is the part of the whole system, which is directly available to the user.
It must hold the whole state of the system - the login credentials, users' data, hierarchy
of groups, available exercises, exercises assignments, users' solutions, and much more.
User must be able to access and modify this state through some kind of an user interface
(e.g., a web application, mobile application, or a command-line tool). We want to implement
a web application as part of this project, but we want to make it possible for other
developers to contribute to the platform and to create their specific clients for
their specific needs.
We decided to split the frontend in two logical parts: a server side and a client side.
The server side is responsible for managing the state and the client side gives instructions
to the server side based on the inputs from the user. This decoupling gives us the ability
to create multiple client side tools which may address different needs of the users.
The first thing we need to address is the communication protocol of this client-server
architecture. There are several options:
- *TCP sockets* -- TCP sockets give a reliable means of a full-duplex communication. All major
operating systems support this protocol and there are libraries which simplify the implementation.
On the other side, it is not possible to initiate a TCP socket from a web browser.
- *WebSockets* -- The WebSocket standard is built on top of TCP. It enables a web browser to connect
to a server over a TCP socket. WebSockets are implemented in recent versions of all modern web browsers
and there are libraries for several programming languages like Python or JavaScript (running in Node.js).
Encryption of the communication over a WebSocket is supported as a standard.
- *HTTP protocol* -- The HTTP protocol is a state-less protocol implemented on top of the TCP protocol.
The communication between the client and server consists of a requests sent by the client and reponses
to these requests sent back by the sever. The client can send as many requests as needed and it may ignore
the responses from the server, but the server must respond only to the requests of the client and
it cannot initiate communication on its own. End-to-end encryption can be achieved easily using SSL (HTTPS).
We chose the HTTP(S) protocol because of the simple implementation in all sorts of operating systems
and runtime environments on both the client and the server side.
The API of the server should expose basic CRUD (Create, Read, Update, Delete) operations. There are some
options on what kind of messages to send over the HTTP:
- SOAP -- a protocol for exchanging XML messages. It is very robust and complex.
- REST -- is a stateless architecture style, not a protocol or a technology. It relies on HTTP (but not
necessarily) and its method verbs (e.g., GET, POST, PUT, DELETE). It can fully implement the CRUD operations.
Even though there are some other technologies we chose the REST style over the HTTP protocolo. It is widely
used, there are many tools available for development and testing, and it is understood by programmers so
it should be easy for a new developer with some experience in client-side applications to get to know with
the ReCodEx API and develop a client application.
#### API server
The API server must handle HTTP requests and manage the state of the application in some kind of a database.
It must also be able to communicate with the backend over ZeroMQ.
We considered several technologies which could be used:
- PHP + Apache -- one of the most widely used technologies for creating web servers. It is a suitable
technology for this kind of a project. It has all the features we need when some additional extensions
are installed (to support LDAP or ZeroMQ).
- ASP.NET (C#), JSP (Java) -- these technologies are very robust and are used to create server technologies in many big
enterpises. Both can run on Windows and Linux servers (ASP.NET using the .NET Core).
- JavaScript (Node.js) -- it is a quite new technology and it is being used to create REST APIs lately.
Applications running on Node.js are quite performant and the number of open-source libraries avialble
on the Internet is very huge.
We chose PHP and Apache mainly because we were familiar with these technologies and we were able to develop
all the features we needed without learning to use a new technology. Since the number of features was quite
high and needed to meet a strict deadline. This does not mean that we would find all the other technologies
superior to PHP in all other aspects - PHP 7 is a mature language with a huge comunity and a wide range of
tools, libraries, and frameworks.
We decided to use an ORM framework to manage the database, namely the widely used PHP ORM Doctrine 2.
This framework has a robust abstraction layer DBAL so the database engine is not very important and it can be changed
without any need for changing the code. We chose an open-source database MariaDB.
To speed up the development process of the PHP server application we decided to use an MVC framework.
After evaluating and trying several frameworks, such as Lumen, Laravel, and Symfony, we ended up using
the framework Nette. This framework is very common in the Czech Republic -- its main developer is a
well-known Czech programmer David Grudl -- and we were already familiar with the patterns used in this
framework (e.g., dependency injection, authentication, routing). There is a good extension for the
Nette framework which makes usage of Doctrine 2 very straighforward.
@todo: what database can be used, how it is mapped and used within code
@todo: authentication, some possibilities and describe used jwt
@todo: solution of forgotten password, why this in particular
@todo: rest api is used for report of backend state and errors, describe why and other possibilities (separate component)
@todo: what files are stored in api, why there are duplicates among api and fileserver
@todo: why are there instances and for which they can be used for, describe licences and its implementation
@todo: groups and hierarchy, describe arbitrary nesting which should be possible within instance and how it is implemented and how it could be implemented
@todo: where is stored which workers can be used by supervisors and which runtimes are available, describe possibilities and why is not implemented automatic solution
@todo: on demand loading of students submission, in-time loading of every other submission, why
#### Web-app
@todo: what technologies can be used on client frontend side, why react was used
@todo: please think about more stuff about api and web-app... thanks ;-)
#
The backend is the part which is hidden to the user and which has only
one purpose: evaluate user’
@todo: describe the configuration inputs of the Backend
@todo: describe the outputs of the Backend
@todo: describe how the backend receives the inputs and how it
communicates the results
## Components
Whole backend is not just one service/component, it is quite complex system on its own.
@todo: describe the inner parts of the Backend (and refer to the Wiki
for the technical description of the components)
### Broker
@todo: gets stuff done, single point of failure and center point of ReCodEx universe
### Fileserver
@todo: stores particular data from frontend and backend, hashing, HTTP API
### Worker
@todo: describe a bit of internal structure in general
@todo: describe how jobs are generally executed
### Monitor
@todo: not necessary component which can be ommited, proxy-like service
## Backend internal communication
@todo: internal backend communication, what communicates with what and why
The Frontend
============
The frontend is the part which is visible to the user of ReCodEx and
which holds the state of the system –
the system, the database of exercises, the assignments of these
exercises to groups of users (i.e., students), and the solutions and
evaluations of them.
Frontend is split into three parts:
- the server-side REST API (“API”) which holds the business logic and
keeps the state of the system consistent
- the relational database (“DB”) which persists the state of the
system
- the client side application (“client”) which simplifies access to
the API for the common users
The centerpiece of this architecture is the API. This component receives
requests from the users and from the Backend, validates them and
modifies the state of the system and persists this modified state in the
DB.
We have created a web application which can communicate with the API
server and present the information received from the server to the user
in a convenient way. The client can be though any application, which can
send HTTP requests and receive the HTTP responses. Users can use general
applications like [cURL ](https://github.com/curl/curl/ ),
[Postman ](https://www.getpostman.com/ ), or create their own specific
client for ReCodEx API.
Frontend capabilities
---------------------
@todo: describe what the frontend is capable of and how it really works,
what are the limitations and how it can be extended
Terminology
-----------
This project was created for the needs of a university and this fact is
reflected into the terminology used throughout the Frontend. A list of
important terms’
### User and user roles
*User* is a person who uses the application. User is granted access to
the application once he or she creates an account directly through the
API or the web application. There are several types of user accounts
depending on the set of permissions – –
been granted. Each user receives only the most basic set of permissions
after he or she creates an account and this role can be changed only by
the administrators of the service:
- *Student* is the most basic role. Student can become member of a
group and submit his solutions to his assignments.
- *Supervisor* can be entitled to manage a group of students.
Supervisor can assign exercises to the students who are members of
his groups and review their solutions submitted to
these assignments.
- *Super-admin* is a user with unlimited rights. This user can perform
any action in the system.
There are two implicit changes of roles:
- Once a *student* is added to a group as its supervisor, his role is
upgraded to a *supervisor* role.
- Once a *supervisor* is removed from the lasts group where he is a
supervisor then his role is downgraded to a *student* role.
These mechanisms do not prevent a single user being a supervisor of one
group and student of a different group as supervisors’
superset of students’
### Login
*Login* is a set of user’
be allowed to access the system as a specific user. We distinguish two
types of logins: local and external.
- *Local login* is user’
during registration.
- *External login* is a mapping of a user profile to an account of
some authentication service (e.g.,
### Instance
*An instance* of ReCodEx is in fact just a set of groups and user
accounts. An instance should correspond to a real entity as a
university, a high-school, an IT company or an HR agency. This approach
enables the system to be shared by multiple independent organizations
without interfering with each other.
Usage of the system by the users of an instance can be limited by
possessing a valid license. It is up to the administrators of the system
to determine the conditions under which they will assign licenses to the
instances.
### Group
*Group* corresponds to a school class or some other unit which gathers
users who will be assigned the same set exercises. Each group can have
multiple supervisors who can manage the students and the list of
assignments.
Groups can form a tree hierarchy of arbitrary depth. This is inspired by the
hierarchy of school classes belonging to the same subject over several school
years. For example, there can be a top level group for a programming class that
contains subgroups for every school year. These groups can then by divided into
actual student groups with respect to lab attendance. Supervisors can create
subgroups of their groups and further manage these subgroups.
### Exercise
*An exercise* consists of textual assignment of a task and a definition
of how a solution to this exercise should be processed and evaluated in
a specific runtime environment (i.e., how to compile a submitted source
code and how to test the correctness of the program). It is a template
which can be instantiated as an *assignment* by a supervisor of a group.
### Assignment
An assignment is an instance of an *exercise* assigned to a specific
*group*. An assignment can modify the text of the task assignment and it
has some additional information which is specific to the group (e.g., a
deadline, the number of points gained for a correct solution, additional
hints for the students in the assignment). The text of the assignment
can be edited and supervisors can translate the assignment into another
language.
### Solution
*A solution* is a set of files which a user submits to a given
*assignment*.
### Submission
*A submission* corresponds to a *solution* being evaluated by the
Backend. A single *solution* can be submitted repeatedly (e.g., when the
Backend encounters an error or when the supervisor changes the assignment).
### Evaluation
*An evaluation* is the processed report received from the Backend after
a *submission* is processed. Evaluation contains points given to the
user based on the quality of his solution measured by the Backend and
the settings of the assignment. Supervisors can review the evaluation
and add bonus points (both positive and negative) if the student
deserves some.
### Runtime environment
*A runtime environment* defines the used programming language or tools
which are needed to process and evaluate a solution. Examples of a
runtime environment can be:
- *Linux + GCC*
- *Linux + Mono*
- *Windows + .NET 4*
- *Bison + Yacc*
### Limits
A correct *solution* of an *assignment* has to pass all specified tests (mostly
checks that it yields the correct output for various inputs) and typically must
also be effective in some sense. The Backend measures the time and memory
consumption of the solution while running. This consumption of resources can be
*limited* and the solution will receive fewer points if it exceeds the given
limits in some test cases defined by the *exercise* .
User management
---------------
@todo: roles and their rights, adding/removing different users, how the
role of a specific user changes
Instances and hierarchy of groups
---------------------------------
@todo: What is an instance, how to create one, what are the licenses and
how do they work. Why can the groups form hierarchies and what are the
benefits –
in the group hierarchy.
Exercises database
------------------
@todo: How the exercises are stored, accessed, who can edit what
### Creating a new exercise
@todo Localized assignments, default settings
### Runtime environments and hardware groups
@todo read this later and see if it still makes sense
ReCodEx is designed to utilize a rather diverse set of workers -- there can be
differences in many aspects, such as the actual hardware running the worker
(which impacts the results of measuring) or installed compilers, interpreters
and other tools needed for evaluation. To address these two examples in
particular, we assign runtime environments and hardware groups to exercises.
The purpose of runtime environments is to specify which tools (and often also
operating system) are required to evaluate a solution of the exercise -- for
example, a C# programming exercise can be evaluated on a Linux worker running
Mono or a Windows worker with the .NET runtime. Such exercise would be assigned
two runtime environments, `Linux+Mono` and `Windows+.NET` (the environment names
are arbitrary strings configured by the administrator).
A hardware group is a set of workers that run on similar hardware (e.g. a
particular quad-core processor model and a SSD hard drive). Workers are assigned
to these groups by the administrator. If this is done correctly, performance
measurements of a submission should yield the same results. Thanks to this fact,
we can use the same resource limits on every worker in a hardware group.
However, limits can differ between runtime environments -- formally speaking,
limits are a function of three arguments: an assignment, a hardware group and a
runtime environment.
### Reference solutions
@todo: how to add one, how to evaluate it
The task of determining appropriate resource limits for exercises is difficult
to do correctly. To aid exercise authors and group supervisors, ReCodEx supports
assigning reference solutions to exercises. Those are example programs that
should cover the main approaches to the implementation. For example, searching
for an integer in an ordered array can be done with a linear search, or better,
using a binary search.
Reference solutions can be evaluated on demand, using a selected hardware group.
The evaluation results are stored and can be used later to determine limits. In
our example problem, we could configure the limits so that the linear
search-based program doesn't finish in time on larger inputs, but a binary
search does.
Note that separate reference solutions should be supplied for all supported
runtime environments.
### Exercise assignments
@todo: Creating instances of an exercise for a specific group of users,
capabilities of settings. Editing limits according to the reference
solution.
Evaluation process
------------------
@todo: How the evaluation process works on the Frontend side.
### Uploading files and file storage
@todo: One by one upload endpoint. Explain different types of the
Uploaded files.
### Automatic detection of the runtime environment
@todo: Users must submit correctly named files –
the extensions.
REST API implementation
-----------------------
@todo: What is the REST API, what are the basic principles –
Headers, JSON.
### Authentication and authorization scopes
@todo: How authentication works –
refreshing. Token scopes usage.
### HTTP requests handling
@todo: Router and routes with specific HTTP methods, preflight, required
headers
### HTTP responses format
@todo: Describe the JSON structure convention of success and error
responses
### Used technologies
@todo: PHP7 – –
used for routing, Presenters actions endpoints, exceptions and
ErrorPresenter, Doctrine 2 –
repositories + conventions, Communication over ZMQ –
problem with the extension and how we reported it and how to treat it in
the future when the bug is solved. Relational database –
Doctine enables us to switch the engine to a different engine if needed
### Data model
@todo: Describe the code-first approach using the Doctrine entities, how
the entities map onto the database schema (refer to the attached schemas
of entities and relational database models), describe the logical
grouping of entities and how they are related:
- user + settings + logins + ACL
- instance + licenses + groups + group membership
- exercise + assignments + localized assignments + runtime
environments + hardware groups
- submission + solution + reference solution + solution evaluation
- comment threads + comments
### API endpoints
@todo: Tell the user about the generated API reference and how the
Swagger UI can be used to access the API directly.
Web Application
---------------
@todo: What is the purpose of the web application and how it interacts
with the REST API.
### Used technologies
@todo: Briefly introduce the used technologies like React, Redux and the
build process. For further details refer to the GitHub wiki
### How to use the application
@todo: Describe the user documentation and the FAQ page.
Backend-Frontend communication protocol
=======================================
@todo: describe the exact methods and respective commands for the
communication
Initiation of a job evaluation
------------------------------
@todo: How does the Frontend initiate the evaluation and how the Backend
can accept it or decline it
Job processing progress monitoring
----------------------------------
When evaluating a job the worker sends progress messages on predefined points of
evaluation chain. The sending place can be on very beginning of the job, when
submit archive is downloaded or at the end of each simple task with its state
(completed, failed, skipped). These messages are sent to broker through existing
ZeroMQ connection. Detailed format of messages can be found on [communication
page](https://github.com/ReCodEx/wiki/wiki/Overall-architecture#commands-from-worker-to-broker).
Broker only resends received progress messages to the monitor component via
ZeroMQ socket. The output message format is the same as the input format.
Monitor parses received messages to JSON format, which is easy to work with in
JavaScript inside web application. All messages are cached (one queue per job)
and can be obtained multiple times through WebSocket communication channel. The
cache is cleared 5 minutes after receiving last message.
Publishing of the results
-------------------------
After job finish the worker packs results directory into single archive and
uploads it to the fileserver through HTTP protocol. The target URL is obtained
from API in headers on job initiation. Then "job done" notification request is
performed to API via broker. Special submissions (reference or asynchronous
submissions) are loaded immediately, other types are loaded on-demand on first
results request.
Loading results means fetching archive from fileserver, parsing the main YAML
file generated by worker and saving data to the database. Also, points are
assigned by score calculator.
User documentation
==================
Web Application
---------------
@todo: Describe different scenarios of the usage of the Web App
### Terminology
@todo: Describe the terminology: Instance, User, Group, Student,
Supervisor, Admin
### Web application requirements
@todo: Describe the requirements of running the web application (modern
web browser, enabled CSS, JavaScript, Cookies & Local storage)
### Scenario \#1: Becoming a user of ReCodEx
#### How to create a user account?
You can create an account if you click on the “*Create account*” menu
item in the left sidebar. You can choose between two types of
registration methods –
password, or pairing your new account with an existing CAS
If you decide a new “*local*” account using the “*Create ReCodEx
account*” form, you will have to provide your details and choose a
password for your account. You will later sign in using your email
address as your username and the password you select.
If you decide to use the CAS
and access your name and email stored in the system and create your
account based on this information. You can change your personal
information or email later on the “*Settings*” page.
When crating your account both ways, you must select an instance your
account will belong to by default. The instance you will select will be
most likely your university or other organization you are a member of.
#### How to get into ReCodEx?
To log in, go to the homepage of ReCodEx and in the left sidebar choose
the menu item “*Sign in*”. Then you must enter your credentials into one
of the two forms –
you should sign with your email and password in the first form called
“*Sign into ReCodEx*”. If you registered using the Charles University
Authentication Service (CAS), you should put your student’
your CAS password into the second form called “Sign into ReCodEx using
CAS
#### How do I sign out of ReCodEx?
If you don’
automatically. However, we recommend you sign out of the application
after you finished your interaction with it. The logout button is placed
in the top section of the left sidebar right under your name. You will
have to expand the sidebar with a button next to the “*ReCodEx*” title
(shown in the picture below).
@todo: Simon's image
#### What to do when you cannot remember your password?
If you can’ ’
authentication, then you can reset your password. You will find a link
saying “*You cannot remember what your password was? Reset your
password.*” under the sign in form. After you click on this link, you
will be asked to submit your email address. An email with a link
containing a special token will be sent to the address you fill in. We
make sure that the person who requested password resetting is really
you. When you click on the link (or you copy & paste it into your web
browser) you will be able to select a new password for your account. The
token is valid only for a couple of minutes, so do not forget to reset
the password as soon as possible, or you will have to request a new link
with a valid token.
If you sign in through CAS
provided by the administrators of the service described on their
website.
#### How to configure your account?
There are several options you have to edit your user account.
- changing your personal information (i.e., name)
- changing your credentials (email and password)
- updating your preferences (e.g., source code viewer/editor settings,
default language)
You can access the settings page through the “*Settings*” button right
under your name in the left sidebar.
### Scenario \#2: User is a student
@todo: describe what it means to be a “student” and what are the
student’
#### How to join a group for my class?
@todo: How to join a specific group
#### Which assignments do I have to solve?
@todo: Where the student can find the list of the assignment he is
expected to solve, what is the first and second deadline.
#### Where can I see details of my classes’
@todo: Where can the user see groups description and details, what
information is available.
#### How to submit a solution of an assignment?
@todo: How does a student submit his solution through the web app
#### Where are the results of my solutions?
@todo: When the results are ready and what the results mean and what to
do about them, when the user is convinced, that his solution is correct
although the results say different
#### How can I discuss my solution with my teacher/group’
@todo: Describe the comments thread behavior (public/private comments),
who else can see the comments, how notifications work (*not implemented
yet*!).
### Scenario \#3: User is supervisor of a group
@todo: describe what it means to be a “supervisor” of a group and what
are the supervisors rights
#### How do I become a supervisor of a group?
@todo: How does a user become a supervisor of a group?
#### How to add or remove a student to my group?
@todo: How to add a specific student to a given group
#### How do I add another supervisor to my group?
@todo: who can add another supervisor, what would be the rights of the
second supervisor
#### How do I create a subgroup of my group?
@todo: What it means to create a subgroup and how to do it.
#### How do I assign an exercise to my students?
@todo: Describe how to access the database of the exercises and what are
the possibilities of assignment setup –
score configuration, limits
#### How do I configure the limits of an assignment and how to choose appropriate limits?
@todo: Describe the form and explain the concept of reference solutions.
How to evaluate the reference solutions for the exercise right now (to
get the up-to-date information).
#### How can I assign some exercises only to some students of the group?
@todo: Describe how to achieve this using subgroups
#### How can I see my students’
@todo Describe where all the students’
can be found, where to look for all solutions of a given student, how to
see results of a specific student’ ’
#### Can I assign points to my students’
@todo If and how to change the score of a solution –
settings, setting points, bonus points, accepting a solution (*not
implemented yet!*). Describe how the student and supervisor will still
be able to see the percentage received from the automatic scoring, but
the awarded points will be overridden.
#### How can I discuss student’
@todo: Describe the comments thread behavior (public/private comments),
who else can see the comments -- same as from the student perspective
### Writing job configuration
To run and evaluate an exercise the backend needs to know the steps how to do
that. This is different for each environment (operation system, programming
language, etc.), so each of the environments needs to have separate
configuration.
Backend works with a powerful, but quite low level description of simple
connected tasks written in YAML syntax. More about the syntax and general task
overview can be found on [separate
page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned
features was user friendly configuration editor, but due to tight deadline and
team composition it did not make it to the first release. However, writing
configuration in the basic format will be always available and allows users to
use the full expressive power of the system.
This section walks through creation of job configuration for _hello world_
exercise. The goal is to compile file _source.c_ and check if it prints `Hello
World!` to the standard output. This is the only test case, let's call it
**A**.
The problem can be split into several tasks:
- compile _source.c_ into _helloworld_ with `/usr/bin/gcc`
- run _helloworld_ and save standard output into _out.txt_
- fetch predefined output (suppose it is already uploaded to fileserver) with
hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_
- compare _out.txt_ and _reference.txt_ by `/usr/bin/diff`
The absolute path of tools can be obtained from system administrator. However,
`/usr/bin/gcc` is location, where the GCC binary is available almost everywhere,
so location of some tools can be (professionally) guessed.
First, write header of the job to the configuration file.
```{.yml}
submission:
job-id: hello-word-job
hw-groups:
- group1
```
Basically it means, that the job _hello-world-job_ needs to be run on workers
that belong to the `group_1` hardware group . Reference files are downloaded
from the default location configured in API (such as
`http://localhost:9999/exercises` ) if not stated explicitly otherwise. Job
execution log will not be saved to result archive.
Next the tasks have to be constructed under _tasks_ section. In this demo job,
every task depends only on previous one. The first task has input file
_source.c_ (if submitted by user) already available in working directory, so
just call the GCC. Compilation is run in sandbox as any other external program
and should have relaxed time and memory limits. In this scenario, worker
defaults are used. If compilation fails, the whole job is immediately terminated
(because the _fatal-failure_ bit is set). Because _bound-directories_ option in
sandbox limits section is mostly shared between all tasks, it can be set in
worker configuration instead of job configuration (suppose this for following
tasks). For configuration of workers please contact your administrator.
```{.yml}
- task-id: "compilation"
type: "initiation"
fatal-failure: true
cmd:
bin: "/usr/bin/gcc"
args:
- "source.c"
- "-o"
- "helloworld"
sandbox:
name: "isolate"
limits:
- hw-group-id: group1
chdir: ${EVAL_DIR}
bound-directories:
- src: ${SOURCE_DIR}
dst: ${EVAL_DIR}
mode: RW
```
The compiled program is executed with time and memory limit set and the standard
output is redirected to a file. This task depends on _compilation_ task, because
the program cannot be executed without being compiled first. It is important to
mark this task with _execution_ type, so exceeded limits will be reported in
frontend.
Time and memory limits set directly for a task have higher priority than worker
defaults. One important constraint is, that these limits cannot exceed limits
set by workers. Worker defaults are present as a safety measure so that a
malformed job configuration cannot block the worker forever. Worker default
limits should be reasonably high, like a gigabyte of memory and several hours of
execution time. For exact numbers please contact your administrator.
It is important to know that if the output of a program (both standard and
error) is redirected to a file, the sandbox disk quotas apply to that file, as
well as the files created directly by the program. In case the outputs are
ignored, they are redirected to `/dev/null` , which means there is no limit on
the output length (as long as the printing fits in the time limit).
```{.yml}
- task-id: "execution_1"
test-id: "A"
type: "execution"
dependencies:
- compilation
cmd:
bin: "helloworld"
sandbox:
name: "isolate"
stdout: ${EVAL_DIR}/out.txt
limits:
- hw-group-id: group1
chdir: ${EVAL_DIR}
time: 0.5
memory: 8192
```
Fetch sample solution from file server. Base URL of file server is in the header
of the job configuration, so only the name of required file (its `sha1sum` in
our case) is necessary.
```{.yml}
- task-id: "fetch_solution_1"
test-id: "A"
dependencies:
- execution
cmd:
bin: "fetch"
args:
- "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b"
- "${SOURCE_DIR}/reference.txt"
```
Comparison of results is quite straightforward. It is important to set the task
type to _evaluation_ , so that the return code is set to 0 if the program is
correct and 1 otherwise. We do not set our own limits, so the default limits are
used.
```{.yml}
- task-id: "judge_1"
test-id: "A"
type: "evaluation"
dependencies:
- fetch_solution_1
cmd:
bin: "/usr/bin/diff"
args:
- "out.txt"
- "reference.txt"
sandbox:
name: "isolate"
limits:
- hw-group-id: group1
chdir: ${EVAL_DIR}
```
<!-- -
// vim: set textwidth=80:
-->
