Introduction
============

Generally, there are a lot of different ways and opinions on how to teach people
something new. However, most people agree that a hands-on experience is one of
the best ways to make the human brain remember a new skill. Learning must be
entertaining and interactive, with fast and frequent feedback. Some kinds of
knowledge are more suitable for this practical type of learning than others, and
fortunately, programming is one of them.

University education system is one of the areas where this knowledge can be
applied. In computer programming, there are several requirements such as the
code being syntactically correct, efficient and easy to read, maintain and
extend. Correctness and efficiency can be tested automatically to help teachers
save time for their research, but checking for bad design, habits and mistakes
is really hard to automate and requires manpower.

Checking programs written by students takes a lot of time and requires a lot of
mechanical, repetitive work. The first idea of an automatic evaluation system
comes from Stanford University profesors in 1965. They implemented a system
which evaluated code in Algol submitted on punch cards. In following years, many
similar products were written.

There are two basic ways of automatically evaluating code -- statically (check
the code without running it; safe, but not much precise) or dynamically (run the
code on testing inputs with checking the outputs against reference ones; needs
sandboxing, but provides good real world experience).

This project focuses on the machine-controlled part of source code evaluation.
First, problems of present software at our university were discussed and similar
projects at other educational institutions were examined. With acquired
knowledge from such projects in production, we set up goals for the new
evaluation system, designed the architecture and implemented a fully operational
solution. The system is now ready for production testing at our university.

Analysis
--------

### Current solution at MFF UK

The ideas presented above are not completely new. There was a group of students, 
who already implemented an evaluation solution for student's homeworks in 2006. 
Its name is [CodEx - The Code Examiner](http://codex.ms.mff.cuni.cz/project/) 
and it has been used with some improvements since then. The original plan was to 
use the system only for basic programming courses, but there is demand for 
adapting it for many different subjects.

CodEx is based on dynamic analysis. It features a web-based interface, where 
supervisors assign exercises to their students and the students have a time 
window to submit the solution. Each solution is compiled and run in sandbox 
(MO-Eval). The metrics which are checked are: corectness of the output, time and 
memory limits. It supports programs written in C, C++, C#, Java, Pascal, Python 
and Haskell.

Current system is old, but robust. There were no major security incidents
during its production usage. However, from today's perspective there are
several drawbacks. The main ones are:

- **web interface** -- The web interface is simple and fully functional. But 
  rapid development in web technologies opens new horizons of how web interface 
  can be made.
- **web api** -- CodEx offers a very limited XML API based on outdated 
  technologies that is not sufficient for users who would like to create custom 
  interfaces such as a command line tool or mobile application.
- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system 
  calls and blocking the bad ones. This can be easily done for single-threaded 
  applications, but proves difficult with multi-threaded ones. In present day,
  parallelism is a very important area of computing, so there is requirement to 
  test multi-threaded applications too.
- **instances** -- Different ways of CodEx usage scenarios requires separate 
  instances (Programming I and II, Java, C#, etc.). This configuration is not 
  user friendly (students have to register in each instance separately) and 
  burdens administrators with unnecessary work. CodEx architecture does not 
  allow sharing hardware between instances, which results in an inefficient use 
  of hardware for evaluation.
- **task extensibility** -- There is a need to test and evaluate complicated 
  programs for classes such as Parallel programming or Compiler principles, 
  which have a more difficult evaluation chain than simple 
  compilation/execution/evaluation provided by CodEx.

Supervisors and current users of CodEx gives us following ideas for improvements:

- keep it simple
- localization (both UI and exercises)
- Markdown support for exercise texts
- tagging exercises and search by tags
- comments, comments, comments (exercises, tests, solutions, ...)
- edit student solution and privately resubmit it
- resubmit solution with saving all results
- mark one student solution as accepted (used for grading this assignment)
- command-line submit tool
- SIS integration for personal user data
- plagiarism detection

After considering all these facts, it is clear that CodEx cannot be used
anymore. The project is too old to just maintain it and extend for modern
technologies. Thus, it needs to be completely rewritten or another solution
must be found.

### Related projects

First of all, some code evaluating projects were found and examined. It is not
a complete list of such evaluators, but just a few projects which are used
these days and can be an inspiration for our project. Each project from the
list has a brief description and some key features mentioned.

#### Progtest

[Progtest](https://progtest.fit.cvut.cz/) is private project from FIT ČVUT in 
Prague. As far as we know it is used for C/C++, Bash programming and 
knowledge-based quizzes. There are several bonus points and penalties and also a 
few hints what is failing in submitted solution. It is very strict on source 
code quality, for example `-pedantic` option of GCC, Valgrind for memory leaks 
or array boundaries checks via `mudflap` library.

#### Codility

[Codility](https://codility.com/) is web based solution primary targeted to
company recruiters. It is commercial product of SaaS type supporting 16
programming languages. The
[UI](http://1.bp.blogspot.com/-_isqWtuEvvY/U8_SbkUMP-I/AAAAAAAAAL0/Hup_amNYU2s/s1600/cui.png)
of Codility is [opensource](https://github.com/Codility/cui), the rest of
source code is not available. One interesting feature is 'task timeline' --
captured progress of writing code for each user.

#### CMS

[CMS](http://cms-dev.github.io/index.html) is an opensource distributed system 
for running and organizing programming contests. It is written in Python and 
contain several modules. CMS supports C/C++, Pascal, Python, PHP and Java. 
PostgreSQL is a single point of failure, all modules heavily depend on database 
connection. Task evaluation can be only three step pipeline -- compilation, 
execution, evaluation. Execution is performed in 
[Isolate](https://github.com/ioi/isolate), sandbox written by consultant of our 
project, Mgr. Martin Mareš, Ph.D.

#### MOE

[MOE](http://www.ucw.cz/moe/) is a grading system written in Shell scripts, C 
and Python. It does not provide a default GUI interface, all actions have to be 
performed from command line. The system does not evaluate submissions in real 
time, results are computed in batch mode after exercise deadline, using Isolate 
for sandboxing. Parts of MOE are used in other systems like CodEx or CMS, but 
the system is generally obsolete.

#### Kattis

[Kattis](http://www.kattis.com/) is another SaaS solution. It provides a clean 
and functional web UI, but the rest of the application is too simple. A nice 
feature is the usage of a [standardized 
format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for 
exercises. Kattis is primarily used by programming contest organizators, company 
recruiters and also some universities.


### ReCodEx goals

From the research above, we set up several goals, which a new system should
have. They mostly reflect drawbacks of current version of CodEx and wishes of
MFF users. No existing tool fits our needs, for example no examined project
provides complex execution/evaluation pipeline to support needs of courses like
Compiler principles. Modifying CodEx is also not an option -- the required
scope of a new solution is too big. To sum up, a new evaluation system has to
be written, with only small parts of reused code from CodEx (for example
judges).

The new project is **ReCodEx -- ReCodEx Code Examiner**. The name should point
to CodEx, previous evaluation solution, but also reflect new approach to solve
issues. **Re** as part of the name means redesigned, rewritten, renewed or
restarted.

Official assignment of the project is available at [web of software project
committee](http://www.ksi.mff.cuni.cz/sw-projekty/zadani/recodex.pdf) (only in
Czech). Most notable features are following:

- modern HTML5 web frontend written in Javascript using a suitable framework
- REST API implemented in PHP, communicating with database, backend and file
  server
- backend is implemented as distributed system on top of message queue framework
  (ZeroMQ) with master-worker architecture
- worker with basic support of Windows environment (without sandbox, no general
  purpose suitable tool available yet)
- evaluation procedure configured in YAML file, compound of small tasks
  connected into arbitrary oriented acyclic graph


Structure of the project
------------------------

The ReCodEx project is divided into two logical parts – the *Backend*
and the *Frontend* – which interact which each other and which cover the
whole area of code examination. Both of these logical parts are
independent of each other in the sense of being installed on separate
machines on different locations and that one of the parts can be
replaced with different implementation and as long as the communication
protocols are preserved, the system will continue to work as expected.

*Backend* is the part which is responsible solely for the process of
evaluation a solution of an exercise. Each evaluation of a solution is
referred to as a *job*. For each job, the system expects a configuration
document of the job, supplementary files for the exercise (e.g., test
inputs, expected outputs, predefined header files), and the solution of
the exercise (typically source codes created by a student). There might
be some specific requirements for the job, such as a specific runtime
environment, specific version of a compiler or the job must be evaluated
on a processor with a specific number of cores. The backend
infrastructure decides whether it will accept a job or decline it based
on the specified requirements. In case it accepts the job, it will be
placed in a queue and processed as soon as possible. The backend
publishes the progress of processing of the queued jobs and the results
of the evaluations can be queried after the job processing is finished.
The backend produces a log of the evaluation and scores the solution
based on the job configuration document.

*Frontend* on the other hand is responsible for the communication with
the users and provides them a convenient access to the Backend
infrastructure. The Frontend manages user accounts and gathers them into
units called groups. There is a database of exercises which can be
assigned to the groups and the users of these groups can submit their
solutions for these assignments. The Frontend will initiate evaluation
of these solutions by the Backend and it will store the results
afterwards. The results will be visible to authorized users and the
results will be awarded with points according to the score given by the
Backend in the evaluation process. The supervisors of the groups can
edit the parameters of the assignments, review the solutions and the
evaluations in detail and award the solutions with bonus points (both
positive and negative) and discuss about the solution with the author of
the solution. Some of the users can be entitled to create new exercises
and extend the database of exercises which can be assigned to the groups
later on.

The Frontend developed as part of this project was created with the
needs of the Faculty of Mathematics and Physics of the Charles
university in Prague in mind. The users are the students and their
teachers, groups correspond to the different courses, the teachers are
the supervisors of these groups. We believe that this model is
applicable to the needs of other universities, schools, and IT
companies, which can use the same system for their needs. It is also
possible to develop their own frontend with their own user management
system for their specific needs and use the possibilities of the Backend
without any changes, as was mentioned in the previous paragraphs.

In the latter parts of the documentation, both of the Backend and
Frontend parts will be introduced separately and covered in more detail.
The communication protocol between these two logical parts will be
described as well.

The Backend
===========

The backend is the part which is hidden to the user and which has only
one purpose: evaluate user’s solutions of their assignments.

@todo: describe the configuration inputs of the Backend

@todo: describe the outputs of the Backend

@todo: describe the inner parts of the Backend (and refer to the Wiki
for the technical description of the components)

@todo: describe how the backend receives the inputs and how it
communicates the results

The Frontend
============

The frontend is the part which is visible to the user of ReCodEx and
which holds the state of the system – the user accounts, their roles in
the system, the database of exercises, the assignments of these
exercises to groups of users (i.e., students), and the solutions and
evaluations of them.

Frontend is split into three parts:

-   the server-side REST API (“API”) which holds the business logic and
    keeps the state of the system consistent

-   the relational database (“DB”) which persists the state of the
    system

-   the client side application (“client”) which simplifies access to
    the API for the common users

The centerpiece of this architecture is the API. This component receives
requests from the users and from the Backend, validates them and
modifies the state of the system and persists this modified state in the
DB.

We have created a web application which can communicate with the API
server and present the information received from the server to the user
in a convenient way. The client can be though any application, which can
send HTTP requests and receive the HTTP responses. Users can use general
applications like [cURL](https://github.com/curl/curl/),
[Postman](https://www.getpostman.com/), or create their own specific
client for ReCodEx API.

Frontend capabilities
---------------------

@todo: describe what the frontend is capable of and how it really works,
what are the limitations and how it can be extended

Terminology
-----------

This project was created for the needs of a university and this fact is
reflected into the terminology used throughout the Frontend. A list of
important terms’ definitions follows to make the meaning unambiguous.

### User and user roles

*User* is a person who uses the application. User is granted access to
the application once he or she creates an account directly through the
API or the web application. There are several types of user accounts
depending on the set of permissions – a so called “role” – they have
been granted. Each user receives only the most basic set of permissions
after he or she creates an account and this role can be changed only by
the administrators of the service:

-   *Student* is the most basic role. Student can become member of a
    group and submit his solutions to his assignments.

-   *Supervisor* can be entitled to manage a group of students.
    Supervisor can assign exercises to the students who are members of
    his groups and review their solutions submitted to
    these assignments.

-   *Super-admin* is a user with unlimited rights. This user can perform
    any action in the system.

There are two implicit changes of roles:

-   Once a *student* is added to a group as its supervisor, his role is
    upgraded to a *supervisor* role.

-   Once a *supervisor* is removed from the lasts group where he is a
    supervisor then his role is downgraded to a *student* role.

These mechanisms do not prevent a single user being a supervisor of one
group and student of a different group as supervisors’ permissions are
superset of students’ permissions.

### Login

*Login* is a set of user’s credentials he must submit to verify he can
be allowed to access the system as a specific user. We distinguish two
types of logins: local and external.

-   *Local login* is user’s email address and a password he chooses
    during registration.

-   *External login* is a mapping of a user profile to an account of
    some authentication service (e.g., [CAS](https://ldap1.cuni.cz/)).

### Instance

*An instance* of ReCodEx is in fact just a set of groups and user
accounts. An instance should correspond to a real entity as a
university, a high-school, an IT company or an HR agency. This approach
enables the system to be shared by multiple independent organizations
without interfering with each other.

Usage of the system by the users of an instance can be limited by
possessing a valid license. It is up to the administrators of the system
to determine the conditions under which they will assign licenses to the
instances.

### Group

*Group* corresponds to a school class or some other unit which gathers
users who will be assigned the same set exercises. Each group can have
multiple supervisors who can manage the students and the list of
assignments.

Groups can form a tree hierarchy of arbitrary depth. This is inspired by the 
hierarchy of school classes belonging to the same subject over several school 
years. For example, there can be a top level group for a programming class that 
contains subgroups for every school year. These groups can then by divided into 
actual student groups with respect to lab attendance. Supervisors can create 
subgroups of their groups and further manage these subgroups.

### Exercise

*An exercise* consists of textual assignment of a task and a definition
of how a solution to this exercise should be processed and evaluated in
a specific runtime environment (i.e., how to compile a submitted source
code and how to test the correctness of the program). It is a template
which can be instantiated as an *assignment* by a supervisor of a group.

### Assignment

An assignment is an instance of an *exercise* assigned to a specific
*group*. An assignment can modify the text of the task assignment and it
has some additional information which is specific to the group (e.g., a
deadline, the number of points gained for a correct solution, additional
hints for the students in the assignment). The text of the assignment
can be edited and supervisors can translate the assignment into another
language.

### Solution

*A solution* is a set of files which a user submits to a given
*assignment*.

### Submission

*A submission* corresponds to a *solution* being evaluated by the
Backend. A single *solution* can be submitted repeatedly (e.g., when the
Backend encounters an error or when the supervisor changes the assignment).

### Evaluation

*An evaluation* is the processed report received from the Backend after
a *submission* is processed. Evaluation contains points given to the
user based on the quality of his solution measured by the Backend and
the settings of the assignment. Supervisors can review the evaluation
and add bonus points (both positive and negative) if the student
deserves some.

### Runtime environment

*A runtime environment* defines the used programming language or tools
which are needed to process and evaluate a solution. Examples of a
runtime environment can be “*Linux + GCC*”, “*Linux + Mono*”, “*Windows
+ .NET 4*”, “*Bison + Yacc.”*

### Limits

A correct *solution* of an *assignment* has to pass all specified tests (mostly 
checks that it yields the correct output for various inputs) and typically must 
also be effective in some sense. The Backend measures the time and memory 
consumption of the solution while running. This consumption of resources can be 
*limited* and the solution will receive fewer points if it exceeds the given 
limits in some test cases defined by the *exercise*.

User management
---------------

@todo: roles and their rights, adding/removing different users, how the
role of a specific user changes

Instances and hierarchy of groups
---------------------------------

@todo: What is an instance, how to create one, what are the licenses and
how do they work. Why can the groups form hierarchies and what are the
benefits – what it means to be an admin of a group, hierarchy of roles
in the group hierarchy.

Exercises database
------------------

@todo: How the exercises are stored, accessed, who can edit what

### Creating a new exercise

@todo Localized assignments, default settings

### Runtime environments and hardware groups

@todo read this later and see if it still makes sense

ReCodEx is designed to utilize a rather diverse set of workers -- there can be 
differences in many aspects, such as the actual hardware running the worker 
(which impacts the results of measuring) or installed compilers, interpreters 
and other tools needed for evaluation. To address these two examples in 
particular, we assign runtime environments and hardware groups to exercises.

The purpose of runtime environments is to specify which tools (and often also 
operating system) are required to evaluate a solution of the exercise -- for 
example, a C# programming exercise can be evaluated on a Linux worker running 
Mono or a Windows worker with the .NET runtime. Such exercise would be assigned 
two runtime environments, `Linux+Mono` and `Windows+.NET` (the environment names 
are arbitrary strings configured by the administrator).

A hardware group is a set of workers that run on similar hardware (e.g. a 
particular quad-core processor model and a SSD hard drive). Workers are assigned 
to these groups by the administrator. If this is done correctly, performance 
measurements of a submission should yield the same results. Thanks to this fact, 
we can use the same resource limits on every worker in a hardware group. 
However, limits can differ between runtime environments -- formally speaking, 
limits are a function of three arguments: an assignment, a hardware group and a 
runtime environment.

### Reference solutions

@todo: how to add one, how to evaluate it

The task of determining appropriate resource limits for exercises is difficult 
to do correctly. To aid exercise authors and group supervisors, ReCodEx supports 
assigning reference solutions to exercises. Those are example programs that 
should cover the main approaches to the implementation. For example, searching 
for an integer in an ordered array can be done with a linear search, or better, 
using a binary search.

Reference solutions can be evaluated on demand, using a selected hardware group. 
The evaluation results are stored and can be used later to determine limits. In 
our example problem, we could configure the limits so that the linear 
search-based program doesn't finish in time on larger inputs, but a binary 
search does.

Note that separate reference solutions should be supplied for all supported 
runtime environments.

### Exercise assignments

@todo: Creating instances of an exercise for a specific group of users,
capabilities of settings. Editing limits according to the reference
solution.

Evaluation process
------------------

@todo: How the evaluation process works on the Frontend side.

### Uploading files and file storage

@todo: One by one upload endpoint. Explain different types of the
Uploaded files.

### Automatic detection of the runtime environment

@todo: Users must submit correctly named files – assuming the RTE from
the extensions.

REST API implementation
-----------------------

@todo: What is the REST API, what are the basic principles – GET, POST,
Headers, JSON.

### Authentication and authorization scopes

@todo: How authentication works – signed JWT, headers, expiration,
refreshing. Token scopes usage.

### HTTP requests handling

@todo: Router and routes with specific HTTP methods, preflight, required
headers

### HTTP responses format

@todo: Describe the JSON structure convention of success and error
responses

### Used technologies

@todo: PHP7 – how it is used for typehints, Nette framework – how it is
used for routing, Presenters actions endpoints, exceptions and
ErrorPresenter, Doctrine 2 – database abstraction, entities and
repositories + conventions, Communication over ZMQ – describe the
problem with the extension and how we reported it and how to treat it in
the future when the bug is solved. Relational database – we use MariaDB,
Doctine enables us to switch the engine to a different engine if needed

### Data model

@todo: Describe the code-first approach using the Doctrine entities, how
the entities map onto the database schema (refer to the attached schemas
of entities and relational database models), describe the logical
grouping of entities and how they are related:

-   user + settings + logins + ACL

-   instance + licences + groups + group membership

-   exercise + assignments + localized assignments + runtime
    environments + hardware groups

-   submission + solution + reference solution + solution evaluation

-   comment threads + comments

### API endpoints

@todo: Tell the user about the generated API reference and how the
Swagger UI can be used to access the API directly.

Web Application
---------------

@todo: What is the purpose of the web application and how it interacts
with the REST API.

### Used technologies

@todo: Briefly introduce the used technologies like React, Redux and the
build process. For further details refer to the GitHub wiki

### How to use the application

@todo: Describe the user documentation and the FAQ page.

Backend-Frontend communication protocol
=======================================

@todo: describe the exact methods and respective commands for the
communication

Initiation of a job evaluation
------------------------------

@todo: How does the Frontend initiate the evaluation and how the Backend
can accept it or decline it

Job processing progress monitoring
----------------------------------

@todo: How does the Backend push the progress of evaluation through the
Monitor to the Frontend

Publishing of the results
-------------------------

@todo: How does the Backend publish the results and how it notifies the
Frontend; How the Frontend can request the results if the notification
does not arrive

User documentation
==================

Web Application
---------------

@todo: Describe different scenarios of the usage of the Web App

### Terminology

@todo: Describe the terminology: Instance, User, Group, Student,
Supervisor, Admin

### Web application requirements

@todo: Describe the requirements of running the web application (modern
web browser, enabled CSS, JavaScript, Cookies & Local storage)

### Scenario \#1: Becoming a user of ReCodEx

#### How to create a user account?

You can create an account if you click on the “*Create account*” menu
item in the left sidebar. You can choose between two types of
registration methods – by creating a local account with a specific
password, or pairing your new account with an existing CAS UK account.

If you decide a new “*local*” account using the “*Create ReCodEx
account*” form, you will have to provide your details and choose a
password for your account. You will later sign in using your email
address as your username and the password you select.

If you decide to use the CAS UK, then we will verify your credentials
and access your name and email stored in the system and create your
account based on this information. You can change your personal
information or email later on the “*Settings*” page.

When crating your account both ways, you must select an instance your
account will belong to by default. The instance you will select will be
most likely your university or other organization you are a member of.

#### How to get into ReCodEx?

To log in, go to the homepage of ReCodEx and in the left sidebar choose
the menu item “*Sign in*”. Then you must enter your credentials into one
of the two forms – if you selected a password during registration, then
you should sign with your email and password in the first form called
“*Sign into ReCodEx*”. If you registered using the Charles University
Authentication Service (CAS), you should put your student’s number and
your CAS password into the second form called “Sign into ReCodEx using
CAS UK”.

#### How do I sign out of ReCodEx?

If you don’t use ReCodEx for a whole day, you will be logged out
automatically. However, we recommend you sign out of the application
after you finished your interaction with it. The logout button is placed
in the top section of the left sidebar right under your name. You will
have to expand the sidebar with a button next to the “*ReCodEx*” title
(shown in the picture below).

![](media/image1.png){width="3.3131944444444446in"
height="0.6173611111111111in"}

#### What to do when you cannot remember your password?

If you can’t remember your password and you don’t use CAS UK
authentication, then you can reset your password. You will find a link
saying “*You cannot remember what your password was? Reset your
password.*” under the sign in form. After you click on this link, you
will be asked to submit your email address. An email with a link
containing a special token will be sent to the address you fill in. We
make sure that the person who requested password resetting is really
you. When you click on the link (or you copy & paste it into your web
browser) you will be able to select a new password for your account. The
token is valid only for a couple of minutes, so do not forget to reset
the password as soon as possible, or you will have to request a new link
with a valid token.

If you sign in through CAS UK, then please follow the instructions
provided by the administrators of the service described on their
website.

#### How to configure your account?

There are several options you have to edit your user account.

-   changing your personal information (i.e., name)

-   changing your credentials (email and password)

-   updating your preferences (e.g., source code viewer/editor settings,
    default language)

You can access the settings page through the “*Settings*” button right
under your name in the left sidebar.

### Scenario \#2: User is a student

@todo: describe what it means to be a “student” and what are the
student’s rights

#### How to join a group for my class?

@todo: How to join a specific group

#### Which assignments do I have to solve? 

@todo: Where the student can find the list of the assignment he is
expected to solve, what is the first and second deadline.

#### Where can I see details of my classes’ group?

@todo: Where can the user see groups description and details, what
information is available.

#### How to submit a solution of an assignment?

@todo: How does a student submit his solution through the web app

#### Where are the results of my solutions?

@todo: When the results are ready and what the results mean and what to
do about them, when the user is convinced, that his solution is correct
although the results say different

#### How can I discuss my solution with my teacher/group’s supervisor directly through the web application?

@todo: Describe the comments thread behavior (public/private comments),
who else can see the comments, how notifications work (*not implemented
yet*!).

### Scenario \#3: User is supervisor of a group

@todo: describe what it means to be a “supervisor” of a group and what
are the supervisors rights

#### How do I become a supervisor of a group?

@todo: How does a user become a supervisor of a group?

#### How to add or remove a student to my group?

@todo: How to add a specific student to a given group

#### How do I add another supervisor to my group?

@todo: who can add another supervisor, what would be the rights of the
second supervisor

#### How do I create a subgroup of my group?

@todo: What it means to create a subgroup and how to do it.

#### How do I assign an exercise to my students?

@todo: Describe how to access the database of the exercises and what are
the possibilities of assignment setup – availability, deadlines, points,
score configuration, limits

#### How do I configure the limits of an assignment and how to choose appropriate limits?

@todo: Describe the form and explain the concept of reference solutions.
How to evaluate the reference solutions for the exercise right now (to
get the up-to-date information).

#### How can I assign some exercises only to some students of the group?

@todo: Describe how to achieve this using subgroups

#### How can I see my students’ solutions?

@todo Describe where all the students’ solutions for a given assignment
can be found, where to look for all solutions of a given student, how to
see results of a specific student’s solution’s evaluation result.

#### Can I assign points to my students’ solutions manually instead of depending on automatic scoring?

@todo If and how to change the score of a solution – assignment
settings, setting points, bonus points, accepting a solution (*not
implemented yet!*). Describe how the student and supervisor will still
be able to see the percentage received from the automatic scoring, but
the awarded points will be overridden.

#### How can I discuss student’s solution with him/her directly through the web application?

@todo: Describe the comments thread behavior (public/private comments),
who else can see the comments


### Writing job configuration

To run and evaluate an exercise the backend needs to know the steps how to do
that. This is different for each environment (operation system, programming
language, etc.), so each of the environments needs to have separate
configuration.

Backend works with a powerful, but quite low level description of simple
connected tasks written in YAML syntax. More about the syntax and general task
overview can be found on [separate
page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned
features was user friendly configuration editor, but due to tight deadline and
team composition it did not make it to the first release. However, writing
configuration in the basic format will be always available and allows users to 
use the full expressive power of the system.

This section walks through creation of job configuration for _hello world_
exercise. The goal is to compile file _source.c_ and check if it prints `Hello 
World!` to the standard output.

The problem can be split into several tasks:

- compile _source.c_ into _helloworld_ with `/usr/bin/gcc`
- run _helloworld_ and save standard output into _out.txt_
- fetch predefined output (suppose it is already uploaded to fileserver) with
  hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_
- compare _out.txt_ and _reference.txt_ by `/usr/bin/diff`

The absolute path of tools can be obtained from system administrator. However,
`/usr/bin/gcc` is location, where the GCC binary is available almost everywhere,
so location of some tools can be (professionally) guessed.

First, write header of the job to the configuration file. 

```{.yml}
submission:
    job-id: hello-word-job
    hw-groups:
        - group1
```

Basically it means, that the job _hello-world-job_ needs to be run on workers
with capabilities of _group1_ group. Reference files are downloaded from the
default location configured in API (probably `http://localhost:9999/exercises`)
if not stated explicitly otherwise. Job execution log will not be saved to
result archive.

Next the tasks have to be constructed under _tasks_ section. In this demo job,
every task depends only on previous one. The first task has input file
_source.c_ (if submitted by user) already available in working directory, so
just call the GCC. Compilation is run in sandbox as any other external program
and should have relaxed time and memory limits. In this scenarion, worker
defaults are used. If compilation fails, whole job is immediately terminated
(_fatal-failure_ bit set). Because _bound-directories_ option in sandbox limits
section is mostly common for all tasks, it can be set in worker configuration
instead of job configuration (suppose this for following tasks). For
configuration of workers please contact your administrator.

```{.yml}
- task-id: "compilation"
  type: "initiation"
  fatal-failure: true
  cmd:
      bin: "/usr/bin/gcc"
      args:
          - "source.c"
          - "-o"
          - "helloworld"
  sandbox:
      name: "isolate"
      limits:
          - hw-group-id: group1
            chdir: ${EVAL_DIR}
            bound-directories:
                - src: ${SOURCE_DIR}
                  dst: ${EVAL_DIR}
                  mode: RW
```

The compiled program is executed with time and memory limit set and standard
output redirected to a file. This task depends on _compilation_ task, because
the program cannot be executed without being compiled first. It is important to
mark this task with _execution_ type, so exceeded limits will be reported in
frontend.

Time and memory limits set directly for a task have higher priority than worker
defaults. One important constraint is, that these limits cannot exceed limits
set by workers. Worker defaults are present as a safety for the sake of
possibility that wrong job configuration can block whole worker forever.  Worker
default limits should be set reasonably high, like gigabyte of memory and couple
hours of execution time. For exact numbers please contact your administrator.

It is good point to remind here, that if output of a program (both standard and
error) is redirected to a file, the sandbox disk quotas holds for these files as
well as for files created directly by the program. But if outputs are ignored,
they are redirected to `/dev/null` file where arbitrary amount of data can be
written.

```{.yml}
- task-id: "execution_1"
  test-id: "A"
  type: "execution"
  dependencies:
      - compilation
  cmd:
      bin: "helloworld"
  sandbox:
      name: "isolate"
	  stdout: ${EVAL_DIR}/out.txt
      limits:
          - hw-group-id: group1
            chdir: ${EVAL_DIR}
            time: 0.5
            memory: 8192
```

Fetch sample solution from fileserver. Base URL of fileserver is in job header,
so only the name of required file (`sha1sum` in our case) is necessary.

```{.yml}
- task-id: "fetch_solution_1"
  test-id: "A"
  dependencies:
      - execution
  cmd:
      bin: "fetch"
      args:
          - "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b"
          - "${SOURCE_DIR}/reference.txt"
```

Comparison of results is quite straightforward. Important is to mark the task as
_evaluation_ type, so the return code represents if the program is correct (0)
or not (other). Worker default limits are used.

```{.yml}
- task-id: "judge_1"
  test-id: "A"
  type: "evaluation"
  dependencies:
      - fetch_solution_1
  cmd:
      bin: "/usr/bin/diff"
      args:
          - "out.txt"
          - "reference.txt"
  sandbox:
      name: "isolate"
      limits:
          - hw-group-id: group1
            chdir: ${EVAL_DIR}
```