recodex-wiki/User-documentation.md

# User Documentation

Users interact with the ReCodEx through the web application. It is required to
use a modern web browser with good HTML5 and CSS3 support. Among others, cookies
and local storage are used. Also a decent JavaScript runtime must be provided by
the browser.

Supported and tested browsers are: Firefox 50+, Chrome 55+, Opera 42+ and Edge
13+. Mobile devices often have problems with internationalization and possibly
lack support for some common features of desktop browsers. In this stage of
development is not possible for us to fine tune the interface for major mobile
browsers on all mobile platforms. However, it is confirmed to work with latest
Google Chrome and Gello browser on Android 7.1+. Issues have been reported with
Firefox that will be fixed in the future. Also, it is confirmed to work with
Safari browser on iOS 10.

Usage of the web application is divided into sections concerning particular user
roles. Under these sections all possible use cases can be found. These sections
are inclusive, so more privileged users need to read instructions for all less
privileged users. Described roles are:

- Student
- Group supervisor
- Group administrator
- Instance administrator
- Superadministrator

## Terminology

**Instance** -- Represents a university, company or some other organization
 unit. Multiple instances can exist in a single ReCodEx installation.

**Group** -- A group of students to which exercises are assigned by a
 supervisor. It should typically correspond with a real world lab group.

**User** -- A person that interacts with the system using the web interface (or
 an alternative client).

**Student** -- A user with least privileges who is subscribed to some groups and
 submits solutions to exercise assignments.

**Supervisor** -- A person responsible for assigning exercises to a group and
 reviewing submissions.

**Admin** -- A person responsible for the maintenance of the system and fixing
 problems supervisors cannot solve.

**Exercise** -- An algorithmic problem that can be assigned to a group.  They
 can be shared by the teachers using an exercise database in ReCodEx.

**Assignment** -- An exercise assigned to a group, possibly with modifications.

**Runtime environment** -- Runtime environment is unique combination of platform
 (OS) and programming language runtime/compiler in specific version. Runtime
 environments are managed by the administrators to reflect abilities of whole
 system.

**Hardware group** -- Hardware group is a set of workers with similar hardware.
 Its purpose is to group workers that are likely to run a program using the same
 amount of resources. Hardware groups are managed byt the system administrators
 who have to keep them up-to-date.

## General Basics

Description of general basics which are the same for all users of ReCodEx web
application follows.

### First Steps in ReCodEx

You can create an account by clicking the "Create account" menu item in the left
sidebar. You can choose between two types of registration methods -- by creating
a local account with a specific password, or pairing your new account with an
existing CAS UK account.

If you decide to create a new local account using the "Create ReCodEx account”
form, you will have to provide your details and choose a password for your
account. Although ReCodEx allows using quite weak passwords, it is wise to use a
bit stronger ones The actual strength is shown in progress bar near the password
field during registration. You will later sign in using your email address as
your username and the password you select.

If you decide to use the CAS UK service, then ReCodEx will verify your CAS
credentials and create a new account based on information stored there (name and
email address). You can change your personal information later on the
"Settings" page.

Regardless of the desired account type, an instance it will belong to must be
selected. The instance will be most likely your university or other organization
you are a member of.

To log in, go to the homepage of ReCodEx and in the left sidebar choose the menu
item "Sign in". Then you must enter your credentials into one of the two forms
-- if you selected a password during registration, then you should sign with
your email and password in the first form called "Sign into ReCodEx". If you
registered using the Charles University Authentication Service (CAS), you should
put your student’s number and your CAS password into the second form called
"Sign into ReCodEx using CAS UK".

There are several options you can edit in your user account:

- changing your personal information (i.e., name)
- changing your credentials (email and password)
- updating your preferences (source code viewer/editor settings, default
  language)

You can access the settings page through the "Settings" button right under
your name in the left sidebar.

If you are not active in ReCodEx for a whole day, you will be logged out
automatically. However, we recommend you sign out of the application after you
finish your interaction with it. The logout button is placed in the top section
of the left sidebar right under your name. You may need to expand the sidebar
with a button next to the "ReCodEx” title (informally known as _hamburger
button_), depending on your screen size.

### Forgotten Password

If you cannot remember your password and you do not use CAS UK authentication,
then you can reset your password. You will find a link saying "Cannot remember
what your password was? Reset your password." under the sign in form. After you
click this link, you will be asked to submit your registration email address. A
message with a link containing a special token will be sent to you by e-mail --
we make sure that the person who requested password resetting is really you.
When you visit the link, you will be able to enter a new password for your
account. The token is valid only for a couple of minutes, so do not forget to
reset the password as soon as possible, or you will have to request a new link
with a valid token.

If you sign in through CAS UK, then please follow the instructions
provided by the administrators of the service described on their
website.

### Dashboard

When you log into the system you should be redirected to your "Dashboard". On
this page you can see some brief information about the groups you are member of.
The information presented there varies with your role in the system -- further
description of dashboard will be provided later on with according roles.

## Student

Student is a default role for every newly registered user. This role has quite
limited capabilities in ReCodEx. Generally, a student can only submit solutions
of exercises in some particular groups. These groups should correspond to
courses he/she attends.

On the "Dashboard" page there is "Groups you are student of" section where you
can find list of your student groups. In first column of every row there is a
brief panel describing concerning group. There is name of the group and
percentage of gained points from course. If you have enough points to
successfully complete the course then this panel has green background with tick
sign. In the second column there is a list of assigned exercises with its
deadlines. If you want to quickly get to the groups page you might want to use
provided "Show group's detail" button.

### Join Group

To be able to submit solutions you have to be a member of the right group. Each
instance has its own group hierarchy, so you can choose only those within your
instance. That is why a list of groups is available from under an instance link
located in the sidebar. This link brings you to instance detail page.

In there you can see a description of the instance and most importantly in
"Groups hierarchy" box there is a hierarchical list of all public groups in the
instance. Please note that groups with plus sign are collapsible and can be
further extended. When you find a group you would like to join, continue by
clicking on "See group's page" link following with "Join group" link.

**Note:** Some groups can be marked as private and these groups are not visible
 in hierarchy and membership cannot be established by students themselves.
 Management of students in this type of groups is in the hands of supervisors.

On the group detail page there are multiple interesting things for you. The
first one is a brief overview containing the information describing the group,
there is a list of supervisors and also the hierarchy of the subgroups. The most
important section is the "Student's dashboard" section. This section contains
the list of assignments and the list of fellow students. If the supervisors of
the group allowed students to see the statistic of their fellow students then
there will also be the number of points each of the students has gained so far.

### Start Solving Assignments

In the "Assignments" box on the group detail page there is a list of assigned
exercises which students are supposed to solve. The assignments are displayed
with their names and deadlines. There are possibly two deadlines, the first one
means that till this datetime student will receive full amount of points in case
of successful solution. Second deadline does not have to be set, but in case it
is, the maximum number of points for successful solution between these two
deadlines can be different.

An assignment link will lead you to assignment detail page where are presented
all known details about assignment. There are of course both deadlines, limit of
submissions which you can make and also full-range description of assignment,
which can be localized. The localization can be on demand switched between all
language variants in tab like box.

Further on the page you can find "Submitted solutions" box where is a list of
submissions with links to result details. But most importantly there is a
"Submit new solution" button on the assignment page which provides an interface
to submit solution of the assignment.

After clicking on submit button, dialog window will show up. In here you can
upload files representing your solution, you can even add some notes to mark the
solution. Your supervisor can also access this note. After you successfully
upload all files necessary for your solution, click the "Submit your solution"
button and let ReCodEx evaluate the solution.

During the execution ReCodEx backend might send evaluation progress state to
your browser which will be displayed in another dialog window. When the whole
execution is finished then a "See the results" button will appear and you can
look at the results of your solution.

### View Results of Submission

On the results detail page there are a lot of information. Apart from assignment
description, which is not connected to your results, there is also the solution
submitter name (supervisor can submit a solution on your behalf), further there
are files which were uploaded on submission and most importantly "Evaluation
details" and "Test results" boxes.

Evaluation details contains overall results of your solution. There are
information such as whether the solution was provided before deadlines, if the
evaluation process successfully finished or if compilation succeeded. After that
you can find a lot of values, most important one is the last, "Total score",
consisting of your score, slash and the maximum number of points for this
assignment. Interestingly the your score value can be higher than the maximum,
which is caused by "Bonus points" item above. If your solution is nice and
supervisor notices it, he/she can assign you additional points for effort. On
the other hand, points can be also subtracted for bad coding habits or even
cheating.

In test results box there is a table of all exercise tests results. Columns
represents these information:

- test case overall result, symbol of yes/no option
- test case name
- percentage of correctness of this particular test
- evaluation status, if test was successfully executed or failed
- memory limit, if supervisor allowed it then percentual memory usage is
  displayed
- time limit, if supervisor allowed it then percentual time usage is displayed

A new feature of web application is "Comments and notes" box where you can
communicate with your supervisors or just write random private notes to your
submission. Adding a note is quite simple, you just write it to text field in
the bottom of box and click on the "Send" button. The button with lock image
underneath can switch visibility of newly created comments.

In case you think the ReCodEx evaluation of your solution is wrong, please use
the comments system described above, or even better notify your supervisor by
another channel (email). Unfortunately there is currently no notification
mechanism for new comment messages.


## Group Supervisor

Group supervisor is typically the lecturer of a course. A user in this role can
modify group description and properties, assign exercises or manage list of
students. Further permissions like managing subgroups or supervisors is
available only for group administrators.

On "Dashboard" page you can find "Groups you supervise" section. Here there are
boxes representing your groups with the list of students attending course and
their points. Student names are clickable with redirection to the profile of the
user where further information about his/hers assignments and solution can be
found. To quickly jump onto groups page, use "Show group's detail" button at
the bottom of the matching group box.

### Manage Group

Locate group you supervise and you want to manage. All your supervised groups
are available in sidebar under "Groups -- supervisor" collapsible menu. If you
click on one of those you will be redirected to group detail page. In addition
to basic group information you can also see "Supervisor's controls" section. In
this section there are lists of current students and assignments.

As a supervisor of group you are able to see "Edit group settings" button
at the top of the page. Following this link will take you to group editation
page with form containing these fields:

- group name which is visible to other users
- external identification which may be used for pairing with entries in an
  information system
- description of group which will be available to users in instance (in
  Markdown)
- set if group is publicly visible (and joinable by students) or private
- options to set if students should be able see statistics of each other
- minimal points threshold which students have to gain to successfully complete
  the course

After filling all necessary fields the form can be sent by clicking on "Edit
group" button and all changes will be applied.

For students management there are "Students" and "Add student" boxes. The first
one is simple list of all students which are attending the course with the
possibility of delete them from the group. That can be done by hitting "Leave
group" button near particular user. The second box is for adding students to the
group. There is a text field for typing name of the student and after clicking
on the magnifier image or pressing enter key there will appear list of matched
users. At this moment just click on the "Join group" button and student will be
signed in to your group.

### Assigning Exercises

Before assigning an exercise, you obviously have to know what exercises are
available. A list of all exercises in the system can be found under "Exercises"
link in sidebar. This page contains a table with exercises names, difficulties
and names of the exercise authors. Further information about exercise is
available by clicking on its name.

On the exercise details page are numerous information about it. There is a box
with all possible localized descriptions and also a box with some additional
information of exercise author, its difficulty, version, etc. There is also a
description for supervisors by exercise author under "Exercise overview" option,
where some important information can be found. And most notably there is an
information about available programming languages for this exercise, under
"Supported runtime environments" section.

If you decide that the exercise is suitable for one of your groups, look for the
"Groups" box at the bottom of the page. There is a list of all groups you
supervise with an "Assign" button which will assign the exercise to the
selected group.

After clicking on the "Assign" button you should be redirected to assignment
editation page. In there you can find two forms, one for editation of assignment
meta information and the second one for setting exercise time and memory limits.

In meta information form you can fill these options:

- name of the assignment which will be visible in a group
- visibility (if an assignment is under construction then you can mark it as not
  visible and students will not see it)
- subform for localized descriptions (new localization can be added by clicking
  on "Add language variant" button, current one can be deleted with "Remove this
  language" button)
    - language of description from dropdown field (English, Czech, German)
    - description in selected language
- score configuration which will be used on students solution evaluation, you
  can find some very simple one already in here, description of score
  configuration can be found further in "Writing score configuration" chapter
- first submission deadline
- maximum points that can be gained before the first deadline; if you want to
  manage all points manually, set it to 0 and then use bonus points, which are
  described in the next subchapter
- second submission deadline, after that students still can submit exercises but
  they are given no points no points (must be after the first deadline)
- maximum points that can be gained between first deadline and second deadline
- submission count limit for the solutions of the students -- this limits the amount of
  attempts a student has at solving the problem
- visibility of memory and time ratios; if true students can see the percentage
  of used memory and time (with respect to the limit) for each test
- minimum percentage of points which each submission must gain to be considered
  correct (if it gets less, it will gain no points)
- whether the assignment is marked as bonus one and points from solving it are
  not included into group threshold limit (that means solving it can get you
  additional points over the limit)

The form has to be submitted with "Edit settings" button otherwise changes will
not be saved.

The same editation page serves also for the purpose of assignment editation, not
only creation. That is why on bottom of the page "Delete the assignment" box
can be found. Clearly the button "Delete" in there can be used to unassign
exercise from group.

The last unexplored area is the time and memory limits form. The whole form is
situated in a box with tabs which are leading to particular runtime
environments. If you wish not to use one of those, locate "Remove" button at the
bottom of the box tab which will delete this environment from the assignment.
Please note that this action is irreversible.

In general, every tab in environments box contains some basic information about
runtime environment and another nested tabbed box. In there you can find all
hardware groups which are available for the exercise and set limits for all test
cases. The time limits have to be filled in seconds (float), memory limits are
in bytes (int). If you are interested in some reference values to particular
test case then you can take a peek on collapsible "Reference solutions'
evaluations" items. If you are satisfied with changes you made to the limits,
save the form with "Change limits" button right under environments box.

### Management of The Solutions of The Students

One of the most important tasks of a group supervisor is checking student
solutions. As automatic evaluation of them cannot catch all problems in the
source code, it is advisable to do a brief manual review of the coding
style of the student and reflect that in assignment bonus points.

On "Assignment detail" page there is an "View student results" button near top
of the page (next to "Edit assignment settings" button). This will redirect you
to a page where is a list of boxes, one box per student. Each student box
contains a list of submissions for this assignment. The row structure of
submission list is the same as the structure in the "Submitted solution"
box. More information about every solution can be shown by clicking on "Show
details" link on the end of solution row.

This page is the same as for students with one exception -- there is an
additional collapsed box "Set bonus points". In unfolded state, there is an
input field for one number (positive or negative integer) and confirmation
button "Set bonus points". After filling intended amount of points and
submitting the form, the data in "Evaluation details" box get immediately
updated. To remove assigned bonus points, submit just the zero number. The bonus
points are not additive, newer value overrides older values.

It is useful to give a feedback about the solution back to the user. For this
you can use the "Comments and notes" box. Make sure that the messages are not
private, so that the student can see them. More detailed description of this box
can be nicely used the "Comments and notes" box. Make sure that the messages are
not private, so the student can see them. More detailed description of this box
is available in student part of user documentation.

One of the discussed concept was marking one solution as accepted. However, due
to lack of frontend developers it is not yet prepared in user interface. We
hope, it will be ready as soon as possible. The button for accepting a solution
will be most probably also on this page.

### Creating Exercises

Link to exercise creation can be found in exercises list which is accessible
through "Exercises" link in sidebar. On the bottom of the exercises list page
you can find "Add exercise" button which will redirect you to exercise editation
page. In this moment exercise is already created so if you just leave this page
exercise will stay in the database. This is also reason why exercise creation
form is the same as the exercise editation form.

Exercise editation page is divided into three separate forms. First one is
supposed to contain meta information about exercise, second one is used for
uploading and management of supplementary files and third one manages runtime
configuration in which exercise can be executed.

First form is located in "Edit exercise settings" and generally contains meta
information needed by frontend which are somehow somewhere visible. In here you
can define:

- exercise name which will be visible to other supervisors
- difficulty of exercise (easy, medium, hard)
- description which will be available only for visitors, may be used for further
  description of exercise (for example information about test cases and how they
  could be scored)
- private/public switch, if exercise is private then only you as author can see
  it, assign it or modify it
- subform containing localized descriptions of exercise, new one can be added
  with "Add language variant" button and current one deleted with "Remove this
  language"
    - language in which this particular description is in (Czech, English,
      German)
    - actual localized description of exercise

After all information is properly set form has to be submitted with "Edit
settings" button.

Management of supplementary files can be found in "Supplementary files" box.
Supplementary files are files which you can use further in job configurations
which have to be provided in all runtime configurations. These files are
uploaded directly to fileserver from where worker can download them and use
during execution according to job configuration.

Files can be uploaded either by drag and drop mechanism or by standard "Add a
file" button. In opened dialog window choose file which should be uploaded. All
chosen files are immediately uploaded to server but to save supplementary files
list you have to hit "Save supplementary files" button. All previously uploaded
files are visible right under drag and drop area, please note that files are
stored on fileserver and cannot be deleted after upload.

The last form on exercise editation page is runtime configurations editation
form. Exercise can have multiple runtime configurations according to the number
of programming languages in which it can be run. Every runtime configuration
corresponds to one programming language because all of them has to have a bit
different job configuration.

New runtime configuration can be added with "Add new runtime configuration"
button this will spawn new tab in runtime configurations box. In here you can
fill following:

- human readable identifier of runtime configuration
- runtime environment which corresponds to programming language
- job configuration in YAML, detailed description of job configuration can be
  found further in this chapter in "Writing job configuration" section

If you are done with changes to runtime configurations save form with "Change
runtime configurations" button. If you want to delete some particular runtime
just hit "Remove" button in the right tab, please note that after this operation
runtime configurations form has to be again saved to apply changes.

All runtime configurations which were added to exercise will be visible to
supervisors and all can be used in assignment, so please be sure that all of the
languages and job configurations are working.

If you choose to delete exercise, at the bottom of the exercise editation page
you can find "Delete the exercise" box where "Delete" button is located. By
clicking on it exercise will be delete from the exercises list and will no
longer be available.

### Reference Solutions of An Exercise

Each exercise should have a set of reference solutions, which are used to tune
time and memory limits of assignments. Values of used time and memory for each
solution are displayed in yellow boxes under forms for setting assignment limits
as described earlier.

However, there is currently no user interface to upload and evaluate reference
solutions. It is possible to use direct REST API calls, but it is not much user
friendly. If you are interested, please look at [API
documentation](https://recodex.github.io/api/), notably sections
_Uploaded-Files_ and _Reference-Exercise-Solutions_. You need to upload the
reference solution files, create a new reference solution and then evaluate the
solution. After that, measured data will be available in the box at assignment
editing page (setting limits section).

We are now working on a better user interface, which will be available soon.
Then its description will be added here.


## Group Administrator

Group administrator is the group supervisor with some additional permissions in
particular group. Namely group administrator is capable of creating a subgroups
in managed group and also adding and deleting supervisors. Administrator of the
particular group can be only one person.

### Creating Subgroups And Managing Supervisors

There is no special link which will get you to groups in which you are
administrator. So you have to get there through "Groups - supervisor" link in
sidebar and choose the right group detail page. If you are there you can see
"Administrator controls" section, here you can either add supervisor to group or
create new subgroup.

Form for creating a subgroup is present right on the group detail page in "Add
subgroup" box. Group can be created with following options:

- name which will be visible in group hierarchy
- external identification, can be for instance ID of group from school system
- some brief description about group
- allow or deny users to see each others statistics from assignments

After filling all the information a group can be created by clicking on "Create
new group" button. If creation is successful then the group is visible in
"Groups hierarchy" box on the top of page. All information filled during
creation can be later modified.

Adding a supervisor to a group is rather easy, on group detail page is an "Add
supervisor" box which contains text field. In there you can type name or
username of any user from system. After filling user name, click on the
magnifier image or press the enter key and all suitable users are searched. If
your chosen supervisor is in the updated list then just click on the "Make
supervisor" button and new supervisor should be successfully set.

Also, existing supervisor can be removed from the group. On the group detail
page there is "Supervisors" box in which all supervisors of the group are
visible. If you are the group administrator, you can see there "Remove
supervisor" buttons right next to supervisors names. After clicking on it some
particular supervisor should not to be supervisor of the group anymore.


## Instance Administrator

Instance administrator can be only one person per instance. In addition to
previous roles this administrator should be able to modify the instance details,
manage licences and take care of top level groups which belong to the instance.

### Instance Management

List of all instances in the system can be found under "Instances" link in the
sidebar. On that page there is a table of instances with their respective
admins. If you are one of them, you can visit its page by clicking on the
instance name. On the instance details page you can find a description of the
instance, current groups hierarchy and a form for creating a new group.

If you want to change some of the instance settings, follow "Edit instance" link
on the instance details page. This will take you to the instance editation page
with corresponding form. In there you can fill following information:

- name of the instance which will be visible to every other user
- brief description of instance and for whom it is intended
- checkbox if instance is open or not which means public or private (hidden from
  potential users)

If you are done with your editation, save filled information by clicking on
"Update instance" button.

If you go back to the instance details page you can find there a "Create new
group" box which is able to add a group to the instance. This form is the same
as the one for creating subgroup in already existing group so we can skip
description of the form fields. After successful creation of the group it will
appear in "Groups hierarchy" box at the top of the page.

### Licences

On the instance details page, there is a box "Licences". On the first line, it
shows it this instance has currently valid licence or not. Then, there are
multiple lines with all licences assigned to this instance. Each line consists
of a note, validity status (if it is valid or revoked by superadministrator) and
the last date of licence validity.

A box "Add new licence" is used for creating new licences. Required fields are
the note and the last day of validity. It is not possible to extend licence
lifetime, a new one should be generated instead. It is possible to have more
than one valid licence at a time. Currently there is no user interface for
revoking licences, this is done manually by superadministrator. If an instance
is to be disabled, all valid licences have to be revoked.


## Superadministrator

Superadministrator is a user with the most privileges and as such superadmin
should be quite a unique role. Ideally, there should be only one user of this
kind, used with special caution and adequate security. With this stated it is
obvious that superadmin can perform any action the API is capable of.

### Management of Users

There are only a few user roles in ReCodEx. Basically there are only three:
_student_, _supervisor_, and _superadmin_. Base role is student which is
assigned to every registered user. Roles are stored in database alongside other
information about user. One user always has only one role at the time. At first
startup of ReCodEx, the administrator has to change the role for his/her account
manually in the database. After that manual intervention into database should
never be needed.

There is a little catch in groups and instances management. Groups can have
admins and supervisors. This setting is valid only per one particular group and
has to be separated from basic role system. This implies that supervisor in one
group can be student in another and simultaneously have global supervisor role.
Changing role from student to supervisor and back is done automatically when the
new privileges are granted to the user, so managing roles by hand in database is
not needed. Previously stated information can be applied to instances as well,
but instances can only have admins.

Roles description:

- Student -- Default role which is used for newly created accounts. Student can
  join or leave public groups and submit solutions of assigned exercises.
- Supervisor -- Inherits all permissions from student role. Can manage groups to
  which he/she belongs to. Supervisor can also view and change groups details,
  manage assigned exercises, view students in group and their solutions for
  assigned exercises. On top of that supervisor can create/delete groups too,
  but only as subgroup of groups he/she belongs to.
- Superadmin -- Inherits all permissions from supervisor role. Most powerful
  user in ReCodEx who should be able to do access any functionality provided by
  the application.


## Writing Score Configuration

An important thing about assignment is how to assign points to particular
solutions. As mentioned previously, the whole job is composed of logical tests.
All of these tests have to contain one essential "evaluation" task. Evaluation
task should output one float number which can be further used for scoring of
particular tests.

Total resulting score of the students solution is then calculated according to a
supplied score config (described below) and using specified calculator. Total
score is also a float between 0 and 1. This number is then multiplied by the
maximum of points awarded for the assignment by the teacher assigning the
exercise -- not the exercise author.

For now, there is only one way how to write score configuration using only
simple score calculator. But the implementation in API is agile enough to handle
upcoming score calculators which might use some more complex scoring algorithms.
This also means that future calculators do not have to use the YAML format for
configuration. In fact, the configuration can be a string in any format.

### Simple Score Calculation

First implemented calculator is simple score calculator with test weights. This
calculator just looks at the score of each test and put them together according
to the test weights specified in assignment configuration. Resulting score is
calculated as a sum of products of score and weight of each test divided by the
sum of all weights. The algorithm in Python would look something like this:

```
sum = 0
weightSum = 0
for t in tests:
  sum += t.score * t.weight
  weightSum += t.weight
score = sum / weightSum
```

Sample score config in YAML format:

```{.yml}
testWeights:
  a: 300   # test with id 'a' has a weight of 300
  b: 200
  c: 100
  d: 100
```


## Writing Job Configuration

To run and evaluate an exercise the backend needs to know the steps how to do
that. This is different for each environment (operation system, programming
language, etc.), so each of the environments needs to have separate
configuration.

Backend works with a powerful, but quite low level description of simple
connected tasks written in YAML syntax. More about the syntax and general task
overview can be found on [separate
page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned
features was user friendly configuration editor, but due to tight deadline and
team composition it did not make it to the first release. However, writing
configuration in the basic format will be always available and allows users to 
use the full expressive power of the system.

This section walks through creation of job configuration for _hello world_
exercise. The goal is to compile file _source.c_ and check if it prints `Hello
World!` to the standard output. This is the only test case named **A**.

The problem can be split into several tasks:

- compile _source.c_ into _helloworld_ with `/usr/bin/gcc`
- run _helloworld_ and save standard output into _out.txt_
- fetch predefined output (suppose it is already uploaded to fileserver) with
  hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_
- compare _out.txt_ and _reference.txt_ by `/usr/bin/diff`

The absolute path of tools can be obtained from system administrator. However,
`/usr/bin/gcc` is location, where the GCC binary is available almost everywhere,
so location of some tools can be (professionally) guessed.

First, write header of the job to the configuration file. 

```{.yml}
submission:
    job-id: hello-word-job
    hw-groups:
        - group1
```

Basically it means, that the job _hello-world-job_ needs to be run on workers
that belong to the `group_1` hardware group . Reference files are downloaded
from the default location configured in API (such as
`http://localhost:9999/exercises`) if not stated explicitly otherwise. Job
execution log will not be saved to result archive.

Next the tasks have to be constructed under _tasks_ section. In this demo job,
every task depends only on previous one. The first task has input file
_source.c_ (if submitted by user) already available in working directory, so
just call the GCC. Compilation is run in sandbox as any other external program
and should have relaxed time and memory limits. In this scenario, worker
defaults are used. If compilation fails, the whole job is immediately terminated
(because the _fatal-failure_ bit is set). Because _bound-directories_ option in 
sandbox limits section is mostly shared between all tasks, it can be set in 
worker configuration instead of job configuration (suppose this for following 
tasks). For configuration of workers please contact your administrator.

Please note that working directory inside sandbox is automatically bounded to
the directory with fetched user source codes and therefore you do not have to
bound it by yourself. Also note that directories inside sandbox can be bound to
different paths, so inside sandbox you have to use special paths. For working
directory inside sandbox you can use ${EVAL_DIR} variable.

```{.yml}
- task-id: "compilation"
  type: "initiation"
  fatal-failure: true
  cmd:
      bin: "/usr/bin/gcc"
      args:
          - "source.c"
          - "-o"
          - "helloworld"
  sandbox:
      name: "isolate"
      limits:
          - hw-group-id: group1
```

The compiled program is executed with time and memory limit set and the standard
output is redirected to a file. This task depends on _compilation_ task, because
the program cannot be executed without being compiled first. It is important to
mark this task with _execution_ type, so exceeded limits will be reported in
frontend.

Time and memory limits set directly for a task have higher priority than worker
defaults. One important constraint is, that these limits cannot exceed limits
set by workers. Worker defaults are present as a safety measure so that a 
malformed job configuration cannot block the worker forever. Worker default 
limits should be reasonably high, like a gigabyte of memory and several hours of 
execution time. For exact numbers please contact your administrator.

It is important to know that if the output of a program (both standard and 
error) is redirected to a file, the sandbox disk quotas apply to that file, as 
well as the files created directly by the program. In case the outputs are 
ignored, they are redirected to `/dev/null`, which means there is no limit on 
the output length (as long as the printing fits in the time limit).

```{.yml}
- task-id: "execution_1"
  test-id: "A"
  type: "execution"
  dependencies:
      - compilation
  cmd:
      bin: "helloworld"
  sandbox:
      name: "isolate"
      stdout: ${EVAL_DIR}/out.txt
      limits:
          - hw-group-id: group1
            time: 0.5
            memory: 8192
```

Fetch sample solution from file server. Base URL of file server is in the header 
of the job configuration, so only the name of required file (its `sha1sum` in 
our case) is necessary.

```{.yml}
- task-id: "fetch_solution_1"
  test-id: "A"
  dependencies:
      - execution
  cmd:
      bin: "fetch"
      args:
          - "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b"
          - "${SOURCE_DIR}/reference.txt"
```

Comparison of results is quite straightforward. It is important to set the task
type to _evaluation_, so that the return code is set to 0 if the program is
correct and 1 otherwise. We do not set our own limits, so the default limits are
used.

```{.yml}
- task-id: "judge_1"
  test-id: "A"
  type: "evaluation"
  dependencies:
      - fetch_solution_1
  cmd:
      bin: "/usr/bin/diff"
      args:
          - "out.txt"
          - "reference.txt"
  sandbox:
      name: "isolate"
      limits:
          - hw-group-id: group1
```

<!---
// vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
-->