Divide rewritten documentation into separate chapters

8 years ago · 6c5736f7ea
parent abec266d97
commit 6c5736f7ea
8 changed files with 4078 additions and 335 deletions
--- a/Analysis.md
+++ b/Analysis.md
--- a/Conclusion.md
+++ b/Conclusion.md
@ -0,0 +1,88 @@
 # Conclusion
 The project of ReCodEx was a great experience with developing a bigger
 application in team of people. We mostly enjoyed the time spent on the project,
 however the deadline was too tight to implement all nice features we thought of.
 Our implementation meets our predefined goals and is ready to be deployed
 instead of current tool CodEx.
 We made several design choices during planning and implementing the project.
 From perspective of today we are mostly happy about them and we believe we have
 done the best we could. One of the biggest questions was the broker/worker
 programming language. The winner was C++, which is really great language
 especially in revisions starting at C++11. But we are not sure if Python would
 not have been better option. From the list of good choices we would like to
 mention early unit testing with continuous integration, using ZeroMQ messaging
 framework and the design with splitting logic into multiple components.
 To sum up, we created an extensible base environment for years of further
 development by numerous student projects. The faculty would benefit from higher
 hardware utilization, less administration effort and a possibility to offer
 ReCodEx as SaaS to partner grammar schools. Students would also appreciate
 state-of-art development tools.
 During the development we were using whole bunch of development tools. We would
 like to thank authors, maintainers and contributors of these tools to help us
 complete this project. Most notably we would like to mention explicitly the top
 projects: Git and GitHub, Visual Studio Code, IDEs from JetBrains, ZeroMQ
 framework, C++ spdlog library, Travis and AppVeyor CI, Nette framework and many
 many others. Thank you.
 Finally, we would like to thank our supervisor, Martin Kruliš, for his support
 and great effort correcting text of this documentation.
 ## Further Improvements
 A lot of work has been done, but it opened a whole bunch of new possibilities
 for subsequent student projects and other kinds of contribution. We would be
 happy to see people contributing to this project to make it even more awesome.
 We are presenting a brief list of features we think might be worth of
 completing. Surely, the list is not complete and may change in time.
 - Finish web frontend. In time of the project submission it does not contain all
  features for the best user experience in all roles. This task is suitable for
  people with basic web programming experience, React and Redux knowledge is an
  advantage.
 - Create web editor of job configuration. ReCodEx job evaluation follows quite
  complex task flow. To create valid configuration, author of exercise has to
  write a pretty long YAML file by hand. This of course can be partially or
  fully automated but was not implemented at time. This task therefore consist
  of creating web based editor of this configuration with prepared task
  templates allowing user to change their properties and connect them. After
  submit, YAML job configuration should be properly created.
 - Write alternative commandline frontend. A lot of users want to submit their
  solutions directly from command line. That is newly possible in ReCodEx
  because of REST API, but no suitable frontend exists yet. There is unfinished
  attempt to create one in NodeJS, accessible in ReCodEx organization on
  [GitHub](https://github.com/ReCodEx/cli). The goal is to finish this project
  or create alternative tool. We would like to see one written in Python.
 - Create mobile frontend. It would be really nice to have mobile application
  when you can read exercise assignments and examine your progress and
  statistics. All is possible due to ReCodEx REST API. The basic Android
  application is in development by ReCodEx team as project for programming
  mobile devices class. The code will be published as part of ReCodEx GitHub
  organization.
 - Design and implement backend monitoring. For administrator would be great to
  examine status of backend (broker, workers, monitor, fileserver) directly in
  browser. Key information are which machines are online, offline or failed,
  which worker is now running a job (maybe with current progress) and view
  configuration of worker. Also some statistics with graphs should be made, for
  example workload of workers. More advanced feature is the ability to restart
  workers on the change of their configurations.
 - Finish .NET sandbox for Windows. We developed initial sandbox for Windows
  environment, [WrapSharp](https://github.com/ReCodEx/wrapsharp). WrapSharp is
  only for .NET platform assemblies and most notably for C# programs and cannot
  be used generally for sandboxing on Windows. The goal is to finish
  implementation and do a really detailed security audit of the project. Also,
  integration to the worker is not fully done yet.
 - SIS integration. A very nice feature is (semi)automatic creation of groups and
  assigning students to them depending on their timetable in student information
  system (SIS). However there is no standardized API for such communication yet,
  but we hope this will be possible in the future. Implementing this feature
  means extending ReCodEx API by a SIS module in PHP.
 <!---
 // vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
 -->
--- a/Implementation.md
+++ b/Implementation.md
--- a/Introduction.md
+++ b/Introduction.md
@ -0,0 +1,365 @@
 # Introduction
 In general there are many different ways and opinions on how to teach people
 something new. However, most people agree that a hands-on experience is one of
 the best ways to make the human brain remember a new skill. Learning must be
 entertaining and interactive, with fast and frequent feedback. Some areas
 are more suitable for this practical way of learning than others, and
 fortunately, programming is one of them.
 University education system is one of the areas where this knowledge can be
 applied. In computer programming, there are several requirements a program
 should satisfy, such as the code being syntactically correct, efficient and easy
 to read, maintain and extend. 
 Checking programs written by students by hand takes time and requires a lot of
 repetitive work -- reviewing source codes, compiling them and
 running them through test scenarios. It is therefore desirable to automate as
 much of this process as possible. 
 The first idea of an automatic evaluation system
 comes from Stanford University professors in 1965. They implemented a system
 which evaluated code in Algol submitted on punch cards. In following years, many
 similar products were written.
 Nowadays properties like correctness and efficiency can be tested
 to a large extent automatically. This fact should be exploited to help teachers
 save time for tasks such as examining bad design, bad coding habits, or logical
 mistakes, which are difficult to perform automatically.
 There are two basic ways of automatically evaluating code:
 - **statically** -- by checking the source code without running it. 
  This is safe, but not very practical.
 - **dynamically** -- by running the code on test inputs and checking the correctness of
  outputs ones. This provides good real world experience, but requires extensive
  security measures).
 This project focuses on the machine-controlled part of source code evaluation.
 First we observed the general concepts of grading systems and discussed the problems of the
 software previously used at the Charles University in Prague.
 Then the new requirements were specified and we examined projects with similar functionality.
 With the acquired knowledge from these projects, we set up
 goals for the new evaluation system, designed the architecture and implemented a
 fully operational solution based on dynamic evaluation. The system is now ready
 for production testing at the university.
 ## Assignment
 The major goal of this project is to create a grading application which will be
 used for programming classes at the Faculty of Mathematics and Physics of the
 Charles University in Prague. However, the application should be designed in a
 modular fashion to be easily extended or even modified to make other ways of
 usage possible.
 The system should be capable of doing a dynamic analysis of the submitted source
 codes. This consists of the following basic steps:
 1. compile the code and check for compilation errors
 2. run the compiled program in a sandbox with predefined inputs
 3. check the constraints of the amount of used memory and time
 4. compare the outputs of the program with the defined expected outputs
 5. award the solution with a numeric score 
 The whole system is intended to help both the teachers (supervisors) and the students.
 To achieve this, it is crucial for us to keep in mind the typical usage scenarios of
 the system and to try to make these tasks as simple as possible. To fulfill this
 task, the project has a great starting point -- there is an old grading system
 currently used at the university (CodEx), so its flaws and weaknesses can be
 addressed. Furthermore, many teachers desire to use and test the new system and
 they are willing to consult our ideas or problems during the development with us.
 ## Current System
 The grading solution currently used at the Faculty of Mathematics and Physics of
 the Charles University in Prague was implemented in 2006 by a group of students.
 It is called [CodEx -- The Code Examiner](http://codex.ms.mff.cuni.cz/project/)
 and it has been used with some improvements since then. The original plan was to
 use the system only for the basic programming courses, but there was a demand for
 adapting it for several different courses.
 CodEx is based on dynamic analysis. It features a web-based interface, where
 supervisors can assign exercises to their students and the students have a time
 window to submit their solutions. Each solution is compiled and run in sandbox
 (MO-Eval). The metrics which are checked are: correctness of the output, time
 and memory limits. It supports programs written in C, C++, C#, Java, Pascal,
 Python and Haskell.
 The system has a database of users. Each user is assigned a role, which
 corresponds to his/her privileges. There are user groups reflecting the
 structure of lectured courses.
 A database of exercises (algorithmic problems) is another part of the project.
 Each exercise consists of a text describing the problem, a configuration of the 
 evaluation (machine-readable instructions on how to evaluate solutions to the
 exercise), time and memory limits for all supported runtimes (e.g. programming
 languages), a configuration for calculating the final score and a set of inputs
 and reference outputs. Exercises are created by instructed privileged users.
 Assigning an exercise to a group means choosing one of the available exercises
 and specifying additional properties: a deadline (optionally a second deadline),
 a maximum amount of points, a maximum number of submissions and a list of
 supported runtime environments.
 The typical use cases for the user roles are the following:
 - **student**
 	- create a new user account via a registration form
 	- join groups (e.g., the courses he attends)
 	- get assignments in the groups
 	- submit a solution to an assignment -- upload one source file and start the
 	  evaluation process
 	- view the results of the solution -- which parts succeeded and failed, the total
 	  number of the acquired points, bonus points
 - **supervisor** (similar to CodEx *operator*)
 	- create a new exercise -- create description text and evaluation configuration
 	  (for each programming environment), upload testing inputs and outputs
 	- assign an exercise to a group -- choose an exercise and set the deadlines, 
 	  the number of allowed submissions, the weights of all test cases and the amount
 	  of points for the correct solutions
 	- modify an assignment
 	- view all of the results of the students in a group
 	- review the automatic solution evaluation -- view the submitted source files 
 	  and optionally set bonus points (including negative points)
 - **administrator**
 	- create groups
 	- alter user privileges -- make supervisor accounts
 	- check system logs
 ### Exercise Evaluation Chain
 The most important part of the system is the evaluation of solutions submitted by
 the students. The process from the source code to final results (score) is
 described in more detail below to give readers a solid overview of what is happening
 during the evaluation process.
 The first thing students have to do is to submit their solutions through the web user
 interface. The system checks assignment invariants (e.g., deadlines, number of
 submissions) and stores the submitted code. The runtime environment is
 automatically detected based on the extension of the input file, and a suitable evaluation
 configuration type is chosen (one exercise can have multiple variants, for
 example C and Java is allowed). This exercise configuration is then used for
 the evaluation process.
 There is a pool of uniform worker machines dedicated to evaluation jobs.
 Incoming jobs are kept in a queue until a free worker picks them. Workers are
 capable of a sequential evaluation of jobs, one at a time.
 The worker obtains the solution and its evaluation configuration, parses it and
 starts executing the instructions contained. Each job should have more test
 cases which examine invalid inputs, corner cases and data of different sizes to
 estimate the program complexity. It is crucial to keep the computer running the worker
 secure and stable, so a sandboxed environment is used for dealing with an
 unknown source code. When the execution is finished, results are saved, and the
 student is notified.
 The output of the worker contains data about the evaluation, such as time and
 memory spent on running the program for each test input and whether its output
 is correct. The system then calculates a numeric score from the data which is
 presented to the student. If the solution is incorrect (e.g., incorrect output,
 exceeds memory or time limits), error messages are also displayed to the student.
 ### Possible Improvements
 The current system is old, but robust. There were no major security incidents
 in the course of its usage. However, from the present day perspective there are
 several major drawbacks:
 - **web interface** -- The web interface is simple and fully functional.
  However, the recent rapid development in web technologies provides us with new
  possibilities of making web interfaces.
 - **public API** -- CodEx offers a very limited public XML API based on outdated
  technologies that are not sufficient for users who would like to create their
  custom interfaces such as a command line tool or a mobile application.
 - **sandboxing** -- the MO-Eval sandbox is based on the principle of monitoring
  system calls and blocking the forbidden ones. This can be sufficient with
  single-threaded programs, but proves to be difficult with multi-threaded ones.
  Nowadays, parallelism is a very important area of computing, it is required that
  multi-threaded programs can be securely tested as well.
 - **instances** -- Different ways of CodEx use require separate
  installations (e.g., Programming I and II, Java, C#). This configuration is
  not user friendly as students have to register in each installation separately
  and burdens administrators with unnecessary work. The CodEx architecture does not
  allow sharing workers between installations which results in an inefficient
  use of hardware for evaluation.
 - **task extensibility** -- There is a need to test and evaluate complicated
  programs for courses such as *Parallel programming* or *Compiler principles*,
  which have a more difficult evaluation chain than simple
  *compilation/execution/evaluation* provided by CodEx.
 ## Requirements
 There are many different formal requirements for the system. Some of them
 are necessary for any system for source code evaluation, some of them are
 specific for university deployment and some of them arose during the ten year
 long lifetime of the old system. There are not many ways of improving CodEx
 experience from the perspective of a student, but a lot of feature requests come
 from administrators and supervisors. The ideas were gathered mostly from our
 personal experience with the system and from meetings with the faculty staff
 who use the current system.
 In general, CodEx features should be preserved, so only the differences are
 presented here. For clear arrangement, all the requirements and wishes are
 presented in groups by the user categories.
 ### Requirements of The Users
 - _group hierarchy_ -- creating an arbitrarily nested tree structure should be
  supported to keep related groups together, such as in the example
  below. CodEx supported only a flat group structure. A group hierarchy also
  allows to archive data from the past courses.
 ```
  Summer term 2016
  |-- Language C# and .NET platform
  |   |-- Labs Monday 10:30
  |   `-- Labs Thursday 9:00
  |-- Programming I
  |   |-- Labs Monday 14:00
   ...
 ```
 - _a database of exercises_ -- teachers should be able to filter the displayed
  exercises according to several criteria, for example by the supported runtime
  environments or by the author. It should also be possible to link exercises to a group
  so that group supervisors do not have to browse hundreds of exercises when
  their group only uses a few of them
 - _advanced exercises_ -- the system should support a more advanced evaluation
  pipeline than basic *compilation/execution/evaluation* which is in CodEx
 - _customizable grading system_ -- teachers need to specify the way of
  calculating the final score which will be allocated to the submissions
  depending on their correctness and quality
 - _marking a solution as accepted_ -- a supervisor should be able to choose
  one of the submitted solutions of a student as accepted. The score of this
  particular solution will be used as the score which the student receives
  for the given assignment instead of the one with the highest score.
 - _solution resubmission_ -- teachers should be able to edit the solutions of the
  student and privately resubmit them, optionally saving all results (including
  temporary ones); this feature can be used to quickly fix obvious errors in the
  solution and see if it is otherwise correct
 - _localization_ -- all texts (the UI and the assignments of the exercises) should
  be translatable into several languages
 - _formatted texts of assignments_ -- Markdown or another lightweight markup language
  should be supported for the formatting of the texts of the exercises
 - _comments_ -- adding both private and public comments to exercises, tests and
  solutions should be supported
 - _plagiarism detection_
 ### Administrative Requirements
 - _independent user interface_ -- the system should allow the use of an alternative
  user interface, such as a command line client; implementation of such clients
  should be as straightforward as possible
 - _privilege separation_ -- there should be at least two roles -- _student_ and
  _supervisor_. The cases when a student of a course is also a teacher of another
  course must be handled correctly.
 - _alternative authentication methods_ -- logging in through a university
  authentication system (e.g. LDAP) and potentially other services, such as
  Github or some other OAuth service, should be supported
 - _querying SIS_ -- loading user data from the university information system (SIS)
  should be supported
 - _sandboxing_ -- there should be a more advanced sandbox which supports
  execution of parallel programs and an easy integration of different programming
  environments and tools; the sandboxed environment should have the minimum
  possible impact on the measurement of results (most importantly on the measured
  duration of execution)
 - _heterogeneous worker pool_ -- there must be a support for submission evaluation
  in multiple programming environments in a single installation to avoid
  unacceptable workload of the administrator (i.e., maintaining a separate
  installation for every course) and high hardware requirements
 - advanced low-level evaluation flow configuration with high-level abstraction
  layer for ordinary configuration cases; the configuration should be able to
  express more complicated flows than just compiling a source code and running
  the program against the test inputs -- for example, some exercises need to build
  the source code with a tool, run some tests, then run the program through
  another tool and perform additional tests
 - use of modern technologies with state-of-the-art compilers
 ### Non-functional Requirements
 - _no installation_ -- the primary user interface of the system must be
  accessible on the computers of the users without the need to install any
  additional software except for a web browser which is installed on almost
  every personal computer
 - _performance_ -- the system must be ready for at least hundreds of students
  and tens of supervisors who are using it at the same time
 - _automated deployment_ -- all of the components of the system must be easy to
  deploy in an automated fashion
 - _open source licensing_ -- the source code should be released under a
  permissive licence allowing further development; this also applies to the used
  libraries and frameworks
 - _multi-platform worker_ -- worker machines running Linux, Windows and
  potentially other operating systems must be supported
 ### Conclusion
 The survey shows that there are a lot of different requirements and wishes for
 the new system. When the system is ready, it is likely that there will be new
 ideas on how to use the system and thus the system must be designed to be easily
 extendable so that these new ideas can be easily implemented, either by us or
 community members. This also means that widely used programming languages and
 techniques should be used so the programmers can quickly understand the code and
 make changes easily.
 ## Related work
 To find out the current state in the field of automatic grading systems, we conducted
 a short market survey of the field of automatic grading systems at universities,
 programming contests, and other places where similar tools are used.
 This is not a complete list of available evaluators, but only of a few projects
 which are used these days and can be of an inspiration for our project. Each
 project on the list is provided with a brief description and some of its key features.
 ### Progtest
 [Progtest](https://progtest.fit.cvut.cz/) is a private project of [FIT
 ČVUT](https://fit.cvut.cz) in Prague. As far as we know it is used for C/C++,
 Bash programming, and knowledge-based quizzes. Each submitted solution can receive
 several bonus points or penalties and also a few hints can be attached of
 what is incorrect in the solution. It is very strict with the source code quality,
 for example the `-pedantic` option of GCC is used; Valgrind is used for detection of
 memory leaks; array boundaries are checked via the `mudflap` library.
 ### Codility
 [Codility](https://codility.com/) is a web based solution primarily targeted at
 company recruiters. It is a commercial product available as SaaS and it
 supports 16 programming languages. The
 [UI](http://1.bp.blogspot.com/-_isqWtuEvvY/U8_SbkUMP-I/AAAAAAAAAL0/Hup_amNYU2s/s1600/cui.png)
 of Codility is [opensource](https://github.com/Codility/cui), the rest of source
 code is not available. One interesting feature is the 'task timeline' -- the captured
 progress of writing the code for each user.
 ### CMS
 [CMS](http://cms-dev.github.io/index.html) is an opensource distributed system
 for running and organizing programming contests. It is written in Python and
 contains several modules. CMS supports C/C++, Pascal, Python, PHP, and Java
 programming languages. PostgreSQL is a single point of failure, all modules
 heavily depend on the database connection. Task evaluation can be only a three
 step pipeline -- compilation, execution, evaluation. Execution is performed in
 [Isolate](https://github.com/ioi/isolate), a sandbox written by the consultant
 of our project, Mgr. Martin Mareš, Ph.D.
 ### MOE
 [MOE](http://www.ucw.cz/moe/) is a grading system written in Shell scripts, C
 and Python. It does not provide a default GUI interface, all actions have to be
 performed from the command line. The system does not evaluate submissions in real
 time, results are computed in a batch mode after the exercise deadline, using Isolate
 for sandboxing. Parts of MOE are used in other systems like CodEx or CMS, but
 the system is obsolete in general.
 ### Kattis
 [Kattis](http://www.kattis.com/) is another SaaS solution. It provides a clean
 and functional web UI, but the rest of the application is too simple. A nice
 feature is the usage of a [standardized 
 format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for
 exercises. Kattis is primarily used by programming contest organizers, company
 recruiters and also some universities.
 <!---
 // vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
 -->
--- a/Submission-flow.md
+++ b/Submission-flow.md
@ -1,90 +0,0 @@
 # Submission flow
 This article will describe in execution flow of submission from the point of submission into web application to the point of evaluation of results from execution. Note that following text is simplified and should only show concepts which are used during execution.
 ## Web Application
 First thing users have to do is to submit their solutions to web application which provides interface to upload multiple files and then submit them. More detailed description follows:
 1. user upload file by file into prepared submit form
 2. after uploading all files connected to assignment user clicks on submit button
 3. Web Application send request to Web API that user wants to evaluate assignment with provided files
 ## Web API
 After user submits solution then web application has to hand over all needed information about submission to broker. Submit endpoint is extended with support of "cross-submitting" which means that user with high privileges (supervisor, administrator) can submit solution for user with lower privileges (student). More detailed description follows:
 1. deadlines and count of submissions are checked
 2. programming language of uploaded files is automatically detected
 3. based on programming language proper job configuration file is obtained
 4. submitted files from user and job configuration are uploaded to fileserver as archive
 5. send message to broker that there is new submission
 6. several information about new submission including websocket channel are sent back to user
 ## Broker
 Broker gets information about new submission from API server. At this point broker has to find suitable worker for execution of this particular submission. When worker is found and is jobless, then broker send detailed submission to worker to evaluation. More detailed description follows:
 1. broker gets message from API about new submission
 2. based on information from API, broker has to choose suitable worker which matches all submission criteria
    1. if suitable worker is not found then broker notices Web API about this information and all execution stops
    2. suitable worker is found
        1. worker has no jobs at the moment, broker immediately sends job evaluation request to it
        2. otherwise evaluation request is queued to this particular worker and waits until all previous jobs are done
 3. broker sends prepared evaluation request to worker with all information which was obtained from API
 ## Worker
 Worker gets request from broker to evaluate particular submission. Next step is to evaluate given submission and upload results to fileserver. After this worker only send broker that submission was evaluated. More detailed description follows:
 1. worker gets evaluation request from broker
 2. worker now has to do some initialization of directories and some internal structures for new incoming evaluation
 3. users submission archive from given address from evaluation request is downloaded and decompressed
 4. job configuration file is located and parsed into internal job structure
 5. tasks which are present in configuration are loaded into tree like structure with dependencies and are divided into internal or external ones
    1. internal tasks are tasks which has no defined limits and contains some internal action
    2. external tasks have defined limits in configuration and are executed in sandbox
 6. last step of initializing of job is to topologically sort tasks and prepare queue with right execution order
 7. after that topologically sorted execution queue is processed
    1. if execution of `inner` task fails worker will immediately stop execution and send back to broker 'internal error during execution' message
    2. execution of `execution` or `evaluation` task fails then worker stops and send back to broker 'execution failed' message
    3. during execution worker can optionally sends current job status back to web application, this communication is described in more detailed [[communication|Overall architecture#communication]] article.
 8. when execution successfully ends results are collected from all executed tasks and written into yaml file
 9. results yaml file alongside with content of job result folder is sent back to fileserver in compressed form
 10. of course there has to be some cleaning after whole evaluation which will mostly delete content of temporarily created folders
 11. last step of worker is to send back to broker message that execution successfully ended
 ## Broker
 Broker gets done message from worker and basically only mark submission as done in its internal structures. After that broker has to tell Web API that execution of particular job ended. More detailed description follows:
 1. broker gets message from worker that execution of job ended
 2. job execution on worker was successful
    1. Web API is notified that job ended successfully
    2. if there are some waiting jobs for worker than the first one is sent for execution
 3. job execution on worker ended with internal error
    1. if job ends with internal error than it is possible to reassign it to another worker
    2. broker keeps track of job reassignments and if number of them reach some predefined constant, job is declared as failed
    3. suitable worker different that the original one is picked and evaluation request is sent to it
 4. job execution on worker failed
    1. Web API is notified that execution of this particular job failed
    2. again if there is waiting job for execution on worker than it is sent for execution
 ## Web API
 Web API is notified about job status from broker. After that API is deciding if evaluated submission will be evaluated immediately or on demand. But for the sake of simplicity we will consider only immediately evaluated results. More detailed description follows:
 1. job arrived with successful evaluation status
    1. results of evaluation are downloaded from fileserver where it was uploaded by worker and then unzipped
    2. result yaml file is loaded and actual time and memory consumption are retrieved
    3. overall results are computed and stored into database
 2. job arrived as failed
    1. job failure is saved into database alongside other reported errors
    2 email is sent to ReCodEx administrator that job failed and cannot be properly executed
 ## Web Application
 Web application has only a simple work to do. It has to obtain results from web API. More detailed description follows:
 1. web application sends request for results of particular submission
 2. if results are available then they are sent back to web application and displayed to user
--- a/User-documentation.md
+++ b/User-documentation.md
@ -0,0 +1,857 @@
 # User Documentation
 Users interact with the ReCodEx through the web application. It is required to
 use a modern web browser with good HTML5 and CSS3 support. Among others, cookies
 and local storage are used. Also a decent JavaScript runtime must be provided by
 the browser.
 Supported and tested browsers are: Firefox 50+, Chrome 55+, Opera 42+ and Edge
 13+. Mobile devices often have problems with internationalization and possibly
 lack support for some common features of desktop browsers. In this stage of
 development is not possible for us to fine tune the interface for major mobile
 browsers on all mobile platforms. However, it is confirmed to work with latest
 Google Chrome and Gello browser on Android 7.1+. Issues have been reported with
 Firefox that will be fixed in the future. Also, it is confirmed to work with
 Safari browser on iOS 10.
 Usage of the web application is divided into sections concerning particular user
 roles. Under these sections all possible use cases can be found. These sections
 are inclusive, so more privileged users need to read instructions for all less
 privileged users. Described roles are:
 - Student
 - Group supervisor
 - Group administrator
 - Instance administrator
 - Superadministrator
 ## Terminology
 **Instance** -- Represents a university, company or some other organization
 unit. Multiple instances can exist in a single ReCodEx installation.
 **Group** -- A group of students to which exercises are assigned by a
 supervisor. It should typically correspond with a real world lab group.
 **User** -- A person that interacts with the system using the web interface (or
 an alternative client).
 **Student** -- A user with least privileges who is subscribed to some groups and
 submits solutions to exercise assignments.
 **Supervisor** -- A person responsible for assigning exercises to a group and
 reviewing submissions.
 **Admin** -- A person responsible for the maintenance of the system and fixing
 problems supervisors cannot solve.
 **Exercise** -- An algorithmic problem that can be assigned to a group.  They
 can be shared by the teachers using an exercise database in ReCodEx.
 **Assignment** -- An exercise assigned to a group, possibly with modifications.
 **Runtime environment** -- Runtime environment is unique combination of platform
 (OS) and programming language runtime/compiler in specific version. Runtime
 environments are managed by the administrators to reflect abilities of whole
 system.
 **Hardware group** -- Hardware group is a set of workers with similar hardware.
 Its purpose is to group workers that are likely to run a program using the same
 amount of resources. Hardware groups are managed byt the system administrators
 who have to keep them up-to-date.
 ## General Basics
 Description of general basics which are the same for all users of ReCodEx web
 application follows.
 ### First Steps in ReCodEx
 You can create an account by clicking the "Create account" menu item in the left
 sidebar. You can choose between two types of registration methods -- by creating
 a local account with a specific password, or pairing your new account with an
 existing CAS UK account.
 If you decide to create a new local account using the "Create ReCodEx account”
 form, you will have to provide your details and choose a password for your
 account. Although ReCodEx allows using quite weak passwords, it is wise to use a
 bit stronger ones The actual strength is shown in progress bar near the password
 field during registration. You will later sign in using your email address as
 your username and the password you select.
 If you decide to use the CAS UK service, then ReCodEx will verify your CAS
 credentials and create a new account based on information stored there (name and
 email address). You can change your personal information later on the
 "Settings" page.
 Regardless of the desired account type, an instance it will belong to must be
 selected. The instance will be most likely your university or other organization
 you are a member of.
 To log in, go to the homepage of ReCodEx and in the left sidebar choose the menu
 item "Sign in". Then you must enter your credentials into one of the two forms
 -- if you selected a password during registration, then you should sign with
 your email and password in the first form called "Sign into ReCodEx". If you
 registered using the Charles University Authentication Service (CAS), you should
 put your student’s number and your CAS password into the second form called
 "Sign into ReCodEx using CAS UK".
 There are several options you can edit in your user account:
 - changing your personal information (i.e., name)
 - changing your credentials (email and password)
 - updating your preferences (source code viewer/editor settings, default
  language)
 You can access the settings page through the "Settings" button right under
 your name in the left sidebar.
 If you are not active in ReCodEx for a whole day, you will be logged out
 automatically. However, we recommend you sign out of the application after you
 finish your interaction with it. The logout button is placed in the top section
 of the left sidebar right under your name. You may need to expand the sidebar
 with a button next to the "ReCodEx” title (informally known as _hamburger
 button_), depending on your screen size.
 ### Forgotten Password
 If you cannot remember your password and you do not use CAS UK authentication,
 then you can reset your password. You will find a link saying "Cannot remember
 what your password was? Reset your password." under the sign in form. After you
 click this link, you will be asked to submit your registration email address. A
 message with a link containing a special token will be sent to you by e-mail --
 we make sure that the person who requested password resetting is really you.
 When you visit the link, you will be able to enter a new password for your
 account. The token is valid only for a couple of minutes, so do not forget to
 reset the password as soon as possible, or you will have to request a new link
 with a valid token.
 If you sign in through CAS UK, then please follow the instructions
 provided by the administrators of the service described on their
 website.
 ### Dashboard
 When you log into the system you should be redirected to your "Dashboard". On
 this page you can see some brief information about the groups you are member of.
 The information presented there varies with your role in the system -- further
 description of dashboard will be provided later on with according roles.
 ## Student
 Student is a default role for every newly registered user. This role has quite
 limited capabilities in ReCodEx. Generally, a student can only submit solutions
 of exercises in some particular groups. These groups should correspond to
 courses he/she attends.
 On the "Dashboard" page there is "Groups you are student of" section where you
 can find list of your student groups. In first column of every row there is a
 brief panel describing concerning group. There is name of the group and
 percentage of gained points from course. If you have enough points to
 successfully complete the course then this panel has green background with tick
 sign. In the second column there is a list of assigned exercises with its
 deadlines. If you want to quickly get to the groups page you might want to use
 provided "Show group's detail" button.
 ### Join Group
 To be able to submit solutions you have to be a member of the right group. Each
 instance has its own group hierarchy, so you can choose only those within your
 instance. That is why a list of groups is available from under an instance link
 located in the sidebar. This link brings you to instance detail page.
 In there you can see a description of the instance and most importantly in
 "Groups hierarchy" box there is a hierarchical list of all public groups in the
 instance. Please note that groups with plus sign are collapsible and can be
 further extended. When you find a group you would like to join, continue by
 clicking on "See group's page" link following with "Join group" link.
 **Note:** Some groups can be marked as private and these groups are not visible
 in hierarchy and membership cannot be established by students themselves.
 Management of students in this type of groups is in the hands of supervisors.
 On the group detail page there are multiple interesting things for you. The
 first one is a brief overview containing the information describing the group,
 there is a list of supervisors and also the hierarchy of the subgroups. The most
 important section is the "Student's dashboard" section. This section contains
 the list of assignments and the list of fellow students. If the supervisors of
 the group allowed students to see the statistic of their fellow students then
 there will also be the number of points each of the students has gained so far.
 ### Start Solving Assignments
 In the "Assignments" box on the group detail page there is a list of assigned
 exercises which students are supposed to solve. The assignments are displayed
 with their names and deadlines. There are possibly two deadlines, the first one
 means that till this datetime student will receive full amount of points in case
 of successful solution. Second deadline does not have to be set, but in case it
 is, the maximum number of points for successful solution between these two
 deadlines can be different.
 An assignment link will lead you to assignment detail page where are presented
 all known details about assignment. There are of course both deadlines, limit of
 submissions which you can make and also full-range description of assignment,
 which can be localized. The localization can be on demand switched between all
 language variants in tab like box.
 Further on the page you can find "Submitted solutions" box where is a list of
 submissions with links to result details. But most importantly there is a
 "Submit new solution" button on the assignment page which provides an interface
 to submit solution of the assignment.
 After clicking on submit button, dialog window will show up. In here you can
 upload files representing your solution, you can even add some notes to mark the
 solution. Your supervisor can also access this note. After you successfully
 upload all files necessary for your solution, click the "Submit your solution"
 button and let ReCodEx evaluate the solution.
 During the execution ReCodEx backend might send evaluation progress state to
 your browser which will be displayed in another dialog window. When the whole
 execution is finished then a "See the results" button will appear and you can
 look at the results of your solution.
 ### View Results of Submission
 On the results detail page there are a lot of information. Apart from assignment
 description, which is not connected to your results, there is also the solution
 submitter name (supervisor can submit a solution on your behalf), further there
 are files which were uploaded on submission and most importantly "Evaluation
 details" and "Test results" boxes.
 Evaluation details contains overall results of your solution. There are
 information such as whether the solution was provided before deadlines, if the
 evaluation process successfully finished or if compilation succeeded. After that
 you can find a lot of values, most important one is the last, "Total score",
 consisting of your score, slash and the maximum number of points for this
 assignment. Interestingly the your score value can be higher than the maximum,
 which is caused by "Bonus points" item above. If your solution is nice and
 supervisor notices it, he/she can assign you additional points for effort. On
 the other hand, points can be also subtracted for bad coding habits or even
 cheating.
 In test results box there is a table of all exercise tests results. Columns
 represents these information:
 - test case overall result, symbol of yes/no option
 - test case name
 - percentage of correctness of this particular test
 - evaluation status, if test was successfully executed or failed
 - memory limit, if supervisor allowed it then percentual memory usage is
  displayed
 - time limit, if supervisor allowed it then percentual time usage is displayed
 A new feature of web application is "Comments and notes" box where you can
 communicate with your supervisors or just write random private notes to your
 submission. Adding a note is quite simple, you just write it to text field in
 the bottom of box and click on the "Send" button. The button with lock image
 underneath can switch visibility of newly created comments.
 In case you think the ReCodEx evaluation of your solution is wrong, please use
 the comments system described above, or even better notify your supervisor by
 another channel (email). Unfortunately there is currently no notification
 mechanism for new comment messages.
 ## Group Supervisor
 Group supervisor is typically the lecturer of a course. A user in this role can
 modify group description and properties, assign exercises or manage list of
 students. Further permissions like managing subgroups or supervisors is
 available only for group administrators.
 On "Dashboard" page you can find "Groups you supervise" section. Here there are
 boxes representing your groups with the list of students attending course and
 their points. Student names are clickable with redirection to the profile of the
 user where further information about his/hers assignments and solution can be
 found. To quickly jump onto groups page, use "Show group's detail" button at
 the bottom of the matching group box.
 ### Manage Group
 Locate group you supervise and you want to manage. All your supervised groups
 are available in sidebar under "Groups -- supervisor" collapsible menu. If you
 click on one of those you will be redirected to group detail page. In addition
 to basic group information you can also see "Supervisor's controls" section. In
 this section there are lists of current students and assignments.
 As a supervisor of group you are able to see "Edit group settings" button
 at the top of the page. Following this link will take you to group editation
 page with form containing these fields:
 - group name which is visible to other users
 - external identification which may be used for pairing with entries in an
  information system
 - description of group which will be available to users in instance (in
  Markdown)
 - set if group is publicly visible (and joinable by students) or private
 - options to set if students should be able see statistics of each other
 - minimal points threshold which students have to gain to successfully complete
  the course
 After filling all necessary fields the form can be sent by clicking on "Edit
 group" button and all changes will be applied.
 For students management there are "Students" and "Add student" boxes. The first
 one is simple list of all students which are attending the course with the
 possibility of delete them from the group. That can be done by hitting "Leave
 group" button near particular user. The second box is for adding students to the
 group. There is a text field for typing name of the student and after clicking
 on the magnifier image or pressing enter key there will appear list of matched
 users. At this moment just click on the "Join group" button and student will be
 signed in to your group.
 ### Assigning Exercises
 Before assigning an exercise, you obviously have to know what exercises are
 available. A list of all exercises in the system can be found under "Exercises"
 link in sidebar. This page contains a table with exercises names, difficulties
 and names of the exercise authors. Further information about exercise is
 available by clicking on its name.
 On the exercise details page are numerous information about it. There is a box
 with all possible localized descriptions and also a box with some additional
 information of exercise author, its difficulty, version, etc. There is also a
 description for supervisors by exercise author under "Exercise overview" option,
 where some important information can be found. And most notably there is an
 information about available programming languages for this exercise, under
 "Supported runtime environments" section.
 If you decide that the exercise is suitable for one of your groups, look for the
 "Groups" box at the bottom of the page. There is a list of all groups you
 supervise with an "Assign" button which will assign the exercise to the
 selected group.
 After clicking on the "Assign" button you should be redirected to assignment
 editation page. In there you can find two forms, one for editation of assignment
 meta information and the second one for setting exercise time and memory limits.
 In meta information form you can fill these options:
 - name of the assignment which will be visible in a group
 - visibility (if an assignment is under construction then you can mark it as not
  visible and students will not see it)
 - subform for localized descriptions (new localization can be added by clicking
  on "Add language variant" button, current one can be deleted with "Remove this
  language" button)
    - language of description from dropdown field (English, Czech, German)
    - description in selected language
 - score configuration which will be used on students solution evaluation, you
  can find some very simple one already in here, description of score
  configuration can be found further in "Writing score configuration" chapter
 - first submission deadline
 - maximum points that can be gained before the first deadline; if you want to
  manage all points manually, set it to 0 and then use bonus points, which are
  described in the next subchapter
 - second submission deadline, after that students still can submit exercises but
  they are given no points no points (must be after the first deadline)
 - maximum points that can be gained between first deadline and second deadline
 - submission count limit for the solutions of the students -- this limits the amount of
  attempts a student has at solving the problem
 - visibility of memory and time ratios; if true students can see the percentage
  of used memory and time (with respect to the limit) for each test
 - minimum percentage of points which each submission must gain to be considered
  correct (if it gets less, it will gain no points)
 - whether the assignment is marked as bonus one and points from solving it are
  not included into group threshold limit (that means solving it can get you
  additional points over the limit)
 The form has to be submitted with "Edit settings" button otherwise changes will
 not be saved.
 The same editation page serves also for the purpose of assignment editation, not
 only creation. That is why on bottom of the page "Delete the assignment" box
 can be found. Clearly the button "Delete" in there can be used to unassign
 exercise from group.
 The last unexplored area is the time and memory limits form. The whole form is
 situated in a box with tabs which are leading to particular runtime
 environments. If you wish not to use one of those, locate "Remove" button at the
 bottom of the box tab which will delete this environment from the assignment.
 Please note that this action is irreversible.
 In general, every tab in environments box contains some basic information about
 runtime environment and another nested tabbed box. In there you can find all
 hardware groups which are available for the exercise and set limits for all test
 cases. The time limits have to be filled in seconds (float), memory limits are
 in bytes (int). If you are interested in some reference values to particular
 test case then you can take a peek on collapsible "Reference solutions'
 evaluations" items. If you are satisfied with changes you made to the limits,
 save the form with "Change limits" button right under environments box.
 ### Management of The Solutions of The Students
 One of the most important tasks of a group supervisor is checking student
 solutions. As automatic evaluation of them cannot catch all problems in the
 source code, it is advisable to do a brief manual review of the coding
 style of the student and reflect that in assignment bonus points.
 On "Assignment detail" page there is an "View student results" button near top
 of the page (next to "Edit assignment settings" button). This will redirect you
 to a page where is a list of boxes, one box per student. Each student box
 contains a list of submissions for this assignment. The row structure of
 submission list is the same as the structure in the "Submitted solution"
 box. More information about every solution can be shown by clicking on "Show
 details" link on the end of solution row.
 This page is the same as for students with one exception -- there is an
 additional collapsed box "Set bonus points". In unfolded state, there is an
 input field for one number (positive or negative integer) and confirmation
 button "Set bonus points". After filling intended amount of points and
 submitting the form, the data in "Evaluation details" box get immediately
 updated. To remove assigned bonus points, submit just the zero number. The bonus
 points are not additive, newer value overrides older values.
 It is useful to give a feedback about the solution back to the user. For this
 you can use the "Comments and notes" box. Make sure that the messages are not
 private, so that the student can see them. More detailed description of this box
 can be nicely used the "Comments and notes" box. Make sure that the messages are
 not private, so the student can see them. More detailed description of this box
 is available in student part of user documentation.
 One of the discussed concept was marking one solution as accepted. However, due
 to lack of frontend developers it is not yet prepared in user interface. We
 hope, it will be ready as soon as possible. The button for accepting a solution
 will be most probably also on this page.
 ### Creating Exercises
 Link to exercise creation can be found in exercises list which is accessible
 through "Exercises" link in sidebar. On the bottom of the exercises list page
 you can find "Add exercise" button which will redirect you to exercise editation
 page. In this moment exercise is already created so if you just leave this page
 exercise will stay in the database. This is also reason why exercise creation
 form is the same as the exercise editation form.
 Exercise editation page is divided into three separate forms. First one is
 supposed to contain meta information about exercise, second one is used for
 uploading and management of supplementary files and third one manages runtime
 configuration in which exercise can be executed.
 First form is located in "Edit exercise settings" and generally contains meta
 information needed by frontend which are somehow somewhere visible. In here you
 can define:
 - exercise name which will be visible to other supervisors
 - difficulty of exercise (easy, medium, hard)
 - description which will be available only for visitors, may be used for further
  description of exercise (for example information about test cases and how they
  could be scored)
 - private/public switch, if exercise is private then only you as author can see
  it, assign it or modify it
 - subform containing localized descriptions of exercise, new one can be added
  with "Add language variant" button and current one deleted with "Remove this
  language"
    - language in which this particular description is in (Czech, English,
      German)
    - actual localized description of exercise
 After all information is properly set form has to be submitted with "Edit
 settings" button.
 Management of supplementary files can be found in "Supplementary files" box.
 Supplementary files are files which you can use further in job configurations
 which have to be provided in all runtime configurations. These files are
 uploaded directly to fileserver from where worker can download them and use
 during execution according to job configuration.
 Files can be uploaded either by drag and drop mechanism or by standard "Add a
 file" button. In opened dialog window choose file which should be uploaded. All
 chosen files are immediately uploaded to server but to save supplementary files
 list you have to hit "Save supplementary files" button. All previously uploaded
 files are visible right under drag and drop area, please note that files are
 stored on fileserver and cannot be deleted after upload.
 The last form on exercise editation page is runtime configurations editation
 form. Exercise can have multiple runtime configurations according to the number
 of programming languages in which it can be run. Every runtime configuration
 corresponds to one programming language because all of them has to have a bit
 different job configuration.
 New runtime configuration can be added with "Add new runtime configuration"
 button this will spawn new tab in runtime configurations box. In here you can
 fill following:
 - human readable identifier of runtime configuration
 - runtime environment which corresponds to programming language
 - job configuration in YAML, detailed description of job configuration can be
  found further in this chapter in "Writing job configuration" section
 If you are done with changes to runtime configurations save form with "Change
 runtime configurations" button. If you want to delete some particular runtime
 just hit "Remove" button in the right tab, please note that after this operation
 runtime configurations form has to be again saved to apply changes.
 All runtime configurations which were added to exercise will be visible to
 supervisors and all can be used in assignment, so please be sure that all of the
 languages and job configurations are working.
 If you choose to delete exercise, at the bottom of the exercise editation page
 you can find "Delete the exercise" box where "Delete" button is located. By
 clicking on it exercise will be delete from the exercises list and will no
 longer be available.
 ### Reference Solutions of An Exercise
 Each exercise should have a set of reference solutions, which are used to tune
 time and memory limits of assignments. Values of used time and memory for each
 solution are displayed in yellow boxes under forms for setting assignment limits
 as described earlier.
 However, there is currently no user interface to upload and evaluate reference
 solutions. It is possible to use direct REST API calls, but it is not much user
 friendly. If you are interested, please look at [API
 documentation](https://recodex.github.io/api/), notably sections
 _Uploaded-Files_ and _Reference-Exercise-Solutions_. You need to upload the
 reference solution files, create a new reference solution and then evaluate the
 solution. After that, measured data will be available in the box at assignment
 editing page (setting limits section).
 We are now working on a better user interface, which will be available soon.
 Then its description will be added here.
 ## Group Administrator
 Group administrator is the group supervisor with some additional permissions in
 particular group. Namely group administrator is capable of creating a subgroups
 in managed group and also adding and deleting supervisors. Administrator of the
 particular group can be only one person.
 ### Creating Subgroups And Managing Supervisors
 There is no special link which will get you to groups in which you are
 administrator. So you have to get there through "Groups - supervisor" link in
 sidebar and choose the right group detail page. If you are there you can see
 "Administrator controls" section, here you can either add supervisor to group or
 create new subgroup.
 Form for creating a subgroup is present right on the group detail page in "Add
 subgroup" box. Group can be created with following options:
 - name which will be visible in group hierarchy
 - external identification, can be for instance ID of group from school system
 - some brief description about group
 - allow or deny users to see each others statistics from assignments
 After filling all the information a group can be created by clicking on "Create
 new group" button. If creation is successful then the group is visible in
 "Groups hierarchy" box on the top of page. All information filled during
 creation can be later modified.
 Adding a supervisor to a group is rather easy, on group detail page is an "Add
 supervisor" box which contains text field. In there you can type name or
 username of any user from system. After filling user name, click on the
 magnifier image or press the enter key and all suitable users are searched. If
 your chosen supervisor is in the updated list then just click on the "Make
 supervisor" button and new supervisor should be successfully set.
 Also, existing supervisor can be removed from the group. On the group detail
 page there is "Supervisors" box in which all supervisors of the group are
 visible. If you are the group administrator, you can see there "Remove
 supervisor" buttons right next to supervisors names. After clicking on it some
 particular supervisor should not to be supervisor of the group anymore.
 ## Instance Administrator
 Instance administrator can be only one person per instance. In addition to
 previous roles this administrator should be able to modify the instance details,
 manage licences and take care of top level groups which belong to the instance.
 ### Instance Management
 List of all instances in the system can be found under "Instances" link in the
 sidebar. On that page there is a table of instances with their respective
 admins. If you are one of them, you can visit its page by clicking on the
 instance name. On the instance details page you can find a description of the
 instance, current groups hierarchy and a form for creating a new group.
 If you want to change some of the instance settings, follow "Edit instance" link
 on the instance details page. This will take you to the instance editation page
 with corresponding form. In there you can fill following information:
 - name of the instance which will be visible to every other user
 - brief description of instance and for whom it is intended
 - checkbox if instance is open or not which means public or private (hidden from
  potential users)
 If you are done with your editation, save filled information by clicking on
 "Update instance" button.
 If you go back to the instance details page you can find there a "Create new
 group" box which is able to add a group to the instance. This form is the same
 as the one for creating subgroup in already existing group so we can skip
 description of the form fields. After successful creation of the group it will
 appear in "Groups hierarchy" box at the top of the page.
 ### Licences
 On the instance details page, there is a box "Licences". On the first line, it
 shows it this instance has currently valid licence or not. Then, there are
 multiple lines with all licences assigned to this instance. Each line consists
 of a note, validity status (if it is valid or revoked by superadministrator) and
 the last date of licence validity.
 A box "Add new licence" is used for creating new licences. Required fields are
 the note and the last day of validity. It is not possible to extend licence
 lifetime, a new one should be generated instead. It is possible to have more
 than one valid licence at a time. Currently there is no user interface for
 revoking licences, this is done manually by superadministrator. If an instance
 is to be disabled, all valid licences have to be revoked.
 ## Superadministrator
 Superadministrator is a user with the most privileges and as such superadmin
 should be quite a unique role. Ideally, there should be only one user of this
 kind, used with special caution and adequate security. With this stated it is
 obvious that superadmin can perform any action the API is capable of.
 ### Management of Users
 There are only a few user roles in ReCodEx. Basically there are only three:
 _student_, _supervisor_, and _superadmin_. Base role is student which is
 assigned to every registered user. Roles are stored in database alongside other
 information about user. One user always has only one role at the time. At first
 startup of ReCodEx, the administrator has to change the role for his/her account
 manually in the database. After that manual intervention into database should
 never be needed.
 There is a little catch in groups and instances management. Groups can have
 admins and supervisors. This setting is valid only per one particular group and
 has to be separated from basic role system. This implies that supervisor in one
 group can be student in another and simultaneously have global supervisor role.
 Changing role from student to supervisor and back is done automatically when the
 new privileges are granted to the user, so managing roles by hand in database is
 not needed. Previously stated information can be applied to instances as well,
 but instances can only have admins.
 Roles description:
 - Student -- Default role which is used for newly created accounts. Student can
  join or leave public groups and submit solutions of assigned exercises.
 - Supervisor -- Inherits all permissions from student role. Can manage groups to
  which he/she belongs to. Supervisor can also view and change groups details,
  manage assigned exercises, view students in group and their solutions for
  assigned exercises. On top of that supervisor can create/delete groups too,
  but only as subgroup of groups he/she belongs to.
 - Superadmin -- Inherits all permissions from supervisor role. Most powerful
  user in ReCodEx who should be able to do access any functionality provided by
  the application.
 ## Writing Score Configuration
 An important thing about assignment is how to assign points to particular
 solutions. As mentioned previously, the whole job is composed of logical tests.
 All of these tests have to contain one essential "evaluation" task. Evaluation
 task should output one float number which can be further used for scoring of
 particular tests.
 Total resulting score of the students solution is then calculated according to a
 supplied score config (described below) and using specified calculator. Total
 score is also a float between 0 and 1. This number is then multiplied by the
 maximum of points awarded for the assignment by the teacher assigning the
 exercise -- not the exercise author.
 For now, there is only one way how to write score configuration using only
 simple score calculator. But the implementation in API is agile enough to handle
 upcoming score calculators which might use some more complex scoring algorithms.
 This also means that future calculators do not have to use the YAML format for
 configuration. In fact, the configuration can be a string in any format.
 ### Simple Score Calculation
 First implemented calculator is simple score calculator with test weights. This
 calculator just looks at the score of each test and put them together according
 to the test weights specified in assignment configuration. Resulting score is
 calculated as a sum of products of score and weight of each test divided by the
 sum of all weights. The algorithm in Python would look something like this:
 ```
 sum = 0
 weightSum = 0
 for t in tests:
  sum += t.score * t.weight
  weightSum += t.weight
 score = sum / weightSum
 ```
 Sample score config in YAML format:
 ```{.yml}
 testWeights:
  a: 300   # test with id 'a' has a weight of 300
  b: 200
  c: 100
  d: 100
 ```
 ## Writing Job Configuration
 To run and evaluate an exercise the backend needs to know the steps how to do
 that. This is different for each environment (operation system, programming
 language, etc.), so each of the environments needs to have separate
 configuration.
 Backend works with a powerful, but quite low level description of simple
 connected tasks written in YAML syntax. More about the syntax and general task
 overview can be found on [separate
 page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned
 features was user friendly configuration editor, but due to tight deadline and
 team composition it did not make it to the first release. However, writing
 configuration in the basic format will be always available and allows users to 
 use the full expressive power of the system.
 This section walks through creation of job configuration for _hello world_
 exercise. The goal is to compile file _source.c_ and check if it prints `Hello
 World!` to the standard output. This is the only test case named **A**.
 The problem can be split into several tasks:
 - compile _source.c_ into _helloworld_ with `/usr/bin/gcc`
 - run _helloworld_ and save standard output into _out.txt_
 - fetch predefined output (suppose it is already uploaded to fileserver) with
  hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_
 - compare _out.txt_ and _reference.txt_ by `/usr/bin/diff`
 The absolute path of tools can be obtained from system administrator. However,
 `/usr/bin/gcc` is location, where the GCC binary is available almost everywhere,
 so location of some tools can be (professionally) guessed.
 First, write header of the job to the configuration file. 
 ```{.yml}
 submission:
    job-id: hello-word-job
    hw-groups:
        - group1
 ```
 Basically it means, that the job _hello-world-job_ needs to be run on workers
 that belong to the `group_1` hardware group . Reference files are downloaded
 from the default location configured in API (such as
 `http://localhost:9999/exercises`) if not stated explicitly otherwise. Job
 execution log will not be saved to result archive.
 Next the tasks have to be constructed under _tasks_ section. In this demo job,
 every task depends only on previous one. The first task has input file
 _source.c_ (if submitted by user) already available in working directory, so
 just call the GCC. Compilation is run in sandbox as any other external program
 and should have relaxed time and memory limits. In this scenario, worker
 defaults are used. If compilation fails, the whole job is immediately terminated
 (because the _fatal-failure_ bit is set). Because _bound-directories_ option in 
 sandbox limits section is mostly shared between all tasks, it can be set in 
 worker configuration instead of job configuration (suppose this for following 
 tasks). For configuration of workers please contact your administrator.
 ```{.yml}
 - task-id: "compilation"
  type: "initiation"
  fatal-failure: true
  cmd:
      bin: "/usr/bin/gcc"
      args:
          - "source.c"
          - "-o"
          - "helloworld"
  sandbox:
      name: "isolate"
      limits:
          - hw-group-id: group1
            chdir: ${EVAL_DIR}
            bound-directories:
                - src: ${SOURCE_DIR}
                  dst: ${EVAL_DIR}
                  mode: RW
 ```
 The compiled program is executed with time and memory limit set and the standard
 output is redirected to a file. This task depends on _compilation_ task, because
 the program cannot be executed without being compiled first. It is important to
 mark this task with _execution_ type, so exceeded limits will be reported in
 frontend.
 Time and memory limits set directly for a task have higher priority than worker
 defaults. One important constraint is, that these limits cannot exceed limits
 set by workers. Worker defaults are present as a safety measure so that a 
 malformed job configuration cannot block the worker forever. Worker default 
 limits should be reasonably high, like a gigabyte of memory and several hours of 
 execution time. For exact numbers please contact your administrator.
 It is important to know that if the output of a program (both standard and 
 error) is redirected to a file, the sandbox disk quotas apply to that file, as 
 well as the files created directly by the program. In case the outputs are 
 ignored, they are redirected to `/dev/null`, which means there is no limit on 
 the output length (as long as the printing fits in the time limit).
 ```{.yml}
 - task-id: "execution_1"
  test-id: "A"
  type: "execution"
  dependencies:
      - compilation
  cmd:
      bin: "helloworld"
  sandbox:
      name: "isolate"
      stdout: ${EVAL_DIR}/out.txt
      limits:
          - hw-group-id: group1
            chdir: ${EVAL_DIR}
            bound-directories:
                - src: ${SOURCE_DIR}
                  dst: ${EVAL_DIR}
                  mode: RW
            time: 0.5
            memory: 8192
 ```
 Fetch sample solution from file server. Base URL of file server is in the header 
 of the job configuration, so only the name of required file (its `sha1sum` in 
 our case) is necessary.
 ```{.yml}
 - task-id: "fetch_solution_1"
  test-id: "A"
  dependencies:
      - execution
  cmd:
      bin: "fetch"
      args:
          - "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b"
          - "${SOURCE_DIR}/reference.txt"
 ```
 Comparison of results is quite straightforward. It is important to set the task
 type to _evaluation_, so that the return code is set to 0 if the program is
 correct and 1 otherwise. We do not set our own limits, so the default limits are
 used.
 ```{.yml}
 - task-id: "judge_1"
  test-id: "A"
  type: "evaluation"
  dependencies:
      - fetch_solution_1
  cmd:
      bin: "/usr/bin/diff"
      args:
          - "out.txt"
          - "reference.txt"
  sandbox:
      name: "isolate"
      limits:
          - hw-group-id: group1
            chdir: ${EVAL_DIR}
            bound-directories:
                - src: ${SOURCE_DIR}
                  dst: ${EVAL_DIR}
                  mode: RW
 ```
 <!---
 // vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
 -->
--- a/Web-API.md
+++ b/Web-API.md
@ -1,90 +0,0 @@
 # Web API
 ## Description
 The Web API provides a controlled access to the evaluation backend. It also 
 enables the use of different user frontends such as default web application, 
 mobile applications, commandline tools and possibly others. The communication 
 goes as HTTP(S) requests in predefined format, nowadays mostly known as 
 [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) format. 
 Results from the API are in plain text in JSON format to be easily parsed in 
 various languages (notably JavaScript).
 This component must be publicly visible on the internet, so it is important to 
 care about security and follow our recommendations. Security and user access 
 restriction are among our primary concerns, so proper roles with permission 
 separation are introduced and maintained. Also some additional checks are made 
 directly in the code, so that a user cannot access information which are out of 
 their authorization.
 ## Architecture
 Web API is written in PHP using [Nette framework](https://nette.org/en/). This framework provides useful components like _Tracy_ for logging and showing errors, _Tester_ for productive unit testing or _Latte_ templating engine. Nette is modern, widely used and great performing software with active developers and user community. Nette can help eliminate security holes, simplify debugging and make coding easier with numerous plugins and extensions. Also, it is published under permissive BSD license.
 API architecture consists of several parts:
 - **router** -- component handling mapping from URL addresses to methods in presenter classes (called endpoints)
 - **presenters** -- classes containing one method per endpoint responsible for 
  fetching and parsing request arguments and performing desired actions
 - **entities** -- classes persisted using a database with an ORM framework
 - **repositories** -- common operations on entities of one type, mostly finding entity by identifier or persisting changes to the database
 - **helpers** -- set of classes solving more complicated internal logic, used from presenters to keep them reasonably small
 Each presenter method has several annotations. They are used for generating REST API documentation in [Swagger](http://swagger.io/), specifying request type and its parameters and specifying one level of access restrictions. Also, there is simple description of the endpoint.
 For specifying the request type (_GET_, _POST_, _DELETE_) annotations with exactly these names without any parameters are used. To describe request parameters `@Param` annotation is used with following arguments:
 - type -- the type of argument, one of _post_ of _query_
 - name -- name of the argument (the key)
 - validation -- validation of the value, see [Nette validation rules](https://doc.nette.org/en/2.4/validators#toc-rules)
 - msg -- description for users about the values this parameter can contain
 - required -- specifies if this option is mandatory (`true`, default) or optional (`false`)
 - description -- description for documentation of the API
 Another annotation is `@LoggedIn` which takes no arguments. It can be placed before a whole class or before a method, so requests from unauthorized users are forbidden. Permissions can be granted or prohibitted by `@UserIsAllowed` annotation. This one is only per method and takes one argument in `key="value"` format. The value specifies which action (_value_) of a resource (_key_) the user needs to be allowed to perform this request. An example of how an annotated endpoint can look like:
 ```{.php}
 /**
 * Create a user account
 * @POST
 * @LoggedIn
 * @UserIsAllowed(users="create")
 * @Param(type="post", name="email", validation="email",
 *    description="An email that will serve as a login name")
 * @Param(type="post", name="name", validation="string:2..",
 *    description="First name")
 * @Param(type="post", name="password", validation="string:1..",
 *    msg="Password cannot be empty.",
 *    description="A password for authentication")
 */
 public function actionCreateAccount() {
    ...
 }
 ```
 The [Doctrine](https://github.com/Kdyby/Doctrine) 
 [ORM](https://en.wikipedia.org/wiki/Object-relational_mapping) framework is used 
 as an object persistence layer. It provides simple to use annotations to specify 
 columns of database tables including types, indexes and also it is possible to 
 make mapping between entities. For detailed info refer to [official 
 documentation](http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/).
 The API is capable of sending email messages. They can inform an administrator about errors and users about submission evaluation or a temporary link to change forgotten password. The [Nette Mail](https://doc.nette.org/en/2.4/mailing) extension provides nice interface for sending messages through external SMTP server (**preferred**) or builtin PHP function `mail`. It is important to set up the mailserver properly to ensure message delivery to the clients. The messages are rendered in HTML format via simple _Latte_ templates.
 ### Authentication
 Instead of relying on PHP sessions, we decided to use an authentication flow 
 based on JWT tokens (RFC 7519). On successful login, the user is issued an 
 access token that they have to send with subsequent requests using the HTTP 
 Authorization header (`Authorization: Bearer <token>`). The token has a limited 
 validity period and has to be renewed periodically using a dedicated API 
 endpoint. 
 To implement this behavior in Nette framework, a new `IUserStorage` 
 implementation was created, along with an `IIdentity` and authenticators for 
 both our internal login service and CAS.
 An advantage of this approach is being able control the authentication process
 completely instead of just receiving session data through a global variable.
--- a/Web-application.md
+++ b/Web-application.md
@ -1,155 +0,0 @@
 # Web application
 ## Description
 The *WebApp* is the default user interface of the ReCodEx project. It is a Single Page Application (SPA) which users access through their web browsers.
 This user interface allows students to join *groups* corresponding to their school classes, submit new *solutions* and see the results as soon as the solution is evaluated by the backend.
 Teachers can manage their *groups* by adding and removing *students*, assigning *exercises* and examining the *automatically evaluated solutions* of their students and their total score gained throughout the semester.
 The actions performed by the user are sent to the *API server* over the HTTPS protocol and the web application does not modify the database or access files on their server directly and all the inputs sent
 from the web application are additionally *validated* on the *API server*.
 The *WebApp* is written in *ECMAScript* (often referred to as JavaScript) and HTML5. The [React](https://facebook.github.io/react) library developed by Facebook is used to manage the user interface and the [Redux](http://redux.js.org/) framework
 created by Dan Abramov is used to manage the application state. Both of these frameworks are industry standards used in production by many companies worldwide. Many other smaller libraries and components are used, notably the [Immutable.js](https://facebook.github.io/immutable-js) and [Format.js](http://formatjs.io/) by Yahoo via *redact-intl* integration library.
 ### Multiplatform approach
 Since the runtime environment of *WebApp* is the user's web browser it does not need any installation on the enduser's device and does not require any particular platform or non-standard software installed on the user's device.
 This application is designed and implemented in a way which should be suitable for modern web browsers on both desktop and mobile devices.
 ## Architecture
 The whole project is written using the next generation of JavaScript referred to as *ECMAScript 6* (also known as *ES6*, *ES.next*, or *Harmony*). Since not all the features introduced in this standard are implemented in today's modern web browsers (like classes and spread operators) and hardly any are implemented in older versions of web browsers which are currently still in use, the source code is transpiled into the older standard *ES5* using [Babel.js](https://babeljs.io/) transpiler and bundled into a single script file using the [webpack](https://webpack.github.io/) moudle bundler. The need for a transpiler also arises from the usage of the *JSX* syntax for declaring React components. To read more about these these tools and their usage please refer to the [installation section](#Installation). The whole bundling process takes place at deployment and is not repeated afterwards.
 ### Routing and server side rendering
 The routing mechanism uses the [react-router](https://github.com/ReactTraining/react-router) and [react-router-redux](https://github.com/reactjs/react-router-redux) libraries. The routing mechanism works both on the server and the client.
 The routes are defined in the `/src/pages/routes.js` file.
 ### Server rendering and the 'loadAsync' convention
 The server matches the URL of the request with the router and pre-renders the contents of the page. The components, which are selected by the router based on the URL, are given the parameters and are attached to the component tree.
 The server script examines all the components matched with the route and looks for a static function called `loadAsync` and if it is found, it will be executed with the params extracted from the URL, the store's dispatch method and the ID of the logged in user if he is logged in. It is expected that this function will return an instance of `Promise`. Server then waits until all the promises are resolved and only after then it goes on and sends a HTTP response to the client.
 ### Client
 Each time the user clicks on a link to a different location of the application the page is not reloaded but the router rather changes the location of the browser to the new URL and replaces the components in the component tree with the ones matched against the new URL the same way the server did. Components should call the `loadAsync` methods themselves in their lifecycle methods.
 ### Redux middleware and helpers
 The redux [store](https://github.com/ReCodEx/web-app/blob/master/src/redux/store.js) is extended with several middlewares - [redux-thunk](https://github.com/gaearon/redux-thunk), [redux-promise-middleware](https://github.com/pburtchaell/redux-promise-middleware), router middleware, accessTokenMiddleware, and apiMiddleware.
 #### apiMiddleware
 The API middleware catches all the `CALL_API` actions and turns them into a HTTP request via the [fetch API](https://developer.mozilla.org/en/docs/Web/API/Fetch_API). The middleware deeply converts data in the *body* part of the request to `FormData` and sets correctly the headers and creates endpoints for the API server before each request is sent. The middleware also handle the responses - converts them to the format expected by the reducers and catches requests' failures.
 #### accessTokenMiddleware
 [Access token middleware](https://github.com/ReCodEx/web-app/blob/master/src/redux/middleware/accessTokenMiddleware.js) stores and keeps the access token once the user authenticates and receives a token from the API server. The middleware listens for dispatched API actions and inserts the token to the request automatically.
 #### Modules
 Modules (sometimes called "ducks") contain a reducer and related action types and action creators. There are many separate reducers and actions for each individual resource and task to make the store tree as flat as possible. Vast majority of the modules is derived from the *resource manager* and are very similar to each other. Modules are placed in the `/src/redux/modules` directory.
 #### Selectors
 Selectors are used from the *containers* to return a specific information from the global redux state. It would be very inconvenient if the containers needed to know the whole structure of the state, so they can rely on the functions provided by the selectors for each module. The [reselect](https://github.com/reactjs/reselect) library is used for creating efficient selectors. Selectors are placed in the `/src/redux/selectors` directory.
 #### Resource manager
 The [resource manager](https://github.com/ReCodEx/web-app/blob/master/src/redux/helpers/resourceManager/index.js) is a set of factory functions which was created to extract the repeating functionality of obtaining and managing the loading phases of a resource or multiple resources at once.
 The factory methods create the reducer, action types, and action creators. It strongly relies on the *redux-promise-middleware*. It was inspired by the [Reddit API example](http://redux.js.org/docs/advanced/ExampleRedditAPI.html) from the Redux library official website.
 Resource manager is placed in the `/src/redux/helpers/resourceManager` directory.
 ### Declarative UI components
 The implementation of *WebApp* is split across more than a hundred small components which are composed together. There are two basic types of React components - presentational and stateful components.
 - **Presentational components** are in fact pure functions which yield their output based solely on the input *props* and application's *context* (e. g. current language has effect on URLs). These components are preferred and should be as simple as possible. These components are placed in the subfolders of `src/components` directory.
 - **Stateful components** are *connected* to the *redux store* and yield information based on the either the *props* and *context*, but also on the current state of the application. These components are more complicated and often make use of React's components' life cycle methods. *Stateful components* are often referred to as *containers* and the source code files are placed in a separate directories called `src/containers` and `src/pages`.
    - There is no technical difference between *containers* and *pages*. *Pages* are used as "root" components for different pages and are registered in the app's [router](https://github.com/ReactTraining/react-router).
 As was mentioned earlier in the text, there is over a hundred components in the whole application and most of them are very simple. Some of the most important and widely used components are described in the following paragraphs.
 #### App
 [App](https://github.com/ReCodEx/web-app/blob/master/src/containers/App/App.js) container is meant to be the root of the *react-router* tree. It checks whether a user is logged in and loads his profile and settings from the API. App container also handles access token refreshing.
 #### LayoutContainer
 [LayoutContainer](https://github.com/ReCodEx/web-app/blob/master/src/containers/LayoutContainer/LayoutContainer.js) handles the dependency injection of the localized links based on the current language stated in the URL. It also controls the state of the sidebar - collapsing and showing the sidebar.
 ##### Layout, Header, Sidebar, Footer
 The [Layout](https://github.com/ReCodEx/web-app/blob/master/src/components/Layout/Layout.js), [Header](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Header/Header.js), [Header](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Footer/Footer.js), and [Sidebar](https://github.com/ReCodEx/web-app/blob/master/src/components/Sidebar/Sidebar/Sidebar.js) components hold the main HTML structure of the page which is displayed to the user and hold links to different parts of the application.
 #### PageContent
 The [PageContent](https://github.com/ReCodEx/web-app/blob/master/src/components/PageContent/PageContent.js) component holds the main content of a page with the common structure for all pages - the *title*, *description*, *[breadcrumbs](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Breadcrumbs/Breadcrumbs.js),*, *content*. The component passes the title and description to the [Helmet](https://github.com/nfl/react-helmet) library which reflects these into the `<head>` section of the HTML document.
 #### ResourceRenderer
 [ResourceRenderer](https://github.com/ReCodEx/web-app/blob/master/src/components/ResourceRenderer/ResourceRenderer.js) component is given a *resource* managed by the *resourceManager* as a *prop* and displays different content based on the state of the given *resource* - still loading, loading failed, fully loaded.
 Passing content for the *loading* and *failed* states though *props* is optional; however, the content for the *loaded* state is required and must be passed as a child to the `ResourceManager`.
 Multiple resources can be passed as an array to the component and it will wait in the *loading* state until some of the resources are still loading. If one of the resources fails to load the component will switch to the *failed* state. When all the files are fully loaded then the component displays the content for the *loaded* state.
 It is worth mentioning that the component is completely stateless and "switching of the states" is the effect of several renderings of the component over time.
 #### Page
 [Page](https://github.com/ReCodEx/web-app/blob/master/src/components/Page/Page.js) combines the two previously described components, since they are often used together for displaying a page content which is dependant on a resource.
 #### Pages
 The main pages of the application are placed in the `/src/pages/` directory. These pages are referenced from *the router* and are responsible mainly for initiating loading of the main resources when the component will be mounted to the DOM.
 #### EvaluationProgressContainer
 #### CommentThreadContainer
 [CommentThreadContainer](https://github.com/ReCodEx/web-app/blob/master/src/containers/CommentThreadContainer/CommentThreadContainer.js) can be used on any page of the *webapp* to create a discussion thread. A unique identification must be assigned to every thread so the comments in the distinct threads do not interleave.
 #### Bootstrap and AdminLTE theme
 The UI of the application is built using the stylesheets from the [Bootstrap framework](http://getbootstrap.com/) from Twitter and the [AdminLTE](https://almsaeedstudio.com/) theme from Abdullah Almsaeed. None of the JavaScript plugins from these libraries is used.
 A package [react-bootstrap](https://react-bootstrap.github.io/) is used. Components specific to the *AdminLTE* theme are implemented and stored in the `src/components/AdminLTE` folder and its subfolders.
 ##### Box, FormBox
 [Box](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Box/Box.js) is a frequently used component for bounding other components like text paragraphs or tables inside. It is in fact a re-styled `Panel` component from Bootstrap. It can be collapsable and can be displayed in different colors and types. [FormBox](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/FormBox/FormBox.js) is just a wrapper of `Box` adjusted for holding forms inside of the body.
 ##### CommentThread
 [CommentThread](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/CommentThread/CommentThread.js) component is used directly by the earlier described `CommentThreadContainer` and uses the HTML markup introduced by the theme author to achieve nice comment threads.
 ##### Avatar, LoadingAvatar, FailedAvatar
 [Avatar](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Avatar/Avatar.js) component is used on many places for displaying a round profile picture of a user. `LoadingAvatar` and `FailedAvatar` are used to mock the visual appearance of the avatar while the image is being downloaded or if the download failed for some reason.
 #### Icons
 The free [FontAwesome](http://fontawesome.io/) icon pack is used in the application via the [react-fontawesome](https://github.com/danawoodman/react-fontawesome) library. Many types of [icons](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Header/Header.js) are defined as components (e. g. `LoadingIcon`, `WarningIcon`, `SuccessIcon`, `DeleteIcon`, and many others) to be used throughout the application so the same symbols are used for the same purposes.
 #### Forms
 The [redux-form](http://redux-form.com/) library is used for the management of forms' states. Several of the implemented components can be used as form fields, namely the [CheckboxField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/CheckboxField.js), [MarkdownTextAreaField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/MarkdownTextAreaField.js), [SourceCodeField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/SourceCodeField.js), [DatetimeField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/DatetimeField.js), [TabbedArrayField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/TabbedArrayField.js), [LanguageSelectField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/LanguageSelectField.js), and a few others.
 ### Localization and globalization
 The whole application is prepared for localization. All the text can be exported from the user interface and translated into several languages. The numbers, dates, and time values are also formatted with respect to the selected language. The [react-intl](https://github.com/yahoo/react-intl) and [Moment.js](http://momentjs.com/) libraries are used to achieve this.
 All the strings can be extracted from the application using a command:
 ```
 $ npm run exportStrings
 ```
 This will create *JSON* files with the exported strings for the *'en'* and *'cs'* locale. If you want to export strings for more languages, you must edit the `/manageTranslations.js` script. The exported strings are placed in the `/src/locales` directory.