Update

Rewritten-docs.md

@@ -38,47 +38,49 @@ Analysis
 
 ### Current solution at MFF UK
 
-The ideas presented above are not completely new. There was a group of students, 
+The ideas presented above are not completely new. There was a group of
-who already implemented an evaluation solution for student's homeworks in 2006. 
+students, who already implemented an evaluation solution for student's
-Its name is [CodEx - The Code Examiner](http://codex.ms.mff.cuni.cz/project/) 
+homeworks in 2006.  Its name is [CodEx - The Code
-and it has been used with some improvements since then. The original plan was to 
+Examiner](http://codex.ms.mff.cuni.cz/project/) and it has been used with some
-use the system only for basic programming courses, but there is demand for 
+improvements since then. The original plan was to use the system only for basic
-adapting it for many different subjects.
+programming courses, but there is demand for adapting it for many different
-
+subjects.
-CodEx is based on dynamic analysis. It features a web-based interface, where 
+
-supervisors assign exercises to their students and the students have a time 
+CodEx is based on dynamic analysis. It features a web-based interface, where
-window to submit the solution. Each solution is compiled and run in sandbox 
+supervisors assign exercises to their students and the students have a time
-(MO-Eval). The metrics which are checked are: corectness of the output, time and 
+window to submit the solution. Each solution is compiled and run in sandbox
-memory limits. It supports programs written in C, C++, C#, Java, Pascal, Python 
+(MO-Eval). The metrics which are checked are: corectness of the output, time
-and Haskell.
+and memory limits. It supports programs written in C, C++, C#, Java, Pascal,
+Python and Haskell.
 
 Current system is old, but robust. There were no major security incidents
 during its production usage. However, from today's perspective there are
 several drawbacks. The main ones are:
 
-- **web interface** -- The web interface is simple and fully functional. But 
+- **web interface** -- The web interface is simple and fully functional. But
-  rapid development in web technologies opens new horizons of how web interface 
+  rapid development in web technologies opens new horizons of how web interface
   can be made.
-- **web api** -- CodEx offers a very limited XML API based on outdated 
+- **web api** -- CodEx offers a very limited XML API based on outdated
-  technologies that is not sufficient for users who would like to create custom 
+  technologies that is not sufficient for users who would like to create custom
   interfaces such as a command line tool or mobile application.
-- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system 
+- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system
-  calls and blocking the bad ones. This can be easily done for single-threaded 
+  calls and blocking the bad ones. This can be easily done for single-threaded
   applications, but proves difficult with multi-threaded ones. In present day,
-  parallelism is a very important area of computing, so there is requirement to 
+  parallelism is a very important area of computing, so there is requirement to
   test multi-threaded applications too.
-- **instances** -- Different ways of CodEx usage scenarios requires separate 
+- **instances** -- Different ways of CodEx usage scenarios requires separate
-  instances (Programming I and II, Java, C#, etc.). This configuration is not 
+  instances (Programming I and II, Java, C#, etc.). This configuration is not
-  user friendly (students have to register in each instance separately) and 
+  user friendly (students have to register in each instance separately) and
-  burdens administrators with unnecessary work. CodEx architecture does not 
+  burdens administrators with unnecessary work. CodEx architecture does not
-  allow sharing hardware between instances, which results in an inefficient use 
+  allow sharing hardware between instances, which results in an inefficient use
   of hardware for evaluation.
-- **task extensibility** -- There is a need to test and evaluate complicated 
+- **task extensibility** -- There is a need to test and evaluate complicated
-  programs for classes such as Parallel programming or Compiler principles, 
+  programs for classes such as Parallel programming or Compiler principles,
-  which have a more difficult evaluation chain than simple 
+  which have a more difficult evaluation chain than simple
   compilation/execution/evaluation provided by CodEx.
 
-Supervisors and current users of CodEx gives us following ideas for improvements:
+Supervisors and current users of CodEx gives us following ideas for
+improvements:
 
 - keep it simple
 - localization (both UI and exercises)
@@ -421,8 +423,12 @@ deserves some.
 
 *A runtime environment* defines the used programming language or tools
 which are needed to process and evaluate a solution. Examples of a
-runtime environment can be “*Linux + GCC*”, “*Linux + Mono*”, “*Windows
+runtime environment can be:
-+ .NET 4*”, “*Bison + Yacc.”*
+
+- *Linux + GCC*
+- *Linux + Mono*
+- *Windows + .NET 4*
+- *Bison + Yacc*
 
 ### Limits
 
@@ -562,14 +568,10 @@ of entities and relational database models), describe the logical
 grouping of entities and how they are related:
 
 -   user + settings + logins + ACL
-
 -   instance + licences + groups + group membership
-
 -   exercise + assignments + localized assignments + runtime
     environments + hardware groups
-
 -   submission + solution + reference solution + solution evaluation
-
 -   comment threads + comments
 
 ### API endpoints
@@ -607,15 +609,34 @@ can accept it or decline it
 Job processing progress monitoring
 ----------------------------------
 
-@todo: How does the Backend push the progress of evaluation through the
+When evaluating a job the worker sends progress messages on predefined points of
-Monitor to the Frontend
+evaluation chain. The sending place can be on very beginning of the job, when
+submit archive is downloaded or at the end of each simple task with its state
+(completed, failed, skipped). These messages are sent to broker through existing
+ZeroMQ connection. Detailed format of messages can be found on [communication
+page](https://github.com/ReCodEx/wiki/wiki/Overall-architecture#commands-from-worker-to-broker).
+
+Broker only resends received progress messages to the monitor component via
+ZeroMQ socket. The output message format is the same as the input format.
+
+Monitor parses received messages to JSON format, which is easy to work with in
+JavaScript inside web application. All messages are cached (one queue per job)
+and can be obtained multiple times through WebSocket communication channel. The
+cache is cleared 5 minutes after receiving last message.
 
 Publishing of the results
 -------------------------
 
-@todo: How does the Backend publish the results and how it notifies the
+After job finish the worker packs results directory into single archive and
-Frontend; How the Frontend can request the results if the notification
+uploads it to the fileserver through HTTP protocol. The target URL is obtained
-does not arrive
+from API in headers on job initiation. Then "job done" notification request is
+performed to API via broker. Special submissions (reference or asynchronous
+submissions) are loaded immediately, other types are loaded on-demand on first
+results request.
+
+Loading results means fetching archive from fileserver, parsing the main YAML
+file generated by worker and saving data to the database. Also, points are
+assigned by score calculator.
 
 User documentation
 ==================
@@ -635,6 +656,7 @@ Supervisor, Admin
 @todo: Describe the requirements of running the web application (modern
 web browser, enabled CSS, JavaScript, Cookies & Local storage)
 
+
 ### Scenario \#1: Becoming a user of ReCodEx
 
 #### How to create a user account?
@@ -678,8 +700,7 @@ in the top section of the left sidebar right under your name. You will
 have to expand the sidebar with a button next to the “*ReCodEx*” title
 (shown in the picture below).
 
-![](media/image1.png){width="3.3131944444444446in"
+@todo: Simon's image
-height="0.6173611111111111in"}
 
 #### What to do when you cannot remember your password?
 
@@ -705,15 +726,14 @@ website.
 There are several options you have to edit your user account.
 
 -   changing your personal information (i.e., name)
-
 -   changing your credentials (email and password)
-
 -   updating your preferences (e.g., source code viewer/editor settings,
     default language)
 
 You can access the settings page through the “*Settings*” button right
 under your name in the left sidebar.
 
+
 ### Scenario \#2: User is a student
 
 @todo: describe what it means to be a “student” and what are the
@@ -749,6 +769,7 @@ although the results say different
 who else can see the comments, how notifications work (*not implemented
 yet*!).
 
+
 ### Scenario \#3: User is supervisor of a group
 
 @todo: describe what it means to be a “supervisor” of a group and what
@@ -804,7 +825,7 @@ the awarded points will be overridden.
 #### How can I discuss student’s solution with him/her directly through the web application?
 
 @todo: Describe the comments thread behavior (public/private comments),
-who else can see the comments
+who else can see the comments -- same as from the student perspective
 
 
 ### Writing job configuration
@@ -824,8 +845,9 @@ configuration in the basic format will be always available and allows users to
 use the full expressive power of the system.
 
 This section walks through creation of job configuration for _hello world_
-exercise. The goal is to compile file _source.c_ and check if it prints `Hello 
+exercise. The goal is to compile file _source.c_ and check if it prints `Hello
-World!` to the standard output.
+World!` to the standard output. This is the only test case, let's call it
+**A**.
 
 The problem can be split into several tasks:
 
@@ -915,7 +937,7 @@ written.
       bin: "helloworld"
   sandbox:
       name: "isolate"
-	  stdout: ${EVAL_DIR}/out.txt
+      stdout: ${EVAL_DIR}/out.txt
       limits:
           - hw-group-id: group1
             chdir: ${EVAL_DIR}