From 8ac570d272b7cfa044a48aeb56562ec0bad937b5 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Thu, 15 Dec 2016 21:15:02 +0100 Subject: [PATCH] Update --- Rewritten-docs.md | 118 +++++++++++++++++++++++++++------------------- 1 file changed, 70 insertions(+), 48 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 9a699d0..58cac5e 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -38,47 +38,49 @@ Analysis ### Current solution at MFF UK -The ideas presented above are not completely new. There was a group of students, -who already implemented an evaluation solution for student's homeworks in 2006. -Its name is [CodEx - The Code Examiner](http://codex.ms.mff.cuni.cz/project/) -and it has been used with some improvements since then. The original plan was to -use the system only for basic programming courses, but there is demand for -adapting it for many different subjects. - -CodEx is based on dynamic analysis. It features a web-based interface, where -supervisors assign exercises to their students and the students have a time -window to submit the solution. Each solution is compiled and run in sandbox -(MO-Eval). The metrics which are checked are: corectness of the output, time and -memory limits. It supports programs written in C, C++, C#, Java, Pascal, Python -and Haskell. +The ideas presented above are not completely new. There was a group of +students, who already implemented an evaluation solution for student's +homeworks in 2006. Its name is [CodEx - The Code +Examiner](http://codex.ms.mff.cuni.cz/project/) and it has been used with some +improvements since then. The original plan was to use the system only for basic +programming courses, but there is demand for adapting it for many different +subjects. + +CodEx is based on dynamic analysis. It features a web-based interface, where +supervisors assign exercises to their students and the students have a time +window to submit the solution. Each solution is compiled and run in sandbox +(MO-Eval). The metrics which are checked are: corectness of the output, time +and memory limits. It supports programs written in C, C++, C#, Java, Pascal, +Python and Haskell. Current system is old, but robust. There were no major security incidents during its production usage. However, from today's perspective there are several drawbacks. The main ones are: -- **web interface** -- The web interface is simple and fully functional. But - rapid development in web technologies opens new horizons of how web interface +- **web interface** -- The web interface is simple and fully functional. But + rapid development in web technologies opens new horizons of how web interface can be made. -- **web api** -- CodEx offers a very limited XML API based on outdated - technologies that is not sufficient for users who would like to create custom +- **web api** -- CodEx offers a very limited XML API based on outdated + technologies that is not sufficient for users who would like to create custom interfaces such as a command line tool or mobile application. -- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system - calls and blocking the bad ones. This can be easily done for single-threaded +- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system + calls and blocking the bad ones. This can be easily done for single-threaded applications, but proves difficult with multi-threaded ones. In present day, - parallelism is a very important area of computing, so there is requirement to + parallelism is a very important area of computing, so there is requirement to test multi-threaded applications too. -- **instances** -- Different ways of CodEx usage scenarios requires separate - instances (Programming I and II, Java, C#, etc.). This configuration is not - user friendly (students have to register in each instance separately) and - burdens administrators with unnecessary work. CodEx architecture does not - allow sharing hardware between instances, which results in an inefficient use +- **instances** -- Different ways of CodEx usage scenarios requires separate + instances (Programming I and II, Java, C#, etc.). This configuration is not + user friendly (students have to register in each instance separately) and + burdens administrators with unnecessary work. CodEx architecture does not + allow sharing hardware between instances, which results in an inefficient use of hardware for evaluation. -- **task extensibility** -- There is a need to test and evaluate complicated - programs for classes such as Parallel programming or Compiler principles, - which have a more difficult evaluation chain than simple +- **task extensibility** -- There is a need to test and evaluate complicated + programs for classes such as Parallel programming or Compiler principles, + which have a more difficult evaluation chain than simple compilation/execution/evaluation provided by CodEx. -Supervisors and current users of CodEx gives us following ideas for improvements: +Supervisors and current users of CodEx gives us following ideas for +improvements: - keep it simple - localization (both UI and exercises) @@ -421,8 +423,12 @@ deserves some. *A runtime environment* defines the used programming language or tools which are needed to process and evaluate a solution. Examples of a -runtime environment can be “*Linux + GCC*”, “*Linux + Mono*”, “*Windows -+ .NET 4*”, “*Bison + Yacc.”* +runtime environment can be: + +- *Linux + GCC* +- *Linux + Mono* +- *Windows + .NET 4* +- *Bison + Yacc* ### Limits @@ -562,14 +568,10 @@ of entities and relational database models), describe the logical grouping of entities and how they are related: - user + settings + logins + ACL - - instance + licences + groups + group membership - - exercise + assignments + localized assignments + runtime environments + hardware groups - - submission + solution + reference solution + solution evaluation - - comment threads + comments ### API endpoints @@ -607,15 +609,34 @@ can accept it or decline it Job processing progress monitoring ---------------------------------- -@todo: How does the Backend push the progress of evaluation through the -Monitor to the Frontend +When evaluating a job the worker sends progress messages on predefined points of +evaluation chain. The sending place can be on very beginning of the job, when +submit archive is downloaded or at the end of each simple task with its state +(completed, failed, skipped). These messages are sent to broker through existing +ZeroMQ connection. Detailed format of messages can be found on [communication +page](https://github.com/ReCodEx/wiki/wiki/Overall-architecture#commands-from-worker-to-broker). + +Broker only resends received progress messages to the monitor component via +ZeroMQ socket. The output message format is the same as the input format. + +Monitor parses received messages to JSON format, which is easy to work with in +JavaScript inside web application. All messages are cached (one queue per job) +and can be obtained multiple times through WebSocket communication channel. The +cache is cleared 5 minutes after receiving last message. Publishing of the results ------------------------- -@todo: How does the Backend publish the results and how it notifies the -Frontend; How the Frontend can request the results if the notification -does not arrive +After job finish the worker packs results directory into single archive and +uploads it to the fileserver through HTTP protocol. The target URL is obtained +from API in headers on job initiation. Then "job done" notification request is +performed to API via broker. Special submissions (reference or asynchronous +submissions) are loaded immediately, other types are loaded on-demand on first +results request. + +Loading results means fetching archive from fileserver, parsing the main YAML +file generated by worker and saving data to the database. Also, points are +assigned by score calculator. User documentation ================== @@ -635,6 +656,7 @@ Supervisor, Admin @todo: Describe the requirements of running the web application (modern web browser, enabled CSS, JavaScript, Cookies & Local storage) + ### Scenario \#1: Becoming a user of ReCodEx #### How to create a user account? @@ -678,8 +700,7 @@ in the top section of the left sidebar right under your name. You will have to expand the sidebar with a button next to the “*ReCodEx*” title (shown in the picture below). -![](media/image1.png){width="3.3131944444444446in" -height="0.6173611111111111in"} +@todo: Simon's image #### What to do when you cannot remember your password? @@ -705,15 +726,14 @@ website. There are several options you have to edit your user account. - changing your personal information (i.e., name) - - changing your credentials (email and password) - - updating your preferences (e.g., source code viewer/editor settings, default language) You can access the settings page through the “*Settings*” button right under your name in the left sidebar. + ### Scenario \#2: User is a student @todo: describe what it means to be a “student” and what are the @@ -749,6 +769,7 @@ although the results say different who else can see the comments, how notifications work (*not implemented yet*!). + ### Scenario \#3: User is supervisor of a group @todo: describe what it means to be a “supervisor” of a group and what @@ -804,7 +825,7 @@ the awarded points will be overridden. #### How can I discuss student’s solution with him/her directly through the web application? @todo: Describe the comments thread behavior (public/private comments), -who else can see the comments +who else can see the comments -- same as from the student perspective ### Writing job configuration @@ -824,8 +845,9 @@ configuration in the basic format will be always available and allows users to use the full expressive power of the system. This section walks through creation of job configuration for _hello world_ -exercise. The goal is to compile file _source.c_ and check if it prints `Hello -World!` to the standard output. +exercise. The goal is to compile file _source.c_ and check if it prints `Hello +World!` to the standard output. This is the only test case, let's call it +**A**. The problem can be split into several tasks: @@ -915,7 +937,7 @@ written. bin: "helloworld" sandbox: name: "isolate" - stdout: ${EVAL_DIR}/out.txt + stdout: ${EVAL_DIR}/out.txt limits: - hw-group-id: group1 chdir: ${EVAL_DIR}