diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 8202e6e..eb261e8 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -9,7 +9,7 @@ Notes: popisovat architektura * V analyze vysvetlit architekturu * Related works nechat jako samostatnou kapitolu -* Poradi - pozadavky -> related works -> analyza +* Poradi - pozadavky -> related works -> analyza * Provazani komponent musi rozumet administrator a tvurce ulohy - obecna kapitola v analyze - puvodni kapitola o analyze byla povedena, jen se tam micha seznam zprav nebo co - to nezajima vsechny @@ -73,7 +73,7 @@ comes from Stanford University professors in 1965. They implemented a system which evaluated code in Algol submitted on punch cards. In following years, many similar products were written. -In today's world, properties like correctness and efficiency can be tested +In the world of today, properties like correctness and efficiency can be tested automatically to a large extent. This fact should be exploited to help teachers save time for tasks such as examining bad design, bad coding habits and logical mistakes, which are difficult to perform automatically. @@ -118,7 +118,7 @@ currently used at the university (CodEx), so its flaws and weaknesses can be addressed. Furthermore, many teachers desire to use and test the new system and they are willing to consult ideas or problems during development with us. -## Current system +## Current System The grading solution currently used at the Faculty of Mathematics and Physics of the Charles University in Prague was implemented in 2006 by a group of students. @@ -175,7 +175,7 @@ Typical use cases for supported user roles are following: - alter user privileges -- make supervisor accounts - check system logs, upgrades and other management -### Exercise evaluation chain +### Exercise Evaluation Chain The most important part of the system is evaluation of solutions submitted by students. The process leading from source code to final results (score) is @@ -206,10 +206,10 @@ was correct. The system then calculates a numeric score from this data, which is presented to the student. If the solution is wrong (incorrect output, uses too much memory,..), error messages are also displayed to the submitter. -### Weaknesses +### Possible Improvements Current system is old, but robust. There were no major security incidents -during its production usage. However, from today's perspective there are +during its production usage. However, from the perspective of today there are several drawbacks. The main ones are: - **web interface** -- The web interface is simple and fully functional. @@ -222,7 +222,7 @@ several drawbacks. The main ones are: calls and blocking the bad ones. This can be easily done for single-threaded applications, but proves difficult with multi-threaded ones. In present day, parallelism is a very important area of computing, so there is requirement to - test multi-threaded applications too. + test multi-threaded applications as well. - **instances** -- Different ways of CodEx usage scenarios requires separate installations (Programming I and II, Java, C#, etc.). This configuration is not user friendly (students have to register in each instance separately) and @@ -249,13 +249,13 @@ In general, CodEx features should be preserved, so only differences are presented here. For clear arrangement all the requirements and wishes are presented grouped by categories. -### System features +### System Features System features represents directly accessible functionality to users of the system. They describe the evaluation system in general and also university addons (mostly administrative features). -#### Requirements of the users +#### Requirements of The Users - _group hierarchy_ -- creating an arbitrarily nested tree structure should be supported to allow keeping related groups together, such as in the example @@ -276,8 +276,7 @@ addons (mostly administrative features). (for example "sum all numbers from given file and write the result to the standard output") and to browse this database - _customizable grading system_ -- teachers need to specify the way of - computation of the final score, which will be awarded to the student's - submissions depending on their quality + computation of the final score, which will be awarded to the submissions of the student depending on their quality - _viewing student details_ -- teachers should be able to view the details of their students (members of their groups), including all submitted solutions - _awarding additional points_ -- adding (or subtracting) points from the final @@ -285,7 +284,7 @@ addons (mostly administrative features). - _marking a solution as accepted_ -- the system should allow marking one particular solution as accepted (used for grading the assignment) by the supervisor -- _solution resubmission_ -- teachers should be able edit student's solutions +- _solution resubmission_ -- teachers should be able edit the solutions of the student and privately resubmit them, optionally saving all results (including temporary ones); this feature can be used to quickly fix obvious errors in the solution and see if it is otherwise viable @@ -298,7 +297,7 @@ addons (mostly administrative features). solutions should be supported - _plagiarism detection_ -#### Administrative requirements +#### Administrative Requirements - _pluggable user interface_ -- the system should allow using an alternative user interface, such as a command line client; implementation of such clients @@ -311,8 +310,8 @@ addons (mostly administrative features). OAuth, should be supported - _querying SIS_ -- loading user data from the university information system should be supported -- _sandboxing_ -- there should be a safe environment in which the students' - solutions are executed to prevent system failures due to malicious code being +- _sandboxing_ -- there should be a safe environment in which the + solutions of the students are executed to prevent system failures due to malicious code being submitted; the sandboxed environment should have the least possible impact on measurement results (most importantly on measured times) - _heterogeneous worker pool_ -- there must be support for submission evaluation @@ -327,7 +326,7 @@ addons (mostly administrative features). another tool and perform additional tests - use of modern technologies with state-of-the-art compilers -### Non-functional requirements +### Non-functional Requirements Non-functional requirements are requirements of technical character with no direct mapping to visible parts of the system. In an ideal world, users should @@ -335,7 +334,7 @@ not know about these features if they work properly, but would be at least annoyed if they did not. - _no installation_ -- the primary user interface of the system must be - accessible on users' computers without the need to install any additional + accessible on the computers of the users without the need to install any additional software - _performance_ -- the system must be ready for at least hundreds of students and tens of supervisors using it at once @@ -452,7 +451,7 @@ restarted. At this point there is a clear idea how the new system will be used and what are the major enhancements for future releases. With this in mind, the overall architecture can be sketched. To sum up, here is a list of key features of the -new system. They come from previous research of current system's drawbacks, +new system. They come from previous research of the current drawbacks of the system, reasonable wishes of university users and our major design choices. - modern HTML5 web frontend written in JavaScript using a suitable framework @@ -469,7 +468,7 @@ chapter. Also a lot of smaller design choices are mentioned including possible options, what is picked to implement and why. But first, we discuss basic concepts of the system. -## Basic concepts +## Basic Concepts The system is designed as a web application. The requirements say that the user interface must be accessible for students without the need to install additional @@ -566,7 +565,7 @@ different for every assignment of the same exercise. This separation is natural for all users, in CodEx it is implemented in similar way and no other considerable solution was found. -### Evaluation unit executed by ReCodEx +### Evaluation Unit Executed by ReCodEx One of the bigger requests for the new system is to support a complex configuration of execution pipeline. The idea comes from lecturers of Compiler @@ -674,9 +673,9 @@ code. To mark the grouping, another task label can be used. Each test must have exactly one _evaluation_ task (to show success or failure to users) and arbitrary number of tasks with other types. -### Evaluation progress state +### Evaluation Progress State -Users want to know the state of their submitted solution (whether it's waiting +Users want to know the state of their submitted solution (whether it is waiting in a queue, compiling, etc.). The very first idea would be to report a state based on "done" messages from compilation, execution and evaluation like many evaluation systems are already providing. However ReCodEx has a more complicated @@ -694,14 +693,14 @@ of the evaluation. However, the benefits of this feature are not worth the effort of implementing it and unnecessarily complicating the job configuration files. -### Results of evaluation +### Results of Evaluation The evaluation data have to be processed and then presented in human readable form. This is done through a one numeric value called points. Also, results of job tests should be available to know what kind of error is in the solution. For more debugging, outputs of tasks could be optionally available for the users. -#### Scoring and assigning points +#### Scoring and Assigning Points The overall concept of grading solutions was presented earlier. To briefly remind that, backend returns only exact measured values (used time and memory, @@ -742,7 +741,7 @@ even student submitting the same code again which evaluates to more points, supervisor can mark a particular solution as marked and used for grading instead of solution with the most points. -#### Evaluation outputs +#### Evaluation Outputs In addition to the exact measured values used for score calculation described in previous chapter, there are also text or binary outputs of the executed tasks. @@ -797,7 +796,7 @@ their filesystem paths or a unique index stored as a value in a relational database. Both approaches are equally good, final decision depends on the actual implementation. -## Structure of the project +## Structure of The Project The ReCodEx project is divided into two logical parts -- the *backend* and the *frontend* -- which interact with each other and together cover the whole area @@ -869,7 +868,7 @@ companies, which can use the same system for their needs. It is also possible to develop a custom frontend with own user management system and use the possibilities of the backend without any changes. -### Possible connection +### Possible Connection One possible configuration of ReCodEx system is illustrated on following picture, where there is one shared backend with three workers and two separate @@ -886,12 +885,12 @@ will be introduced separately and covered in more detail. The communication protocol between these two logical parts will be described as well. -## Implementation analysis +## Implementation Analysis Some of the most important implementation problems or interesting observations will be discussed in this chapter. -### Communication between the backend components +### Communication Between the Backend Components Overall design of the project is discussed above. There are bunch of components with their own responsibility. Important thing to design is communication of @@ -944,7 +943,7 @@ notable chance we would have to do the same with RabbitMQ). It also gives us complete control over the transmitted messages and communication patterns. However, all of the three options would have been possible to use. -### File transfers +### File Transfers There has to be a way to access files stored on the fileserver (and also upload them )from both worker and frontend server machines. The protocol used for this @@ -988,7 +987,7 @@ We chose HTTPS because it is widely used and client libraries exist in all relevant environments. In addition, it is highly probable we will have to run an HTTP server, because it is intended for ReCodEx to have a web frontend. -### Frontend - backend communication +### Frontend - Backend Communication Our choices when considering how clients will communicate with the backend have to stem from the fact that ReCodEx should primarily be a web application. This @@ -1041,7 +1040,7 @@ HTTP(S). The broker is responsible for keeping track of available workers and distributing jobs that it receives from the frontend between them. -#### Worker management +#### Worker Management It is intended for the broker to be a fixed part of the backend infrastructure to which workers connect at will. Thanks to this design, workers can be added @@ -1091,7 +1090,7 @@ university). However, with a large amount of jobs, some workers will easily become overloaded. The implementation must allow for a simple replacement of the load balancing strategy so that this problem can be solved in the near future. -#### Forwarding jobs +#### Forwarding Jobs Information about a job can be divided in two disjoint parts -- what the worker needs to know to process it and what the broker needs to forward it to the @@ -1110,7 +1109,7 @@ approach leads to a more efficient network and resource utilization (the broker doesn't have to process data that it doesn't need), but also makes the job submission flow more complicated. -#### Further requirements +#### Further Requirements The broker can be viewed as a central point of the backend. While it has only two primary, closely related responsibilities, other requirements have arisen @@ -1155,7 +1154,7 @@ internal structures and executed. After that, results are uploaded back to fileserver. These steps are the basic ones which are really necessary for whole execution and have to be executed in this precise order. -#### Job configuration +#### Job Configuration Jobs as work units can quite vary and do completely different things, that means configuration and worker has to be prepared for this kind of generality. @@ -1185,7 +1184,7 @@ scenario on different worker. That is why if inner task fails it is sent back to broker which will reassign job to another worker. More on this subject should be discussed in broker assigning algorithms section. -#### Job working directories +#### Job Working Directories There is also question about working directory or directories of job, which directories should be used and what for. There is one simple answer on this @@ -1215,7 +1214,7 @@ searching through this system should be easy. In addition if solutions of users have access only to evaluation directory then they do not have access to unnecessary files which is better for overall security of whole ReCodEx. -#### Job variables +#### Job Variables As mentioned above worker has job directories but users who are writing and managing job configurations do not know where they are (on some particular @@ -1231,7 +1230,7 @@ triple dot is textual description. This format was used because of special dollar sign character which cannot be used within filesystem path, braces are there only to border textual description of variable. -#### Supplementary files +#### Supplementary Files Interesting problem is with supplementary files (inputs, sample outputs). There are two approaches which can be observed. Supplementary files can be downloaded @@ -1408,7 +1407,7 @@ can be used. But this solution is not automatic and worker would have to 'touch' cache files whenever they are accessed. This solution is maybe a bit better than the one with last access time and might be implemented in future releases. -#### Caching flow +#### Caching Flow Having cleaner as separated component and caching itself handled in worker is kind of blurry and is not clearly observable that it works without problems. @@ -1498,7 +1497,7 @@ options were already used languages C++, PHP, JavaScript and Python. At the end, the Python language was chosen for its simplicity, great support for all used technologies and also there are free Python developers in out team. -### API server +### API Server The API server must handle HTTP requests and manage the state of the application in some kind of a database. The API server will be a RESTful service and will @@ -1557,7 +1556,7 @@ Lumen, Laravel, and Symfony, we ended up using Nette. as components or extensions for Nette. There is for example a Nette extension which makes integration of Doctrine 2 very straightforward. -#### Architecture of the system +#### Architecture of The System The Nette framework is an MVP (Model, View, Presenter) framework. It has many tools for creating complex websites and we need only a subset of them or we use @@ -1597,7 +1596,7 @@ the signature: the payload can contain any data which can identify the user and metadata of the token (i.e., the time when the token was issued, the time of expiration). The last part is a digital signature contains a digital signature of the header and payload and it ensures that nobody can issue their own token -and steal someone's identity. Both of these characteristics give us the +and steal the identity of someone. Both of these characteristics give us the opportunity to validate the token without storing all of the tokens in the database. @@ -1610,7 +1609,7 @@ including generating the signature and signature verification is done through a widely used third-party library which lowers the risk of having a bug in the implementation of this critical security feature. -##### Backend state monitoring +##### Backend Monitoring The next thing related to communication with the backend is monitoring its current state. This concerns namely which workers are available for processing @@ -1645,7 +1644,7 @@ The web application ("WebApp") is one of the possible client applications of the ReCodEx system. Creating a web application as the first client application has several advantages: -- no installation or setup is required on the user's device +- no installation or setup is required on the device of the user - works on all platforms including mobile devices - when a new version is released, all the clients will use this version without any need for manual instalation of the update @@ -1659,9 +1658,9 @@ for implementation of a website. There are two basic ways how to create a website these days: -- **server-side approach** - user's actions are processed on the server and the +- **server-side approach** - the actions of the user are processed on the server and the HTML code with the results of the action is generated on the server and sent - back to the user's Internet browser. The client does not handle any logic + back to the web browser of the user. The client does not handle any logic (apart from rendering of the user interface and some basic user interaction) and is therefore very simple. The server can use the API server for processing of the actions so the business logic of the server can be very simple as well. @@ -1670,7 +1669,7 @@ There are two basic ways how to create a website these days: CSS files). This results in longer loading times of the website. - **server-side rendering with asynchronous updates (AJAX)** - a slightly different approach is to render the page on the server as in the previous case - but then execute user's actions asynchronously using the `XMLHttpRequest` + but then execute the actions of the user asynchronously using the `XMLHttpRequest` JavaScript functionality. Which creates a HTTP request and transfers only the part of the website which will be updated. - **client-side approach** - the opposite approach is to transfer the @@ -1694,12 +1693,13 @@ We decided to use the third approach -- to create a fully client-side application which would be familiar and intuitive for a user who is used to modern web applications. -#### Used technologies +#### Used Technologies We examined several frameworks which are commonly used to speed up the development of a web application. There are several open source options available with a large number of tools, tutorials, and libraries. From the many -options there are two main frameworks worth considering: +options (Backbone, Ember, Vue, Cycle.js, ...) there are two main frameworks worth +considering: - **Angular 2** - it is a new framework which was developed by Google. This framework is very complex and provides the developer with many tools which @@ -1731,7 +1731,7 @@ We decided to use React and Redux over Angular 2 for several reasons: any significant improvements and features over React so it would not be worth learning the paradigms of a new framework -#### User interface design +#### User Interface Design There is no artist on the team so we had to come up with an idea how to create a visually appealing application with this handicap. User interfaces created by @@ -1744,12 +1744,12 @@ the Bootstrap components for React and with the stylesheets from AdminLTE the application looks good and is distingushable form the many websites using the Bootstrap framework with very little work. -#### WebApp architecture +#### WebApp Architecture The whole project is written using the next generation of JavaScript referred to as *ECMAScript 6* (also known as *ES6*, *ES.next*, or *Harmony*). Since not all -of the features introduced in this standard are implemented in today's modern -web browsers (like classes and the spread operator) and hardly any are +of the features introduced in this standard are implemented in the modern +web browsers of today (like classes and the spread operator) and hardly any are implemented in the older versions of the web browsers which are currently still in use, the source code is transpiled into the older standard *ES5* using [Babel.js](https://babeljs.io/) transpiler and bundled into a single script file @@ -1760,13 +1760,13 @@ the [installation documentation](#Installation). The whole bundling process takes place at deployment and is not repeated afterwards when running in production. -##### State management +##### State Management @todo: Describe how Redux works @todo: Describe where which part of the state is persisted (logged in user) and how -# User documentation +# User Documentation Users interact with the ReCodEx through the web application. It is required to use a modern web browser with good HTML5 and CSS3 support. Among others, cookies @@ -1828,12 +1828,12 @@ privileged users. Described roles are: amount of resources. Hardware groups are managed byt the system administrators who have to keep them up-to-date. -## General basics +## General Basics Description of general basics which are the same for all users of ReCodEx web application follows. -### First steps in ReCodEx +### First Steps in ReCodEx You can create an account by clicking the "Create account" menu item in the left sidebar. You can choose between two types of registration methods -- by creating @@ -1881,7 +1881,7 @@ of the left sidebar right under your name. You may need to expand the sidebar with a button next to the "ReCodEx” title (informally known as _hamburger button_), depending on your screen size. -### Forgotten password +### Forgotten Password If you cannot remember your password and you do not use CAS UK authentication, then you can reset your password. You will find a link saying "Cannot remember @@ -1921,7 +1921,7 @@ sign. In the second column there is a list of assigned exercises with its deadlines. If you want to quickly get to the groups page you might want to use provided "Show group's detail" button. -### Join group and start solving assignments +### Join Group and Start Solving Assignments To be able to submit solutions you have to be a member of the right group. Each instance has its own group hierarchy, so you can choose only those within your @@ -1938,12 +1938,13 @@ clicking on "See group's page" link following with "Join group" link. in hierarchy and membership cannot be established by students themselves. Management of students in this type of groups is in the hands of supervisors. -On the group detail page there are multiple interesting things for you. First -one is brief overview with information describing the group, there is list with -supervisors and also hierarchy of subgroups. Most importantly, there is the -"Student's dashboard" section. This section contains list of assignments and -a list of fellow students. If supervisors of groups allowed students to see each -other's statistics there will also be the number of points the students gained. +On the group detail page there are multiple interesting things for you. The first +one is a brief overview containing the information describing the group, there is a list of +supervisors and also the hierarchy of the subgroups. The most important section +is the "Student's dashboard" section. This section contains the list of assignments and +the list of fellow students. If the supervisors of the group allowed students to see the +statistic of their fellow students then there will also be the number of +points each of the students has gained so far. In the "Assignments" box on the group detail page there is a list of assigned exercises which students are supposed to solve. The assignments are displayed @@ -2015,7 +2016,7 @@ another channel (email). Unfortunately there is currently no notification mechanism for new comment messages. -## Group supervisor +## Group Supervisor Group supervisor is typically the lecturer of a course. A user in this role can modify group description and properties, assign exercises or manage list of @@ -2024,12 +2025,12 @@ available only for group administrators. On "Dashboard" page you can find "Groups you supervise" section. Here there are boxes representing your groups with the list of students attending course and -their points. Student names are clickable with redirection to user's profile +their points. Student names are clickable with redirection to the profile of the user where further information about his/hers assignments and solution can be found. To quickly jump onto groups page, use "Show group's detail" button at the bottom of the matching group box. -### Manage group +### Manage Group Locate group you supervise and you want to manage. All your supervised groups are available in sidebar under "Groups -- supervisor" collapsible menu. If you @@ -2063,7 +2064,7 @@ on the magnifier image or pressing enter key there will appear list of matched users. At this moment just click on the "Join group" button and student will be signed in to your group. -### Assigning exercises +### Assigning Exercises Before assigning an exercise, you obviously have to know what exercises are available. A list of all exercises in the system can be found under "Exercises" @@ -2108,7 +2109,7 @@ In meta information form you can fill these options: - second submission deadline, after that students still can submit exercises but they are given no points no points (must be after the first deadline) - maximum points that can be gained between first deadline and second deadline -- submission count limit for students' solutions -- limits the amount of +- submission count limit for the solutions of the students -- this limits the amount of attempts a student has at solving the problem - visibility of memory and time ratios; if true students can see the percentage of used memory and time (with respect to the limit) for each test @@ -2141,19 +2142,19 @@ test case then you can take a peek on collapsible "Reference solutions' evaluations" items. If you are satisfied with changes you made to the limits, save the form with "Change limits" button right under environments box. -### Students' solutions management +### Management of The Solutions of The Students One of the most important tasks of a group supervisor is checking student solutions. As automatic evaluation of them cannot catch all problems in the -source code, it is advisable to do a brief manual review of student's coding -style and reflect that in assignment bonus points. +source code, it is advisable to do a brief manual review of the coding +style of the student and reflect that in assignment bonus points. On "Assignment detail" page there is an "View student results" button near top of the page (next to "Edit assignment settings" button). This will redirect you to a page where is a list of boxes, one box per student. Each student box contains a list of submissions for this assignment. The row structure of -submission list is the same as the structure in student's "Submitted solution" -box. More information about every solution can be showed by clicking on "Show +submission list is the same as the structure in the "Submitted solution" +box. More information about every solution can be shown by clicking on "Show details" link on the end of solution row. This page is the same as for students with one exception -- there is an @@ -2176,7 +2177,7 @@ to lack of frontend developers it is not yet prepared in user interface. We hope, it will be ready as soon as possible. The button for accepting a solution will be most probably also on this page. -### Creating exercises +### Creating Exercises Link to exercise creation can be found in exercises list which is accessible through "Exercises" link in sidebar. On the bottom of the exercises list page @@ -2253,7 +2254,7 @@ you can find "Delete the exercise" box where "Delete" button is located. By clicking on it exercise will be delete from the exercises list and will no longer be available. -### Exercise's reference solutions +### Reference Solutions of An Exercise Each exercise should have a set of reference solutions, which are used to tune time and memory limits of assignments. Values of used time and memory for each @@ -2273,14 +2274,14 @@ We are now working on a better user interface, which will be available soon. Then its description will be added here. -## Group administrator +## Group Administrator Group administrator is the group supervisor with some additional permissions in particular group. Namely group administrator is capable of creating a subgroups in managed group and also adding and deleting supervisors. Administrator of the particular group can be only one person. -### Creating subgroups and managing supervisors +### Creating Subgroups And Managing Supervisors There is no special link which will get you to groups in which you are administrator. So you have to get there through "Groups - supervisor" link in @@ -2315,13 +2316,13 @@ supervisor" buttons right next to supervisors names. After clicking on it some particular supervisor should not to be supervisor of the group anymore. -## Instance administrator +## Instance Administrator Instance administrator can be only one person per instance. In addition to previous roles this administrator should be able to modify the instance details, manage licences and take care of top level groups which belong to the instance. -### Instance management +### Instance Management List of all instances in the system can be found under "Instances" link in the sidebar. On that page there is a table of instances with their respective @@ -2370,7 +2371,7 @@ should be quite a unique role. Ideally, there should be only one user of this kind, used with special caution and adequate security. With this stated it is obvious that superadmin can perform any action the API is capable of. -### Users management +### Management of Users There are only a few user roles in ReCodEx. Basically there are only three: _student_, _supervisor_, and _superadmin_. Base role is student which is @@ -2403,7 +2404,7 @@ Roles description: the application. -## Writing score configuration +## Writing Score Configuration An important thing about assignment is how to assign points to particular solutions. As mentioned previously, the whole job is composed of logical tests. @@ -2423,7 +2424,7 @@ upcoming score calculators which might use some more complex scoring algorithms. This also means that future calculators do not have to use the YAML format for configuration. In fact, the configuration can be a string in any format. -### Simple score calculation +### Simple Score Calculation First implemented calculator is simple score calculator with test weights. This calculator just looks at the score of each test and put them together according @@ -2451,7 +2452,7 @@ testWeights: ``` -## Writing job configuration +## Writing Job Configuration To run and evaluate an exercise the backend needs to know the steps how to do that. This is different for each environment (operation system, programming @@ -2469,8 +2470,7 @@ use the full expressive power of the system. This section walks through creation of job configuration for _hello world_ exercise. The goal is to compile file _source.c_ and check if it prints `Hello -World!` to the standard output. This is the only test case, let's call it -**A**. +World!` to the standard output. This is the only test case named **A**. The problem can be split into several tasks: @@ -2627,7 +2627,7 @@ used. - API notification using curl, authentication using HTTP Basic Auth - asynchronous resending progress messages -### Additional libraries +### Additional Libraries Broker implementation depends on several open-source C and C++ libraries. @@ -2635,16 +2635,16 @@ Broker implementation depends on several open-source C and C++ libraries. HTTP protocol. Due to lack of documentation of all C++ bindings the plain C API is used. - **cppzmq** -- Cppzmq is a simple C++ wrapper for core ZeroMQ C API. It - basicaly contains only one header file, but its API fits into borker's object - architecture. + basicaly contains only one header file, but its API fits into the object + architecture of the broker. - **spdlog** -- Spdlog is small, fast and modern logging library used for system - logging. It is highly customizable and configurable from broker's - configuration. + logging. It is highly customizable and configurable from the + configuration of the broker. - **yaml-cpp** -- Yaml-cpp is used for parsing borker configuration text file in YAML format. - **boost-filesystem** -- Boost filesystem is used for managing logging directory (create if necessary) and parsing filesystem paths from strings as - written in broker's configuration. Filesystem operations will be included in + written in the configuration of the broker. Filesystem operations will be included in future releases of C++ standard, so this dependency may be removed in the future. - **boost-program_options** -- Boost program options is used for @@ -2662,7 +2662,7 @@ and HTTP access. To keep the stored data safe, the fileserver should not be visible from public internet. Instead, it should be accessed indirectly through the REST API. -### File deduplication +### File Deduplication From our analysis of the requirements, it is certain we need to implement a means of dealing with duplicate files. @@ -2680,12 +2680,12 @@ possible to change the hash function to something else, because the naming strategy is fully contained in the fileserver (special care must be taken to maintain backward compatibility). -### Storage structure +### Storage Structure Fileserver stores its data in following structure: - `./submissions//` -- folder that contains files submitted by users - (student's solutions to assignments). `` is an identifier received from + (the solutions to the assignments of the student). `` is an identifier received from the REST API. - `./submission_archives/.zip` -- ZIP archives of all submissions. These are created automatically when a submission is uploaded. `` is an identifier @@ -2700,7 +2700,7 @@ Fileserver stores its data in following structure: ## Worker -The worker's job is to securely execute a job according to its configuration and +The job of the worker is to securely execute a job according to its configuration and upload results back for latter processing. After receiving an evaluation request, worker has to do following: @@ -2713,7 +2713,7 @@ request, worker has to do following: - upload the results of the evaluation to the fileserver - notify broker that the evaluation finished -### Internal structure +### Internal Structure Worker is logicaly divided into two parts: @@ -2734,7 +2734,7 @@ there is no big overhead copying data between threads. This multi-threaded design allows the worker to keep sending `ping` messages even when it is processing a job. -### Capability identification +### Capability Identification There are possibly multiple worker instances in a ReCodEx installation and each one can run on different hardware, operating system, or have different tools @@ -2757,7 +2757,7 @@ For example, they can show which runtime environments are installed or whether this worker measures time precisely. Headers are also configured manually by an administrator. -### Running student submissions +### Running Student Submissions Student submissions are executed in a sandbox environment to prevent them from damaging the host system and also to restrict the amount of used resources. @@ -2785,7 +2785,7 @@ maximum time allowed for the task. This formula works well both for short and long tasks, but the timeout should never be reached if Isolate works properly -- it should always end itself in time. -### Directories and files +### Directories and Files During a job execution the worker has to handle several files -- input archive with submitted sources and job configuration, temporary files generated during @@ -2798,8 +2798,8 @@ directories (tasks can do whatever is allowed on the target system), but it is advised not to write outside them. In addition, sandboxed tasks are usually restricted to use only a specific (evaluation) directory. -The following directory structure is used for execution. The worker's working -directory (root of the following paths) is shared for multiple instances on the +The following directory structure is used for execution. The working +directory of the worker (root of the following paths) is shared for multiple instances on the same computer. - `downloads/${WORKER_ID}/${JOB_ID}` -- place to store the downloaded archive @@ -2826,7 +2826,7 @@ for comparison and exit code reflecting if the result is correct (0) of wrong (1). This interface lacks support for returning additional data by the judges, for -example similarity of the two files calculated as Levenshtein's edit distance. +example similarity of the two files calculated as the edit distance of Levenshtein. To allow passing these additional values an extended judge interface can be implemented: @@ -2853,7 +2853,7 @@ If more values are needed for score computation, multiple judges can be used in sequence and the values used together. However, extended judge interface should comply most of possible use cases. -### Additional libraries +### Additional Libraries Worker implementation depends on several open-source C and C++ libraries. All of them are multi-platform, so both Linux and Windows builds are possible. @@ -2865,14 +2865,12 @@ them are multi-platform, so both Linux and Windows builds are possible. Actual supported formats depends on installed packages on target system, but at least ZIP and TAR.GZ should be available. - **cppzmq** -- Cppzmq is a simple C++ wrapper for core ZeroMQ C API. It - basicaly contains only one header file, but its API fits into worker's object - architecture. + basicaly contains only one header file, but its API fits into the object architecture of the worker. - **spdlog** -- Spdlog is small, fast and modern logging library. It is used for all of the logging, both system and job logs. It is highly customizable and - configurable from worker's configuration. + configurable from the configuration of the worker. - **yaml-cpp** -- Yaml-cpp is used for parsing and creating text files in YAML - format. That includes worker's configuration, job's configuration and job's - results. + format. That includes the configuration of the worker, the configuration and the results of a job. - **boost-filesystem** -- Boost filesystem is used for multi-platform manipulation with files and directories. However, these operations will be included in future releases of C++ standard, so this dependency may be @@ -2900,7 +2898,7 @@ using HTTPS, it is required to use a proxy for monitor to provide encryption over WebSockets. If this is not done, browsers of the users will block unencrypted connection and will not show the progress to the users. -### Message flow +### Message Flow ![Message flow inside monitor](https://raw.githubusercontent.com/ReCodEx/wiki/master/images/Monitor_arch.png) @@ -2919,7 +2917,7 @@ Interpreter Lock](https://wiki.python.org/moin/GlobalInterpreterLock), so there is actualy no parallelism in the performance point of view, but proper synchronization is still required. -### Handling of incomming messages +### Handling of Incomming Messages Incomming ZeroMQ progress message is received and parsed to JSON format (same as our WebSocket communication format). JSON string is then passed to _thread 2_ @@ -2935,15 +2933,15 @@ command (normally FINISHED) is received, then are permanently deleted. This caching mechanism was implemented because early testing shows, that first couple of messages are missed quite often. -Messages from client's queue are sent through corresponding WebSocket connection +Messages from the queue of the client are sent through corresponding WebSocket connection via main event loop as soon as possible. This approach with separate queue per connection is easy to implement and guarantees reliability and order of message delivery. ## Cleaner -Cleaner component is tightly bound to the worker. It manages worker's cache -folder, mainly deletes outdated files. Every cleaner instance maintains one +Cleaner component is tightly bound to the worker. It manages the cache folder of the worker, +mainly deletes outdated files. Every cleaner instance maintains one cache folder, which can be used by multiple workers. This means on one server there can be numerous instances of workers with the same cache folder, but there should be only one cleaner instance. @@ -2957,7 +2955,7 @@ and then ends. This means that the cleaner has to be run repeatedly, for example using cron, systemd timer or Windows task scheduler. For proper function of the cleaner a suitable cronning interval has to be used. It is recommended to use 24 hour interval which is sufficient enough for intended usage. The value is set -in cleanr's configuration file. +in hte configuration file of the cleaner. ## REST API @@ -2970,7 +2968,7 @@ in cleanr's configuration file. - Automatic detection of the runtime environment - users must submit correctly named files, assuming the RTE from the extensions -### Used technologies +### Used Technologies @todo: PHP7 – how it is used for typehints, Nette framework – how it is used for routing, Presenters actions endpoints, exceptions and @@ -2994,7 +2992,7 @@ grouping of entities and how they are related: - submission + solution + reference solution + solution evaluation - comment threads + comments -### Request handling +### Request Handling A typical scenario for handling an API request is matching the HTTP request with a corresponding handler routine which creates a response object, that is then @@ -3041,7 +3039,7 @@ or two simple conditions. With this two concepts together it is possible to easily cover all cases of permission checking with quite a small amount of code. -### Uploading files +### Uploading Files There are two cases when users need to upload files using the API -- submitting solutions to an assignment and creating a new exercise. In both of these cases, @@ -3054,7 +3052,7 @@ Storing and removing files from the server is done through the `App\Helpers\UploadedFileStorage` class which maps the files to their records in the database using the `App\Model\Entity\UploadedFile` entity. -### Forgotten password +### Forgotten Password When user finds out that he/she does not remember a password, he/she requests a password reset and fills in his/her unique email. A temporary access token is @@ -3069,22 +3067,23 @@ and can be injected into any presenter. This solution is quite safe and user can handle it on its own, so administrator does not have to worry about it. -### Job configuration parsing and modifying +### Job Configuration Parsing and Modifying @todo how the YAML is parsed @todo how it can be changed and where it is used @todo how it can be stored to a new YAML -### Solution loading +### Solution Loading When a solution evaluation is finished by the backend, the results are saved to the fileserver and the API is notified by the broker. The results are parsed and stored in the database. -For results of reference solutions' evaluations and for asynchronous solution -evaluations (e.g., resubmitted by the administrator) the result is processed +For the results of the evaluations of the reference solutions and for the +asynchronously evaluated solutions of the students (e.g., resubmitted by +the administrator) the result is processed right after the notification from backend is received and the author of the -solution will be notified by an email. +solution will be notified by an email after the results are processed. When a student submits his/her solution directly through the client application we do not parse the results right away but we postpone this until the student @@ -3092,7 +3091,7 @@ we do not parse the results right away but we postpone this until the student save some resources when the solution results are not important (e.g., the student finds a bug in his solution before the submission has been evaluated). -#### Parsing of the results +#### Parsing of The Results The results are stored in a YAML file. We map the contents of the file to the classes of the `App\Helpers\EvaluationResults` namespace. This process @@ -3100,12 +3099,12 @@ validates the file and gives us access to all of the information through an interface of a class and not only using associative arrays. This is very similar to how the job configuration files are processed. -## API endpoints +## API Endpoints @todo: Tell the user about the generated API reference and how the Swagger UI can be used to access the API directly. -## Web application +## Web Application @todo: what to mention: - used libraries, JSX, ... @@ -3113,7 +3112,7 @@ Swagger UI can be used to access the API directly. - server side rendering - maybe more ... -## Communication protocol +## Communication Protocol Detailed communication inside the ReCodEx system is captured in the following image and described in sections below. Red connections are through ZeroMQ @@ -3124,15 +3123,15 @@ no empty frames (unles explicitly specified otherwise). ![Communication schema](https://github.com/ReCodEx/wiki/raw/master/images/Backend_Connections.png) -### Broker - Worker communication +### Broker - Worker Communication Broker acts as server when communicating with worker. Listening IP address and port are configurable, protocol family is TCP. Worker socket is of DEALER type, broker one is ROUTER type. Because of that, very first part of every (multipart) -message from broker to worker must be target worker's socket identity (which is +message from broker to worker must be target the socket identity of the worker (which is saved on its **init** command). -#### Commands from broker to worker: +#### Commands from Broker to Worker: - **eval** -- evaluate a job. Requires 3 message frames: - `job_id` -- identifier of the job (in ASCII representation -- we avoid @@ -3146,7 +3145,7 @@ saved on its **init** command). shut down and restarted without the other side noticing. - **pong** -- reply to **ping** command, no arguments -#### Commands from worker to broker: +#### Commands from Worker to Broker: - **init** -- introduce self to the broker. Useful on startup or after reestablishing lost connection. Requires at least 2 arguments: @@ -3224,7 +3223,7 @@ is reliable and can outlive short term issues with connection without problems. Also, increasing delay of ping messages does not flood the network when there are problems. We experienced no issues since we are using this protocol. -### Worker - File Server communication +### Worker - File Server Communication Worker is communicating with file server only from _execution thread_. Supported protocol is HTTP optionally with SSL encryption (**recommended**). If supported @@ -3232,10 +3231,10 @@ by server and used version of libcurl, HTTP/2 standard is also available. File server should be set up to require basic HTTP authentication and worker is capable to send corresponding credentials with each request. -#### Worker side +#### Worker Side Workers comunicate with the file server in both directions -- they download -student's submissions and then upload evaluation results. Internally, worker is +the submissions of the student and then upload evaluation results. Internally, worker is using libcurl C library with very similar setup. In both cases it can verify HTTPS certificate (on Linux against system cert list, on Windows against downloaded one from CURL website during installation), support basic HTTP @@ -3265,18 +3264,18 @@ with proper configuration. Relevant commands for communication with workers: successful upload returns JSON `{ "result": "OK" }` as body of returned page. If not specified otherwise, `zip` format of archives is used. Symbol `/` in API -description is root of file server's domain. If the domain is for example +description is root of the domain of the file server. If the domain is for example `fs.recodex.org` with SSL support, getting input file for one task could look as GET request to `https://fs.recodex.org/tasks/8b31e12787bdae1b5766ebb8534b0adc10a1c34c`. -### Broker - Monitor communication +### Broker - Monitor Communication Broker communicates with monitor also through ZeroMQ over TCP protocol. Type of socket is same on both sides, ROUTER. Monitor is set to act as server in this -communication, its IP address and port are configurable in monitor's config -file. ZeroMQ socket ID (set on monitor's side) is "recodex-monitor" and must be +communication, its IP address and port are configurable in the config of the monitor +file. ZeroMQ socket ID (set on the side of the monitor) is "recodex-monitor" and must be sent as first frame of every multipart message -- see ZeroMQ ROUTER socket documentation for more info. @@ -3287,26 +3286,26 @@ communication so that the workers do not have to know too many network services. Monitor is treated as a somewhat optional part of whole solution, so no special effort on communication realibility was made. -#### Commands from monitor to broker: +#### Commands from Monitor to Broker: Because there is no need for the monitor to communicate with the broker, there are no commands so far. Any message from monitor to broker is logged and discarded. -#### Commands from broker to monitor: +#### Commands from Broker to Monitor: - **progress** -- notification about progress with job evaluation. This communication is only redirected as is from worker, more info can be found in "Broker - Worker Communication" chapter above. -### Broker - Web API communication +### Broker - Web API Communication Broker communicates with main REST API through ZeroMQ connection over TCP. Socket type on broker side is ROUTER, on frontend part it is DEALER. Broker acts as a server, its IP address and port is configurable in the API. -#### Commands from API to broker: +#### Commands from API to Broker: - **eval** -- evaluate a job. Requires at least 4 frames: - `job_id` -- identifier of this job (in ASCII representation -- we avoid @@ -3322,7 +3321,9 @@ as a server, its IP address and port is configurable in the API. source code - `result_url` -- remote URI where results will be pushed to -#### Commands from broker to API (all are responses to **eval** command): +#### Commands from Broker to API: + +All are responses to **eval** command. - **ack** -- this is first message which is sent back to frontend right after eval command arrives, basically it means "Hi, I am all right and am capable of @@ -3334,13 +3335,13 @@ as a server, its IP address and port is configurable in the API. broker finds that it cannot handle the job after it was confirmed. In such cases it uses the frontend REST API to mark the job as failed. -#### Asynchronous communication between broker and API +#### Asynchronous Communication Between Broker And API Only a fraction of the errors that can happen during evaluation can be detected while there is a ZeroMQ connection between the API and broker. To notify the frontend of the rest, the API exposes an endpoint for the broker for this purpose. Broker uses this endpoint whenever the status of a job changes -(it's finished, it failed permanently, the only worker capable of processing +(it is finished, it failed permanently, the only worker capable of processing it disconnected...). When a request for sending a report arrives from the backend then the type of @@ -3353,7 +3354,7 @@ they are stored in the persistent database for further consideration. For the details of this interface please refer to the attached API documentation and the `broker-reports/` endpoint group. -### File Server - Web API communication +### File Server - Web API Communication File server has a REST API for interaction with other parts of ReCodEx. Description of communication with workers is in "Worker - File Server @@ -3380,7 +3381,7 @@ in time. Web API calls these fileserver endpoints with standard HTTP requests. There are no special commands involved. There is no communication in opposite direction. -### Monitor - Web app communication +### Monitor - Web App Communication Monitor interacts with web application through WebSocket connection. Monitor acts as server and browsers are connecting to it. IP address and port are @@ -3413,14 +3414,14 @@ Message format: - **task_state** -- state of task with id **task_id**. Present only if **command** is "TASK". Value is one of "COMPLETED", "FAILED" and "SKIPPED". -### Web app - Web API communication +### Web App - Web API Communication -Provided web application runs as javascript client inside user's browser. It -communicates with REST API on the server through standard HTTP requests. -Documentation of the main REST API is in separate -[document](https://recodex.github.io/api/) due to its extensiveness. Results are -returned as JSON payload, which is simply parsed in web application and -presented to the users. +The provided web application runs as a JavaScript process inside the browser of the user. +It communicates with the REST API on the server through the standard HTTP requests. +Documentation of the main REST API is in a separate +[document](https://recodex.github.io/api/) due to its extensiveness. The results are +returned encoded in JSON which is simply processed by the web application and +presented to the user in an appropriate way.