From 88953c7ffb4b8be34cc9b8f28e3efc84f14a2f66 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Thu, 12 Jan 2017 13:34:17 +0100 Subject: [PATCH] User doc is now right after analysis --- Rewritten-docs.md | 1149 +++++++++++++++++++++++---------------------- 1 file changed, 576 insertions(+), 573 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index ef77ff6..b9b1aa1 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1637,783 +1637,786 @@ be implemented in next releases. -# The Backend - -The backend is the part which is hidden to the user and which has only -one purpose: evaluate user’s solutions of their assignments. +# User documentation -@todo: describe the configuration inputs of the Backend +@todo: Describe different scenarios of the usage of the Web App -@todo: describe the outputs of the Backend +@todo: Describe the requirements of running the web application (modern web browser, enabled CSS, JavaScript, Cookies & Local storage) -@todo: describe how the backend receives the inputs and how it -communicates the results +## Terminology -## Components +@todo: Describe the terminology: Instance, User, Group, Student, +Supervisor, Admin -Whole backend is not just one service/component, it is quite complex system on its own. +## General basics -@todo: describe the inner parts of the Backend (and refer to the Wiki -for the technical description of the components) +@todo: actions which are available for all users -### Broker +@todo: how to solve problems with ReCodEx, first supervisors, then administrators, etc... -@todo: gets stuff done, single point of failure and center point of ReCodEx universe +### First steps in ReCodEx -### Fileserver +You can create an account if you click on the “*Create account*” menu +item in the left sidebar. You can choose between two types of +registration methods – by creating a local account with a specific +password, or pairing your new account with an existing CAS UK account. -@todo: stores particular data from frontend and backend, hashing, HTTP API +If you decide a new “*local*” account using the “*Create ReCodEx +account*” form, you will have to provide your details and choose a +password for your account. You will later sign in using your email +address as your username and the password you select. -### Worker +If you decide to use the CAS UK, then we will verify your credentials +and access your name and email stored in the system and create your +account based on this information. You can change your personal +information or email later on the “*Settings*” page. -@todo: describe a bit of internal structure in general +When creating your account both ways, you must select an instance your +account will belong to by default. The instance you will select will be +most likely your university or other organization you are a member of. -@todo: describe how jobs are generally executed +To log in, go to the homepage of ReCodEx and in the left sidebar choose +the menu item “*Sign in*”. Then you must enter your credentials into one +of the two forms – if you selected a password during registration, then +you should sign with your email and password in the first form called +“*Sign into ReCodEx*”. If you registered using the Charles University +Authentication Service (CAS), you should put your student’s number and +your CAS password into the second form called “Sign into ReCodEx using +CAS UK”. -### Monitor +There are several options you can edit in your user account: -@todo: not necessary component which can be omitted, proxy-like service +- changing your personal information (i.e., name) +- changing your credentials (email and password) +- updating your preferences (e.g., source code viewer/editor settings, + default language) -## Backend internal communication +You can access the settings page through the “*Settings*” button right +under your name in the left sidebar. -@todo: internal backend communication, what communicates with what and why +If you don’t use ReCodEx for a whole day, you will be logged out +automatically. However, we recommend you sign out of the application +after you finished your interaction with it. The logout button is placed +in the top section of the left sidebar right under your name. You will +have to expand the sidebar with a button next to the “*ReCodEx*” title +(shown in the picture below). -The Frontend -============ +### Forgotten password -The frontend is the part which is visible to the user of ReCodEx and -which holds the state of the system – the user accounts, their roles in -the system, the database of exercises, the assignments of these -exercises to groups of users (i.e., students), and the solutions and -evaluations of them. +If you can’t remember your password and you don’t use CAS UK +authentication, then you can reset your password. You will find a link +saying “*You cannot remember what your password was? Reset your +password.*” under the sign in form. After you click on this link, you +will be asked to submit your email address. An email with a link +containing a special token will be sent to the address you fill in. We +make sure that the person who requested password resetting is really +you. When you click on the link (or you copy & paste it into your web +browser) you will be able to select a new password for your account. The +token is valid only for a couple of minutes, so do not forget to reset +the password as soon as possible, or you will have to request a new link +with a valid token. -Frontend is split into three parts: +If you sign in through CAS UK, then please follow the instructions +provided by the administrators of the service described on their +website. -- the server-side REST API (“API”) which holds the business logic and - keeps the state of the system consistent -- the relational database (“DB”) which persists the state of the - system +## Student -- the client side application (“client”) which simplifies access to - the API for the common users +@todo: describe what it means to be a “student” and what are the +student’s rights -The centerpiece of this architecture is the API. This component receives -requests from the users and from the Backend, validates them and -modifies the state of the system and persists this modified state in the -DB. +### Join group and start solving assignments -We have created a web application which can communicate with the API -server and present the information received from the server to the user -in a convenient way. The client can be though any application, which can -send HTTP requests and receive the HTTP responses. Users can use general -applications like [cURL](https://github.com/curl/curl/), -[Postman](https://www.getpostman.com/), or create their own specific -client for ReCodEx API. +@todo: How to join a specific group -Frontend capabilities ---------------------- +@todo: Where can the user see groups description and details, what +information is available. -@todo: describe what the frontend is capable of and how it really works, -what are the limitations and how it can be extended +@todo: Where the student can find the list of the assignment he is +expected to solve, what is the first and second deadline. -Terminology ------------ +@todo: How does a student submit his solution through the web app -This project was created for the needs of a university and this fact is -reflected into the terminology used throughout the Frontend. A list of -important terms’ definitions follows to make the meaning unambiguous. +@todo: When the results are ready and what the results mean and what to +do about them, when the user is convinced, that his solution is correct +although the results say different -### User and user roles +@todo: Describe the comments thread behavior (public/private comments), +who else can see the comments, how notifications work (*not implemented +yet*!). -*User* is a person who uses the application. User is granted access to -the application once he or she creates an account directly through the -API or the web application. There are several types of user accounts -depending on the set of permissions – a so called “role” – they have -been granted. Each user receives only the most basic set of permissions -after he or she creates an account and this role can be changed only by -the administrators of the service: -- *Student* is the most basic role. Student can become member of a - group and submit his solutions to his assignments. +## Group supervisor -- *Supervisor* can be entitled to manage a group of students. - Supervisor can assign exercises to the students who are members of - his groups and review their solutions submitted to - these assignments. +@todo: describe what it means to be a “supervisor” of a group and what +are the supervisors rights -- *Super-admin* is a user with unlimited rights. This user can perform - any action in the system. +### Create groups and manage them -There are two implicit changes of roles: +@todo: How does a user become a supervisor of a group? -- Once a *student* is added to a group as its supervisor, his role is - upgraded to a *supervisor* role. +@todo: How to add a specific student to a given group -- Once a *supervisor* is removed from the lasts group where he is a - supervisor then his role is downgraded to a *student* role. +### Assigning exercises -These mechanisms do not prevent a single user being a supervisor of one -group and student of a different group as supervisors’ permissions are -superset of students’ permissions. +@todo: Describe how to access the database of the exercises and what are +the possibilities of assignment setup – availability, deadlines, points, +score configuration, limits -### Login +@todo: How can I assign some exercises only to some students of the group? Describe how to achieve this using subgroups -*Login* is a set of user’s credentials he must submit to verify he can -be allowed to access the system as a specific user. We distinguish two -types of logins: local and external. +### Students' solutions management -- *Local login* is user’s email address and a password he chooses - during registration. +@todo Describe where all the students’ solutions for a given assignment +can be found, where to look for all solutions of a given student, how to +see results of a specific student’s solution’s evaluation result. -- *External login* is a mapping of a user profile to an account of - some authentication service (e.g., [CAS](https://ldap1.cuni.cz/)). +@todo Can I assign points to my students’ solutions manually instead of depending on automatic scoring? If and how to change the score of a solution – assignment +settings, setting points, bonus points, accepting a solution (*not +implemented yet!*). Describe how the student and supervisor will still +be able to see the percentage received from the automatic scoring, but +the awarded points will be overridden. -### Instance +@todo: Describe the comments thread behavior (public/private comments), +who else can see the comments -- same as from the student perspective -*An instance* of ReCodEx is in fact just a set of groups and user -accounts. An instance should correspond to a real entity as a -university, a high-school, an IT company or an HR agency. This approach -enables the system to be shared by multiple independent organizations -without interfering with each other. +### Creating exercises -Usage of the system by the users of an instance can be limited by -possessing a valid license. It is up to the administrators of the system -to determine the conditions under which they will assign licenses to the -instances. +@todo: how to create exercise, what has to be provided during creation, who can create exercises -### Group +@todo: Describe the form and explain the concept of reference solutions. +How to evaluate the reference solutions for the exercise right now (to +get the up-to-date information). -*Group* corresponds to a school class or some other unit which gathers -users who will be assigned the same set exercises. Each group can have -multiple supervisors who can manage the students and the list of -assignments. -Groups can form a tree hierarchy of arbitrary depth. This is inspired by the -hierarchy of school classes belonging to the same subject over several school -years. For example, there can be a top level group for a programming class that -contains subgroups for every school year. These groups can then by divided into -actual student groups with respect to lab attendance. Supervisors can create -subgroups of their groups and further manage these subgroups. +## Group administrator -### Exercise +@todo: who is this? -*An exercise* consists of textual assignment of a task and a definition -of how a solution to this exercise should be processed and evaluated in -a specific runtime environment (i.e., how to compile a submitted source -code and how to test the correctness of the program). It is a template -which can be instantiated as an *assignment* by a supervisor of a group. +### Creating subgroups and managing supervisors -### Assignment +@todo: What it means to create a subgroup and how to do it. -An assignment is an instance of an *exercise* assigned to a specific -*group*. An assignment can modify the text of the task assignment and it -has some additional information which is specific to the group (e.g., a -deadline, the number of points gained for a correct solution, additional -hints for the students in the assignment). The text of the assignment -can be edited and supervisors can translate the assignment into another -language. +@todo: who can add another supervisor, what would be the rights of the +second supervisor -### Solution -*A solution* is a set of files which a user submits to a given -*assignment*. +## Superadministrator -### Submission +Superadmin is user with the most priviledges and as such superadmin should be +quite unique role. Ideally there should be only one of this kind, used with +special caution and adequate security. With this stated it is obvious that +superadmin can perform any action the API is capable of. -*A submission* corresponds to a *solution* being evaluated by the -Backend. A single *solution* can be submitted repeatedly (e.g., when the -Backend encounters an error or when the supervisor changes the assignment). +### Users management -### Evaluation +There are only few roles to which users can belong in ReCodEx. Basically there +are only three: _student_, _supervisor_, and _superadmin_. Base role is student +which is assigned to every registered user. Roles are stored in database +alongside other information about user. One user always has only one role at the +time. At first startup of ReCodEx administrator should create his account and +then change role in database by hand. After that manual intervention into +database should never be needed. -*An evaluation* is the processed report received from the Backend after -a *submission* is processed. Evaluation contains points given to the -user based on the quality of his solution measured by the Backend and -the settings of the assignment. Supervisors can review the evaluation -and add bonus points (both positive and negative) if the student -deserves some. +There is a little catch in groups and instances management. Groups can have +admins and supervisors. This setting is valid only per one particular group and +has to be separated from basic role system. This implies that supervisor in one +group can be student in another and simultaneously have global supervisor role. +Changing role from student to supervisor and back is done automatically by +application and should not be managed by hand in database! Previously stated +information can be applied to instances as well, but instances can only have +admins. -### Runtime environment +Roles description: -*A runtime environment* defines the used programming language or tools -which are needed to process and evaluate a solution. Examples of a -runtime environment can be: +- Student -- Default role which is used for newly created accounts. Student can + join or leave public groups and submit solutions of assigned exercises. +- Supervisor -- Inherits all permissions from student role. Can manage groups to + which he/she belongs to. Supervisor can also view and change groups details, + manage assigned exercises, view students in group and their solutions for + assigned exercises. On top of that supervisor can create/delete groups too, + but only as subgroup of groups he/she belongs to. +- Superadmin -- Inherits all permissions from supervisor role. Most powerful + user in ReCodEx who should be able to do everything which is provided by + application. -- *Linux + GCC* -- *Linux + Mono* -- *Windows + .NET 4* -- *Bison + Yacc* -### Limits +## Writing job configuration -A correct *solution* of an *assignment* has to pass all specified tests (mostly -checks that it yields the correct output for various inputs) and typically must -also be effective in some sense. The Backend measures the time and memory -consumption of the solution while running. This consumption of resources can be -*limited* and the solution will receive fewer points if it exceeds the given -limits in some test cases defined by the *exercise*. +To run and evaluate an exercise the backend needs to know the steps how to do +that. This is different for each environment (operation system, programming +language, etc.), so each of the environments needs to have separate +configuration. -User management ---------------- +Backend works with a powerful, but quite low level description of simple +connected tasks written in YAML syntax. More about the syntax and general task +overview can be found on [separate +page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned +features was user friendly configuration editor, but due to tight deadline and +team composition it did not make it to the first release. However, writing +configuration in the basic format will be always available and allows users to +use the full expressive power of the system. -@todo: roles and their rights, adding/removing different users, how the -role of a specific user changes +This section walks through creation of job configuration for _hello world_ +exercise. The goal is to compile file _source.c_ and check if it prints `Hello +World!` to the standard output. This is the only test case, let's call it +**A**. -Instances and hierarchy of groups ---------------------------------- +The problem can be split into several tasks: -@todo: What is an instance, how to create one, what are the licenses and -how do they work. Why can the groups form hierarchies and what are the -benefits – what it means to be an admin of a group, hierarchy of roles -in the group hierarchy. +- compile _source.c_ into _helloworld_ with `/usr/bin/gcc` +- run _helloworld_ and save standard output into _out.txt_ +- fetch predefined output (suppose it is already uploaded to fileserver) with + hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_ +- compare _out.txt_ and _reference.txt_ by `/usr/bin/diff` -Exercises database ------------------- +The absolute path of tools can be obtained from system administrator. However, +`/usr/bin/gcc` is location, where the GCC binary is available almost everywhere, +so location of some tools can be (professionally) guessed. -@todo: How the exercises are stored, accessed, who can edit what +First, write header of the job to the configuration file. -### Creating a new exercise +```{.yml} +submission: + job-id: hello-word-job + hw-groups: + - group1 +``` -@todo Localized assignments, default settings +Basically it means, that the job _hello-world-job_ needs to be run on workers +that belong to the `group_1` hardware group . Reference files are downloaded +from the default location configured in API (such as +`http://localhost:9999/exercises`) if not stated explicitly otherwise. Job +execution log will not be saved to result archive. -### Runtime environments and hardware groups +Next the tasks have to be constructed under _tasks_ section. In this demo job, +every task depends only on previous one. The first task has input file +_source.c_ (if submitted by user) already available in working directory, so +just call the GCC. Compilation is run in sandbox as any other external program +and should have relaxed time and memory limits. In this scenario, worker +defaults are used. If compilation fails, the whole job is immediately terminated +(because the _fatal-failure_ bit is set). Because _bound-directories_ option in +sandbox limits section is mostly shared between all tasks, it can be set in +worker configuration instead of job configuration (suppose this for following +tasks). For configuration of workers please contact your administrator. -@todo read this later and see if it still makes sense +```{.yml} +- task-id: "compilation" + type: "initiation" + fatal-failure: true + cmd: + bin: "/usr/bin/gcc" + args: + - "source.c" + - "-o" + - "helloworld" + sandbox: + name: "isolate" + limits: + - hw-group-id: group1 + chdir: ${EVAL_DIR} + bound-directories: + - src: ${SOURCE_DIR} + dst: ${EVAL_DIR} + mode: RW +``` -ReCodEx is designed to utilize a rather diverse set of workers -- there can be -differences in many aspects, such as the actual hardware running the worker -(which impacts the results of measuring) or installed compilers, interpreters -and other tools needed for evaluation. To address these two examples in -particular, we assign runtime environments and hardware groups to exercises. +The compiled program is executed with time and memory limit set and the standard +output is redirected to a file. This task depends on _compilation_ task, because +the program cannot be executed without being compiled first. It is important to +mark this task with _execution_ type, so exceeded limits will be reported in +frontend. -The purpose of runtime environments is to specify which tools (and often also -operating system) are required to evaluate a solution of the exercise -- for -example, a C# programming exercise can be evaluated on a Linux worker running -Mono or a Windows worker with the .NET runtime. Such exercise would be assigned -two runtime environments, `Linux+Mono` and `Windows+.NET` (the environment names -are arbitrary strings configured by the administrator). +Time and memory limits set directly for a task have higher priority than worker +defaults. One important constraint is, that these limits cannot exceed limits +set by workers. Worker defaults are present as a safety measure so that a +malformed job configuration cannot block the worker forever. Worker default +limits should be reasonably high, like a gigabyte of memory and several hours of +execution time. For exact numbers please contact your administrator. -A hardware group is a set of workers that run on similar hardware (e.g. a -particular quad-core processor model and a SSD hard drive). Workers are assigned -to these groups by the administrator. If this is done correctly, performance -measurements of a submission should yield the same results. Thanks to this fact, -we can use the same resource limits on every worker in a hardware group. -However, limits can differ between runtime environments -- formally speaking, -limits are a function of three arguments: an assignment, a hardware group and a -runtime environment. +It is important to know that if the output of a program (both standard and +error) is redirected to a file, the sandbox disk quotas apply to that file, as +well as the files created directly by the program. In case the outputs are +ignored, they are redirected to `/dev/null`, which means there is no limit on +the output length (as long as the printing fits in the time limit). -### Reference solutions +```{.yml} +- task-id: "execution_1" + test-id: "A" + type: "execution" + dependencies: + - compilation + cmd: + bin: "helloworld" + sandbox: + name: "isolate" + stdout: ${EVAL_DIR}/out.txt + limits: + - hw-group-id: group1 + chdir: ${EVAL_DIR} + time: 0.5 + memory: 8192 +``` -@todo: how to add one, how to evaluate it +Fetch sample solution from file server. Base URL of file server is in the header +of the job configuration, so only the name of required file (its `sha1sum` in +our case) is necessary. -The task of determining appropriate resource limits for exercises is difficult -to do correctly. To aid exercise authors and group supervisors, ReCodEx supports -assigning reference solutions to exercises. Those are example programs that -should cover the main approaches to the implementation. For example, searching -for an integer in an ordered array can be done with a linear search, or better, -using a binary search. +```{.yml} +- task-id: "fetch_solution_1" + test-id: "A" + dependencies: + - execution + cmd: + bin: "fetch" + args: + - "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b" + - "${SOURCE_DIR}/reference.txt" +``` -Reference solutions can be evaluated on demand, using a selected hardware group. -The evaluation results are stored and can be used later to determine limits. In -our example problem, we could configure the limits so that the linear -search-based program doesn't finish in time on larger inputs, but a binary -search does. +Comparison of results is quite straightforward. It is important to set the task +type to _evaluation_, so that the return code is set to 0 if the program is +correct and 1 otherwise. We do not set our own limits, so the default limits are +used. -Note that separate reference solutions should be supplied for all supported -runtime environments. +```{.yml} +- task-id: "judge_1" + test-id: "A" + type: "evaluation" + dependencies: + - fetch_solution_1 + cmd: + bin: "/usr/bin/diff" + args: + - "out.txt" + - "reference.txt" + sandbox: + name: "isolate" + limits: + - hw-group-id: group1 + chdir: ${EVAL_DIR} +``` -### Exercise assignments -@todo: Creating instances of an exercise for a specific group of users, -capabilities of settings. Editing limits according to the reference -solution. -Evaluation process ------------------- +# The Backend -@todo: How the evaluation process works on the Frontend side. +The backend is the part which is hidden to the user and which has only +one purpose: evaluate user’s solutions of their assignments. -### Uploading files and file storage +@todo: describe the configuration inputs of the Backend -@todo: One by one upload endpoint. Explain different types of the -Uploaded files. +@todo: describe the outputs of the Backend -### Automatic detection of the runtime environment +@todo: describe how the backend receives the inputs and how it +communicates the results -@todo: Users must submit correctly named files – assuming the RTE from -the extensions. +## Components -REST API implementation ------------------------ +Whole backend is not just one service/component, it is quite complex system on its own. -@todo: What is the REST API, what are the basic principles – GET, POST, -Headers, JSON. +@todo: describe the inner parts of the Backend (and refer to the Wiki +for the technical description of the components) -### Authentication and authorization scopes +### Broker -@todo: How authentication works – signed JWT, headers, expiration, -refreshing. Token scopes usage. +@todo: gets stuff done, single point of failure and center point of ReCodEx universe -### HTTP requests handling +### Fileserver -@todo: Router and routes with specific HTTP methods, preflight, required -headers +@todo: stores particular data from frontend and backend, hashing, HTTP API -### HTTP responses format +### Worker -@todo: Describe the JSON structure convention of success and error -responses +@todo: describe a bit of internal structure in general -### Used technologies +@todo: describe how jobs are generally executed -@todo: PHP7 – how it is used for typehints, Nette framework – how it is -used for routing, Presenters actions endpoints, exceptions and -ErrorPresenter, Doctrine 2 – database abstraction, entities and -repositories + conventions, Communication over ZMQ – describe the -problem with the extension and how we reported it and how to treat it in -the future when the bug is solved. Relational database – we use MariaDB, -Doctine enables us to switch the engine to a different engine if needed +### Monitor -### Data model +@todo: not necessary component which can be omitted, proxy-like service -@todo: Describe the code-first approach using the Doctrine entities, how -the entities map onto the database schema (refer to the attached schemas -of entities and relational database models), describe the logical -grouping of entities and how they are related: +## Backend internal communication -- user + settings + logins + ACL -- instance + licenses + groups + group membership -- exercise + assignments + localized assignments + runtime - environments + hardware groups -- submission + solution + reference solution + solution evaluation -- comment threads + comments +@todo: internal backend communication, what communicates with what and why -### API endpoints +The Frontend +============ -@todo: Tell the user about the generated API reference and how the -Swagger UI can be used to access the API directly. +The frontend is the part which is visible to the user of ReCodEx and +which holds the state of the system – the user accounts, their roles in +the system, the database of exercises, the assignments of these +exercises to groups of users (i.e., students), and the solutions and +evaluations of them. -Web Application ---------------- +Frontend is split into three parts: -@todo: What is the purpose of the web application and how it interacts -with the REST API. +- the server-side REST API (“API”) which holds the business logic and + keeps the state of the system consistent -### Used technologies +- the relational database (“DB”) which persists the state of the + system -@todo: Briefly introduce the used technologies like React, Redux and the -build process. For further details refer to the GitHub wiki +- the client side application (“client”) which simplifies access to + the API for the common users -### How to use the application +The centerpiece of this architecture is the API. This component receives +requests from the users and from the Backend, validates them and +modifies the state of the system and persists this modified state in the +DB. -@todo: Describe the user documentation and the FAQ page. +We have created a web application which can communicate with the API +server and present the information received from the server to the user +in a convenient way. The client can be though any application, which can +send HTTP requests and receive the HTTP responses. Users can use general +applications like [cURL](https://github.com/curl/curl/), +[Postman](https://www.getpostman.com/), or create their own specific +client for ReCodEx API. -Backend-Frontend communication protocol -======================================= +Frontend capabilities +--------------------- -@todo: describe the exact methods and respective commands for the -communication +@todo: describe what the frontend is capable of and how it really works, +what are the limitations and how it can be extended -Initiation of a job evaluation ------------------------------- +Terminology +----------- -@todo: How does the Frontend initiate the evaluation and how the Backend -can accept it or decline it +This project was created for the needs of a university and this fact is +reflected into the terminology used throughout the Frontend. A list of +important terms’ definitions follows to make the meaning unambiguous. -Job processing progress monitoring ----------------------------------- +### User and user roles -When evaluating a job the worker sends progress messages on predefined points of -evaluation chain. The sending place can be on very beginning of the job, when -submit archive is downloaded or at the end of each simple task with its state -(completed, failed, skipped). These messages are sent to broker through existing -ZeroMQ connection. Detailed format of messages can be found on [communication -page](https://github.com/ReCodEx/wiki/wiki/Overall-architecture#commands-from-worker-to-broker). +*User* is a person who uses the application. User is granted access to +the application once he or she creates an account directly through the +API or the web application. There are several types of user accounts +depending on the set of permissions – a so called “role” – they have +been granted. Each user receives only the most basic set of permissions +after he or she creates an account and this role can be changed only by +the administrators of the service: -Broker only resends received progress messages to the monitor component via -ZeroMQ socket. The output message format is the same as the input format. +- *Student* is the most basic role. Student can become member of a + group and submit his solutions to his assignments. -Monitor parses received messages to JSON format, which is easy to work with in -JavaScript inside web application. All messages are cached (one queue per job) -and can be obtained multiple times through WebSocket communication channel. The -cache is cleared 5 minutes after receiving last message. +- *Supervisor* can be entitled to manage a group of students. + Supervisor can assign exercises to the students who are members of + his groups and review their solutions submitted to + these assignments. + +- *Super-admin* is a user with unlimited rights. This user can perform + any action in the system. + +There are two implicit changes of roles: -Publishing of the results -------------------------- +- Once a *student* is added to a group as its supervisor, his role is + upgraded to a *supervisor* role. -After job finish the worker packs results directory into single archive and -uploads it to the fileserver through HTTP protocol. The target URL is obtained -from API in headers on job initiation. Then "job done" notification request is -performed to API via broker. Special submissions (reference or asynchronous -submissions) are loaded immediately, other types are loaded on-demand on first -results request. +- Once a *supervisor* is removed from the lasts group where he is a + supervisor then his role is downgraded to a *student* role. -Loading results means fetching archive from fileserver, parsing the main YAML -file generated by worker and saving data to the database. Also, points are -assigned by score calculator. +These mechanisms do not prevent a single user being a supervisor of one +group and student of a different group as supervisors’ permissions are +superset of students’ permissions. +### Login -# User documentation +*Login* is a set of user’s credentials he must submit to verify he can +be allowed to access the system as a specific user. We distinguish two +types of logins: local and external. -@todo: Describe different scenarios of the usage of the Web App +- *Local login* is user’s email address and a password he chooses + during registration. -@todo: Describe the requirements of running the web application (modern web browser, enabled CSS, JavaScript, Cookies & Local storage) +- *External login* is a mapping of a user profile to an account of + some authentication service (e.g., [CAS](https://ldap1.cuni.cz/)). -## Terminology +### Instance -@todo: Describe the terminology: Instance, User, Group, Student, -Supervisor, Admin +*An instance* of ReCodEx is in fact just a set of groups and user +accounts. An instance should correspond to a real entity as a +university, a high-school, an IT company or an HR agency. This approach +enables the system to be shared by multiple independent organizations +without interfering with each other. -## General basics +Usage of the system by the users of an instance can be limited by +possessing a valid license. It is up to the administrators of the system +to determine the conditions under which they will assign licenses to the +instances. -@todo: actions which are available for all users +### Group -@todo: how to solve problems with ReCodEx, first supervisors, then administrators, etc... +*Group* corresponds to a school class or some other unit which gathers +users who will be assigned the same set exercises. Each group can have +multiple supervisors who can manage the students and the list of +assignments. -### First steps in ReCodEx +Groups can form a tree hierarchy of arbitrary depth. This is inspired by the +hierarchy of school classes belonging to the same subject over several school +years. For example, there can be a top level group for a programming class that +contains subgroups for every school year. These groups can then by divided into +actual student groups with respect to lab attendance. Supervisors can create +subgroups of their groups and further manage these subgroups. -You can create an account if you click on the “*Create account*” menu -item in the left sidebar. You can choose between two types of -registration methods – by creating a local account with a specific -password, or pairing your new account with an existing CAS UK account. +### Exercise -If you decide a new “*local*” account using the “*Create ReCodEx -account*” form, you will have to provide your details and choose a -password for your account. You will later sign in using your email -address as your username and the password you select. +*An exercise* consists of textual assignment of a task and a definition +of how a solution to this exercise should be processed and evaluated in +a specific runtime environment (i.e., how to compile a submitted source +code and how to test the correctness of the program). It is a template +which can be instantiated as an *assignment* by a supervisor of a group. -If you decide to use the CAS UK, then we will verify your credentials -and access your name and email stored in the system and create your -account based on this information. You can change your personal -information or email later on the “*Settings*” page. +### Assignment -When creating your account both ways, you must select an instance your -account will belong to by default. The instance you will select will be -most likely your university or other organization you are a member of. +An assignment is an instance of an *exercise* assigned to a specific +*group*. An assignment can modify the text of the task assignment and it +has some additional information which is specific to the group (e.g., a +deadline, the number of points gained for a correct solution, additional +hints for the students in the assignment). The text of the assignment +can be edited and supervisors can translate the assignment into another +language. -To log in, go to the homepage of ReCodEx and in the left sidebar choose -the menu item “*Sign in*”. Then you must enter your credentials into one -of the two forms – if you selected a password during registration, then -you should sign with your email and password in the first form called -“*Sign into ReCodEx*”. If you registered using the Charles University -Authentication Service (CAS), you should put your student’s number and -your CAS password into the second form called “Sign into ReCodEx using -CAS UK”. +### Solution -There are several options you can edit in your user account: +*A solution* is a set of files which a user submits to a given +*assignment*. -- changing your personal information (i.e., name) -- changing your credentials (email and password) -- updating your preferences (e.g., source code viewer/editor settings, - default language) +### Submission -You can access the settings page through the “*Settings*” button right -under your name in the left sidebar. +*A submission* corresponds to a *solution* being evaluated by the +Backend. A single *solution* can be submitted repeatedly (e.g., when the +Backend encounters an error or when the supervisor changes the assignment). -If you don’t use ReCodEx for a whole day, you will be logged out -automatically. However, we recommend you sign out of the application -after you finished your interaction with it. The logout button is placed -in the top section of the left sidebar right under your name. You will -have to expand the sidebar with a button next to the “*ReCodEx*” title -(shown in the picture below). +### Evaluation -### Forgotten password +*An evaluation* is the processed report received from the Backend after +a *submission* is processed. Evaluation contains points given to the +user based on the quality of his solution measured by the Backend and +the settings of the assignment. Supervisors can review the evaluation +and add bonus points (both positive and negative) if the student +deserves some. -If you can’t remember your password and you don’t use CAS UK -authentication, then you can reset your password. You will find a link -saying “*You cannot remember what your password was? Reset your -password.*” under the sign in form. After you click on this link, you -will be asked to submit your email address. An email with a link -containing a special token will be sent to the address you fill in. We -make sure that the person who requested password resetting is really -you. When you click on the link (or you copy & paste it into your web -browser) you will be able to select a new password for your account. The -token is valid only for a couple of minutes, so do not forget to reset -the password as soon as possible, or you will have to request a new link -with a valid token. +### Runtime environment -If you sign in through CAS UK, then please follow the instructions -provided by the administrators of the service described on their -website. +*A runtime environment* defines the used programming language or tools +which are needed to process and evaluate a solution. Examples of a +runtime environment can be: +- *Linux + GCC* +- *Linux + Mono* +- *Windows + .NET 4* +- *Bison + Yacc* -## Student +### Limits -@todo: describe what it means to be a “student” and what are the -student’s rights +A correct *solution* of an *assignment* has to pass all specified tests (mostly +checks that it yields the correct output for various inputs) and typically must +also be effective in some sense. The Backend measures the time and memory +consumption of the solution while running. This consumption of resources can be +*limited* and the solution will receive fewer points if it exceeds the given +limits in some test cases defined by the *exercise*. -### Join group and start solving assignments +User management +--------------- -@todo: How to join a specific group +@todo: roles and their rights, adding/removing different users, how the +role of a specific user changes -@todo: Where can the user see groups description and details, what -information is available. +Instances and hierarchy of groups +--------------------------------- -@todo: Where the student can find the list of the assignment he is -expected to solve, what is the first and second deadline. +@todo: What is an instance, how to create one, what are the licenses and +how do they work. Why can the groups form hierarchies and what are the +benefits – what it means to be an admin of a group, hierarchy of roles +in the group hierarchy. -@todo: How does a student submit his solution through the web app +Exercises database +------------------ -@todo: When the results are ready and what the results mean and what to -do about them, when the user is convinced, that his solution is correct -although the results say different +@todo: How the exercises are stored, accessed, who can edit what -@todo: Describe the comments thread behavior (public/private comments), -who else can see the comments, how notifications work (*not implemented -yet*!). +### Creating a new exercise +@todo Localized assignments, default settings -## Group supervisor +### Runtime environments and hardware groups -@todo: describe what it means to be a “supervisor” of a group and what -are the supervisors rights +@todo read this later and see if it still makes sense -### Create groups and manage them +ReCodEx is designed to utilize a rather diverse set of workers -- there can be +differences in many aspects, such as the actual hardware running the worker +(which impacts the results of measuring) or installed compilers, interpreters +and other tools needed for evaluation. To address these two examples in +particular, we assign runtime environments and hardware groups to exercises. -@todo: How does a user become a supervisor of a group? +The purpose of runtime environments is to specify which tools (and often also +operating system) are required to evaluate a solution of the exercise -- for +example, a C# programming exercise can be evaluated on a Linux worker running +Mono or a Windows worker with the .NET runtime. Such exercise would be assigned +two runtime environments, `Linux+Mono` and `Windows+.NET` (the environment names +are arbitrary strings configured by the administrator). -@todo: How to add a specific student to a given group +A hardware group is a set of workers that run on similar hardware (e.g. a +particular quad-core processor model and a SSD hard drive). Workers are assigned +to these groups by the administrator. If this is done correctly, performance +measurements of a submission should yield the same results. Thanks to this fact, +we can use the same resource limits on every worker in a hardware group. +However, limits can differ between runtime environments -- formally speaking, +limits are a function of three arguments: an assignment, a hardware group and a +runtime environment. -### Assigning exercises +### Reference solutions -@todo: Describe how to access the database of the exercises and what are -the possibilities of assignment setup – availability, deadlines, points, -score configuration, limits +@todo: how to add one, how to evaluate it -@todo: How can I assign some exercises only to some students of the group? Describe how to achieve this using subgroups +The task of determining appropriate resource limits for exercises is difficult +to do correctly. To aid exercise authors and group supervisors, ReCodEx supports +assigning reference solutions to exercises. Those are example programs that +should cover the main approaches to the implementation. For example, searching +for an integer in an ordered array can be done with a linear search, or better, +using a binary search. -### Students' solutions management +Reference solutions can be evaluated on demand, using a selected hardware group. +The evaluation results are stored and can be used later to determine limits. In +our example problem, we could configure the limits so that the linear +search-based program doesn't finish in time on larger inputs, but a binary +search does. -@todo Describe where all the students’ solutions for a given assignment -can be found, where to look for all solutions of a given student, how to -see results of a specific student’s solution’s evaluation result. +Note that separate reference solutions should be supplied for all supported +runtime environments. -@todo Can I assign points to my students’ solutions manually instead of depending on automatic scoring? If and how to change the score of a solution – assignment -settings, setting points, bonus points, accepting a solution (*not -implemented yet!*). Describe how the student and supervisor will still -be able to see the percentage received from the automatic scoring, but -the awarded points will be overridden. +### Exercise assignments -@todo: Describe the comments thread behavior (public/private comments), -who else can see the comments -- same as from the student perspective +@todo: Creating instances of an exercise for a specific group of users, +capabilities of settings. Editing limits according to the reference +solution. -### Creating exercises +Evaluation process +------------------ -@todo: how to create exercise, what has to be provided during creation, who can create exercises +@todo: How the evaluation process works on the Frontend side. -@todo: Describe the form and explain the concept of reference solutions. -How to evaluate the reference solutions for the exercise right now (to -get the up-to-date information). +### Uploading files and file storage + +@todo: One by one upload endpoint. Explain different types of the +Uploaded files. +### Automatic detection of the runtime environment -## Group administrator +@todo: Users must submit correctly named files – assuming the RTE from +the extensions. -@todo: who is this? +REST API implementation +----------------------- -### Creating subgroups and managing supervisors +@todo: What is the REST API, what are the basic principles – GET, POST, +Headers, JSON. -@todo: What it means to create a subgroup and how to do it. +### Authentication and authorization scopes -@todo: who can add another supervisor, what would be the rights of the -second supervisor +@todo: How authentication works – signed JWT, headers, expiration, +refreshing. Token scopes usage. +### HTTP requests handling -## Superadministrator +@todo: Router and routes with specific HTTP methods, preflight, required +headers -Superadmin is user with the most priviledges and as such superadmin should be -quite unique role. Ideally there should be only one of this kind, used with -special caution and adequate security. With this stated it is obvious that -superadmin can perform any action the API is capable of. +### HTTP responses format -### Users management +@todo: Describe the JSON structure convention of success and error +responses -There are only few roles to which users can belong in ReCodEx. Basically there -are only three: _student_, _supervisor_, and _superadmin_. Base role is student -which is assigned to every registered user. Roles are stored in database -alongside other information about user. One user always has only one role at the -time. At first startup of ReCodEx administrator should create his account and -then change role in database by hand. After that manual intervention into -database should never be needed. +### Used technologies -There is a little catch in groups and instances management. Groups can have -admins and supervisors. This setting is valid only per one particular group and -has to be separated from basic role system. This implies that supervisor in one -group can be student in another and simultaneously have global supervisor role. -Changing role from student to supervisor and back is done automatically by -application and should not be managed by hand in database! Previously stated -information can be applied to instances as well, but instances can only have -admins. +@todo: PHP7 – how it is used for typehints, Nette framework – how it is +used for routing, Presenters actions endpoints, exceptions and +ErrorPresenter, Doctrine 2 – database abstraction, entities and +repositories + conventions, Communication over ZMQ – describe the +problem with the extension and how we reported it and how to treat it in +the future when the bug is solved. Relational database – we use MariaDB, +Doctine enables us to switch the engine to a different engine if needed -Roles description: +### Data model -- Student -- Default role which is used for newly created accounts. Student can - join or leave public groups and submit solutions of assigned exercises. -- Supervisor -- Inherits all permissions from student role. Can manage groups to - which he/she belongs to. Supervisor can also view and change groups details, - manage assigned exercises, view students in group and their solutions for - assigned exercises. On top of that supervisor can create/delete groups too, - but only as subgroup of groups he/she belongs to. -- Superadmin -- Inherits all permissions from supervisor role. Most powerful - user in ReCodEx who should be able to do everything which is provided by - application. +@todo: Describe the code-first approach using the Doctrine entities, how +the entities map onto the database schema (refer to the attached schemas +of entities and relational database models), describe the logical +grouping of entities and how they are related: +- user + settings + logins + ACL +- instance + licenses + groups + group membership +- exercise + assignments + localized assignments + runtime + environments + hardware groups +- submission + solution + reference solution + solution evaluation +- comment threads + comments -## Writing job configuration +### API endpoints -To run and evaluate an exercise the backend needs to know the steps how to do -that. This is different for each environment (operation system, programming -language, etc.), so each of the environments needs to have separate -configuration. +@todo: Tell the user about the generated API reference and how the +Swagger UI can be used to access the API directly. -Backend works with a powerful, but quite low level description of simple -connected tasks written in YAML syntax. More about the syntax and general task -overview can be found on [separate -page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned -features was user friendly configuration editor, but due to tight deadline and -team composition it did not make it to the first release. However, writing -configuration in the basic format will be always available and allows users to -use the full expressive power of the system. +Web Application +--------------- -This section walks through creation of job configuration for _hello world_ -exercise. The goal is to compile file _source.c_ and check if it prints `Hello -World!` to the standard output. This is the only test case, let's call it -**A**. +@todo: What is the purpose of the web application and how it interacts +with the REST API. -The problem can be split into several tasks: +### Used technologies -- compile _source.c_ into _helloworld_ with `/usr/bin/gcc` -- run _helloworld_ and save standard output into _out.txt_ -- fetch predefined output (suppose it is already uploaded to fileserver) with - hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_ -- compare _out.txt_ and _reference.txt_ by `/usr/bin/diff` +@todo: Briefly introduce the used technologies like React, Redux and the +build process. For further details refer to the GitHub wiki -The absolute path of tools can be obtained from system administrator. However, -`/usr/bin/gcc` is location, where the GCC binary is available almost everywhere, -so location of some tools can be (professionally) guessed. +### How to use the application -First, write header of the job to the configuration file. +@todo: Describe the user documentation and the FAQ page. -```{.yml} -submission: - job-id: hello-word-job - hw-groups: - - group1 -``` +Backend-Frontend communication protocol +======================================= -Basically it means, that the job _hello-world-job_ needs to be run on workers -that belong to the `group_1` hardware group . Reference files are downloaded -from the default location configured in API (such as -`http://localhost:9999/exercises`) if not stated explicitly otherwise. Job -execution log will not be saved to result archive. +@todo: describe the exact methods and respective commands for the +communication -Next the tasks have to be constructed under _tasks_ section. In this demo job, -every task depends only on previous one. The first task has input file -_source.c_ (if submitted by user) already available in working directory, so -just call the GCC. Compilation is run in sandbox as any other external program -and should have relaxed time and memory limits. In this scenario, worker -defaults are used. If compilation fails, the whole job is immediately terminated -(because the _fatal-failure_ bit is set). Because _bound-directories_ option in -sandbox limits section is mostly shared between all tasks, it can be set in -worker configuration instead of job configuration (suppose this for following -tasks). For configuration of workers please contact your administrator. +Initiation of a job evaluation +------------------------------ -```{.yml} -- task-id: "compilation" - type: "initiation" - fatal-failure: true - cmd: - bin: "/usr/bin/gcc" - args: - - "source.c" - - "-o" - - "helloworld" - sandbox: - name: "isolate" - limits: - - hw-group-id: group1 - chdir: ${EVAL_DIR} - bound-directories: - - src: ${SOURCE_DIR} - dst: ${EVAL_DIR} - mode: RW -``` +@todo: How does the Frontend initiate the evaluation and how the Backend +can accept it or decline it -The compiled program is executed with time and memory limit set and the standard -output is redirected to a file. This task depends on _compilation_ task, because -the program cannot be executed without being compiled first. It is important to -mark this task with _execution_ type, so exceeded limits will be reported in -frontend. +Job processing progress monitoring +---------------------------------- -Time and memory limits set directly for a task have higher priority than worker -defaults. One important constraint is, that these limits cannot exceed limits -set by workers. Worker defaults are present as a safety measure so that a -malformed job configuration cannot block the worker forever. Worker default -limits should be reasonably high, like a gigabyte of memory and several hours of -execution time. For exact numbers please contact your administrator. +When evaluating a job the worker sends progress messages on predefined points of +evaluation chain. The sending place can be on very beginning of the job, when +submit archive is downloaded or at the end of each simple task with its state +(completed, failed, skipped). These messages are sent to broker through existing +ZeroMQ connection. Detailed format of messages can be found on [communication +page](https://github.com/ReCodEx/wiki/wiki/Overall-architecture#commands-from-worker-to-broker). -It is important to know that if the output of a program (both standard and -error) is redirected to a file, the sandbox disk quotas apply to that file, as -well as the files created directly by the program. In case the outputs are -ignored, they are redirected to `/dev/null`, which means there is no limit on -the output length (as long as the printing fits in the time limit). +Broker only resends received progress messages to the monitor component via +ZeroMQ socket. The output message format is the same as the input format. -```{.yml} -- task-id: "execution_1" - test-id: "A" - type: "execution" - dependencies: - - compilation - cmd: - bin: "helloworld" - sandbox: - name: "isolate" - stdout: ${EVAL_DIR}/out.txt - limits: - - hw-group-id: group1 - chdir: ${EVAL_DIR} - time: 0.5 - memory: 8192 -``` +Monitor parses received messages to JSON format, which is easy to work with in +JavaScript inside web application. All messages are cached (one queue per job) +and can be obtained multiple times through WebSocket communication channel. The +cache is cleared 5 minutes after receiving last message. -Fetch sample solution from file server. Base URL of file server is in the header -of the job configuration, so only the name of required file (its `sha1sum` in -our case) is necessary. +Publishing of the results +------------------------- -```{.yml} -- task-id: "fetch_solution_1" - test-id: "A" - dependencies: - - execution - cmd: - bin: "fetch" - args: - - "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b" - - "${SOURCE_DIR}/reference.txt" -``` +After job finish the worker packs results directory into single archive and +uploads it to the fileserver through HTTP protocol. The target URL is obtained +from API in headers on job initiation. Then "job done" notification request is +performed to API via broker. Special submissions (reference or asynchronous +submissions) are loaded immediately, other types are loaded on-demand on first +results request. + +Loading results means fetching archive from fileserver, parsing the main YAML +file generated by worker and saving data to the database. Also, points are +assigned by score calculator. -Comparison of results is quite straightforward. It is important to set the task -type to _evaluation_, so that the return code is set to 0 if the program is -correct and 1 otherwise. We do not set our own limits, so the default limits are -used. -```{.yml} -- task-id: "judge_1" - test-id: "A" - type: "evaluation" - dependencies: - - fetch_solution_1 - cmd: - bin: "/usr/bin/diff" - args: - - "out.txt" - - "reference.txt" - sandbox: - name: "isolate" - limits: - - hw-group-id: group1 - chdir: ${EVAL_DIR} -```