From 4c4599d5decfa5cc55a31fd45e2945bedf25b328 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Thu, 12 Jan 2017 11:16:06 +0100 Subject: [PATCH 1/7] User doc restruct --- Rewritten-docs.md | 90 +++++++++++++++++------------------------------ 1 file changed, 33 insertions(+), 57 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 795d883..ef77ff6 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -2069,7 +2069,9 @@ Supervisor, Admin @todo: actions which are available for all users -### How to create a user account? +@todo: how to solve problems with ReCodEx, first supervisors, then administrators, etc... + +### First steps in ReCodEx You can create an account if you click on the “*Create account*” menu item in the left sidebar. You can choose between two types of @@ -2086,12 +2088,10 @@ and access your name and email stored in the system and create your account based on this information. You can change your personal information or email later on the “*Settings*” page. -When crating your account both ways, you must select an instance your +When creating your account both ways, you must select an instance your account will belong to by default. The instance you will select will be most likely your university or other organization you are a member of. -### How to get into ReCodEx? - To log in, go to the homepage of ReCodEx and in the left sidebar choose the menu item “*Sign in*”. Then you must enter your credentials into one of the two forms – if you selected a password during registration, then @@ -2101,7 +2101,15 @@ Authentication Service (CAS), you should put your student’s number and your CAS password into the second form called “Sign into ReCodEx using CAS UK”. -### How do I sign out of ReCodEx? +There are several options you can edit in your user account: + +- changing your personal information (i.e., name) +- changing your credentials (email and password) +- updating your preferences (e.g., source code viewer/editor settings, + default language) + +You can access the settings page through the “*Settings*” button right +under your name in the left sidebar. If you don’t use ReCodEx for a whole day, you will be logged out automatically. However, we recommend you sign out of the application @@ -2110,9 +2118,7 @@ in the top section of the left sidebar right under your name. You will have to expand the sidebar with a button next to the “*ReCodEx*” title (shown in the picture below). -@todo: Simon's image - -### What to do when you cannot remember your password? +### Forgotten password If you can’t remember your password and you don’t use CAS UK authentication, then you can reset your password. You will find a link @@ -2131,50 +2137,28 @@ If you sign in through CAS UK, then please follow the instructions provided by the administrators of the service described on their website. -### How to configure your account? - -There are several options you have to edit your user account. - -- changing your personal information (i.e., name) -- changing your credentials (email and password) -- updating your preferences (e.g., source code viewer/editor settings, - default language) - -You can access the settings page through the “*Settings*” button right -under your name in the left sidebar. - ## Student @todo: describe what it means to be a “student” and what are the student’s rights -### How to join a group for my class? +### Join group and start solving assignments @todo: How to join a specific group -### Which assignments do I have to solve? - -@todo: Where the student can find the list of the assignment he is -expected to solve, what is the first and second deadline. - -### Where can I see details of my classes’ group? - @todo: Where can the user see groups description and details, what information is available. -### How to submit a solution of an assignment? +@todo: Where the student can find the list of the assignment he is +expected to solve, what is the first and second deadline. @todo: How does a student submit his solution through the web app -### Where are the results of my solutions? - @todo: When the results are ready and what the results mean and what to do about them, when the user is convinced, that his solution is correct although the results say different -### How can I discuss my solution with my teacher/group’s supervisor directly through the web application? - @todo: Describe the comments thread behavior (public/private comments), who else can see the comments, how notifications work (*not implemented yet*!). @@ -2185,63 +2169,55 @@ yet*!). @todo: describe what it means to be a “supervisor” of a group and what are the supervisors rights -### How do I become a supervisor of a group? +### Create groups and manage them @todo: How does a user become a supervisor of a group? -### How to add or remove a student to my group? - @todo: How to add a specific student to a given group -### How do I assign an exercise to my students? +### Assigning exercises @todo: Describe how to access the database of the exercises and what are the possibilities of assignment setup – availability, deadlines, points, score configuration, limits -### How do I configure the limits of an assignment and how to choose appropriate limits? +@todo: How can I assign some exercises only to some students of the group? Describe how to achieve this using subgroups -@todo: Describe the form and explain the concept of reference solutions. -How to evaluate the reference solutions for the exercise right now (to -get the up-to-date information). - -### How can I assign some exercises only to some students of the group? - -@todo: Describe how to achieve this using subgroups - -### How can I see my students’ solutions? +### Students' solutions management @todo Describe where all the students’ solutions for a given assignment can be found, where to look for all solutions of a given student, how to see results of a specific student’s solution’s evaluation result. -### Can I assign points to my students’ solutions manually instead of depending on automatic scoring? - -@todo If and how to change the score of a solution – assignment +@todo Can I assign points to my students’ solutions manually instead of depending on automatic scoring? If and how to change the score of a solution – assignment settings, setting points, bonus points, accepting a solution (*not implemented yet!*). Describe how the student and supervisor will still be able to see the percentage received from the automatic scoring, but the awarded points will be overridden. -### How can I discuss student’s solution with him/her directly through the web application? - @todo: Describe the comments thread behavior (public/private comments), who else can see the comments -- same as from the student perspective +### Creating exercises + +@todo: how to create exercise, what has to be provided during creation, who can create exercises + +@todo: Describe the form and explain the concept of reference solutions. +How to evaluate the reference solutions for the exercise right now (to +get the up-to-date information). + ## Group administrator @todo: who is this? -### How do I add another supervisor to my group? +### Creating subgroups and managing supervisors + +@todo: What it means to create a subgroup and how to do it. @todo: who can add another supervisor, what would be the rights of the second supervisor -### How do I create a subgroup of my group? - -@todo: What it means to create a subgroup and how to do it. - ## Superadministrator From 88953c7ffb4b8be34cc9b8f28e3efc84f14a2f66 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Thu, 12 Jan 2017 13:34:17 +0100 Subject: [PATCH 2/7] User doc is now right after analysis --- Rewritten-docs.md | 1149 +++++++++++++++++++++++---------------------- 1 file changed, 576 insertions(+), 573 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index ef77ff6..b9b1aa1 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1637,783 +1637,786 @@ be implemented in next releases. -# The Backend - -The backend is the part which is hidden to the user and which has only -one purpose: evaluate user’s solutions of their assignments. +# User documentation -@todo: describe the configuration inputs of the Backend +@todo: Describe different scenarios of the usage of the Web App -@todo: describe the outputs of the Backend +@todo: Describe the requirements of running the web application (modern web browser, enabled CSS, JavaScript, Cookies & Local storage) -@todo: describe how the backend receives the inputs and how it -communicates the results +## Terminology -## Components +@todo: Describe the terminology: Instance, User, Group, Student, +Supervisor, Admin -Whole backend is not just one service/component, it is quite complex system on its own. +## General basics -@todo: describe the inner parts of the Backend (and refer to the Wiki -for the technical description of the components) +@todo: actions which are available for all users -### Broker +@todo: how to solve problems with ReCodEx, first supervisors, then administrators, etc... -@todo: gets stuff done, single point of failure and center point of ReCodEx universe +### First steps in ReCodEx -### Fileserver +You can create an account if you click on the “*Create account*” menu +item in the left sidebar. You can choose between two types of +registration methods – by creating a local account with a specific +password, or pairing your new account with an existing CAS UK account. -@todo: stores particular data from frontend and backend, hashing, HTTP API +If you decide a new “*local*” account using the “*Create ReCodEx +account*” form, you will have to provide your details and choose a +password for your account. You will later sign in using your email +address as your username and the password you select. -### Worker +If you decide to use the CAS UK, then we will verify your credentials +and access your name and email stored in the system and create your +account based on this information. You can change your personal +information or email later on the “*Settings*” page. -@todo: describe a bit of internal structure in general +When creating your account both ways, you must select an instance your +account will belong to by default. The instance you will select will be +most likely your university or other organization you are a member of. -@todo: describe how jobs are generally executed +To log in, go to the homepage of ReCodEx and in the left sidebar choose +the menu item “*Sign in*”. Then you must enter your credentials into one +of the two forms – if you selected a password during registration, then +you should sign with your email and password in the first form called +“*Sign into ReCodEx*”. If you registered using the Charles University +Authentication Service (CAS), you should put your student’s number and +your CAS password into the second form called “Sign into ReCodEx using +CAS UK”. -### Monitor +There are several options you can edit in your user account: -@todo: not necessary component which can be omitted, proxy-like service +- changing your personal information (i.e., name) +- changing your credentials (email and password) +- updating your preferences (e.g., source code viewer/editor settings, + default language) -## Backend internal communication +You can access the settings page through the “*Settings*” button right +under your name in the left sidebar. -@todo: internal backend communication, what communicates with what and why +If you don’t use ReCodEx for a whole day, you will be logged out +automatically. However, we recommend you sign out of the application +after you finished your interaction with it. The logout button is placed +in the top section of the left sidebar right under your name. You will +have to expand the sidebar with a button next to the “*ReCodEx*” title +(shown in the picture below). -The Frontend -============ +### Forgotten password -The frontend is the part which is visible to the user of ReCodEx and -which holds the state of the system – the user accounts, their roles in -the system, the database of exercises, the assignments of these -exercises to groups of users (i.e., students), and the solutions and -evaluations of them. +If you can’t remember your password and you don’t use CAS UK +authentication, then you can reset your password. You will find a link +saying “*You cannot remember what your password was? Reset your +password.*” under the sign in form. After you click on this link, you +will be asked to submit your email address. An email with a link +containing a special token will be sent to the address you fill in. We +make sure that the person who requested password resetting is really +you. When you click on the link (or you copy & paste it into your web +browser) you will be able to select a new password for your account. The +token is valid only for a couple of minutes, so do not forget to reset +the password as soon as possible, or you will have to request a new link +with a valid token. -Frontend is split into three parts: +If you sign in through CAS UK, then please follow the instructions +provided by the administrators of the service described on their +website. -- the server-side REST API (“API”) which holds the business logic and - keeps the state of the system consistent -- the relational database (“DB”) which persists the state of the - system +## Student -- the client side application (“client”) which simplifies access to - the API for the common users +@todo: describe what it means to be a “student” and what are the +student’s rights -The centerpiece of this architecture is the API. This component receives -requests from the users and from the Backend, validates them and -modifies the state of the system and persists this modified state in the -DB. +### Join group and start solving assignments -We have created a web application which can communicate with the API -server and present the information received from the server to the user -in a convenient way. The client can be though any application, which can -send HTTP requests and receive the HTTP responses. Users can use general -applications like [cURL](https://github.com/curl/curl/), -[Postman](https://www.getpostman.com/), or create their own specific -client for ReCodEx API. +@todo: How to join a specific group -Frontend capabilities ---------------------- +@todo: Where can the user see groups description and details, what +information is available. -@todo: describe what the frontend is capable of and how it really works, -what are the limitations and how it can be extended +@todo: Where the student can find the list of the assignment he is +expected to solve, what is the first and second deadline. -Terminology ------------ +@todo: How does a student submit his solution through the web app -This project was created for the needs of a university and this fact is -reflected into the terminology used throughout the Frontend. A list of -important terms’ definitions follows to make the meaning unambiguous. +@todo: When the results are ready and what the results mean and what to +do about them, when the user is convinced, that his solution is correct +although the results say different -### User and user roles +@todo: Describe the comments thread behavior (public/private comments), +who else can see the comments, how notifications work (*not implemented +yet*!). -*User* is a person who uses the application. User is granted access to -the application once he or she creates an account directly through the -API or the web application. There are several types of user accounts -depending on the set of permissions – a so called “role” – they have -been granted. Each user receives only the most basic set of permissions -after he or she creates an account and this role can be changed only by -the administrators of the service: -- *Student* is the most basic role. Student can become member of a - group and submit his solutions to his assignments. +## Group supervisor -- *Supervisor* can be entitled to manage a group of students. - Supervisor can assign exercises to the students who are members of - his groups and review their solutions submitted to - these assignments. +@todo: describe what it means to be a “supervisor” of a group and what +are the supervisors rights -- *Super-admin* is a user with unlimited rights. This user can perform - any action in the system. +### Create groups and manage them -There are two implicit changes of roles: +@todo: How does a user become a supervisor of a group? -- Once a *student* is added to a group as its supervisor, his role is - upgraded to a *supervisor* role. +@todo: How to add a specific student to a given group -- Once a *supervisor* is removed from the lasts group where he is a - supervisor then his role is downgraded to a *student* role. +### Assigning exercises -These mechanisms do not prevent a single user being a supervisor of one -group and student of a different group as supervisors’ permissions are -superset of students’ permissions. +@todo: Describe how to access the database of the exercises and what are +the possibilities of assignment setup – availability, deadlines, points, +score configuration, limits -### Login +@todo: How can I assign some exercises only to some students of the group? Describe how to achieve this using subgroups -*Login* is a set of user’s credentials he must submit to verify he can -be allowed to access the system as a specific user. We distinguish two -types of logins: local and external. +### Students' solutions management -- *Local login* is user’s email address and a password he chooses - during registration. +@todo Describe where all the students’ solutions for a given assignment +can be found, where to look for all solutions of a given student, how to +see results of a specific student’s solution’s evaluation result. -- *External login* is a mapping of a user profile to an account of - some authentication service (e.g., [CAS](https://ldap1.cuni.cz/)). +@todo Can I assign points to my students’ solutions manually instead of depending on automatic scoring? If and how to change the score of a solution – assignment +settings, setting points, bonus points, accepting a solution (*not +implemented yet!*). Describe how the student and supervisor will still +be able to see the percentage received from the automatic scoring, but +the awarded points will be overridden. -### Instance +@todo: Describe the comments thread behavior (public/private comments), +who else can see the comments -- same as from the student perspective -*An instance* of ReCodEx is in fact just a set of groups and user -accounts. An instance should correspond to a real entity as a -university, a high-school, an IT company or an HR agency. This approach -enables the system to be shared by multiple independent organizations -without interfering with each other. +### Creating exercises -Usage of the system by the users of an instance can be limited by -possessing a valid license. It is up to the administrators of the system -to determine the conditions under which they will assign licenses to the -instances. +@todo: how to create exercise, what has to be provided during creation, who can create exercises -### Group +@todo: Describe the form and explain the concept of reference solutions. +How to evaluate the reference solutions for the exercise right now (to +get the up-to-date information). -*Group* corresponds to a school class or some other unit which gathers -users who will be assigned the same set exercises. Each group can have -multiple supervisors who can manage the students and the list of -assignments. -Groups can form a tree hierarchy of arbitrary depth. This is inspired by the -hierarchy of school classes belonging to the same subject over several school -years. For example, there can be a top level group for a programming class that -contains subgroups for every school year. These groups can then by divided into -actual student groups with respect to lab attendance. Supervisors can create -subgroups of their groups and further manage these subgroups. +## Group administrator -### Exercise +@todo: who is this? -*An exercise* consists of textual assignment of a task and a definition -of how a solution to this exercise should be processed and evaluated in -a specific runtime environment (i.e., how to compile a submitted source -code and how to test the correctness of the program). It is a template -which can be instantiated as an *assignment* by a supervisor of a group. +### Creating subgroups and managing supervisors -### Assignment +@todo: What it means to create a subgroup and how to do it. -An assignment is an instance of an *exercise* assigned to a specific -*group*. An assignment can modify the text of the task assignment and it -has some additional information which is specific to the group (e.g., a -deadline, the number of points gained for a correct solution, additional -hints for the students in the assignment). The text of the assignment -can be edited and supervisors can translate the assignment into another -language. +@todo: who can add another supervisor, what would be the rights of the +second supervisor -### Solution -*A solution* is a set of files which a user submits to a given -*assignment*. +## Superadministrator -### Submission +Superadmin is user with the most priviledges and as such superadmin should be +quite unique role. Ideally there should be only one of this kind, used with +special caution and adequate security. With this stated it is obvious that +superadmin can perform any action the API is capable of. -*A submission* corresponds to a *solution* being evaluated by the -Backend. A single *solution* can be submitted repeatedly (e.g., when the -Backend encounters an error or when the supervisor changes the assignment). +### Users management -### Evaluation +There are only few roles to which users can belong in ReCodEx. Basically there +are only three: _student_, _supervisor_, and _superadmin_. Base role is student +which is assigned to every registered user. Roles are stored in database +alongside other information about user. One user always has only one role at the +time. At first startup of ReCodEx administrator should create his account and +then change role in database by hand. After that manual intervention into +database should never be needed. -*An evaluation* is the processed report received from the Backend after -a *submission* is processed. Evaluation contains points given to the -user based on the quality of his solution measured by the Backend and -the settings of the assignment. Supervisors can review the evaluation -and add bonus points (both positive and negative) if the student -deserves some. +There is a little catch in groups and instances management. Groups can have +admins and supervisors. This setting is valid only per one particular group and +has to be separated from basic role system. This implies that supervisor in one +group can be student in another and simultaneously have global supervisor role. +Changing role from student to supervisor and back is done automatically by +application and should not be managed by hand in database! Previously stated +information can be applied to instances as well, but instances can only have +admins. -### Runtime environment +Roles description: -*A runtime environment* defines the used programming language or tools -which are needed to process and evaluate a solution. Examples of a -runtime environment can be: +- Student -- Default role which is used for newly created accounts. Student can + join or leave public groups and submit solutions of assigned exercises. +- Supervisor -- Inherits all permissions from student role. Can manage groups to + which he/she belongs to. Supervisor can also view and change groups details, + manage assigned exercises, view students in group and their solutions for + assigned exercises. On top of that supervisor can create/delete groups too, + but only as subgroup of groups he/she belongs to. +- Superadmin -- Inherits all permissions from supervisor role. Most powerful + user in ReCodEx who should be able to do everything which is provided by + application. -- *Linux + GCC* -- *Linux + Mono* -- *Windows + .NET 4* -- *Bison + Yacc* -### Limits +## Writing job configuration -A correct *solution* of an *assignment* has to pass all specified tests (mostly -checks that it yields the correct output for various inputs) and typically must -also be effective in some sense. The Backend measures the time and memory -consumption of the solution while running. This consumption of resources can be -*limited* and the solution will receive fewer points if it exceeds the given -limits in some test cases defined by the *exercise*. +To run and evaluate an exercise the backend needs to know the steps how to do +that. This is different for each environment (operation system, programming +language, etc.), so each of the environments needs to have separate +configuration. -User management ---------------- +Backend works with a powerful, but quite low level description of simple +connected tasks written in YAML syntax. More about the syntax and general task +overview can be found on [separate +page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned +features was user friendly configuration editor, but due to tight deadline and +team composition it did not make it to the first release. However, writing +configuration in the basic format will be always available and allows users to +use the full expressive power of the system. -@todo: roles and their rights, adding/removing different users, how the -role of a specific user changes +This section walks through creation of job configuration for _hello world_ +exercise. The goal is to compile file _source.c_ and check if it prints `Hello +World!` to the standard output. This is the only test case, let's call it +**A**. -Instances and hierarchy of groups ---------------------------------- +The problem can be split into several tasks: -@todo: What is an instance, how to create one, what are the licenses and -how do they work. Why can the groups form hierarchies and what are the -benefits – what it means to be an admin of a group, hierarchy of roles -in the group hierarchy. +- compile _source.c_ into _helloworld_ with `/usr/bin/gcc` +- run _helloworld_ and save standard output into _out.txt_ +- fetch predefined output (suppose it is already uploaded to fileserver) with + hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_ +- compare _out.txt_ and _reference.txt_ by `/usr/bin/diff` -Exercises database ------------------- +The absolute path of tools can be obtained from system administrator. However, +`/usr/bin/gcc` is location, where the GCC binary is available almost everywhere, +so location of some tools can be (professionally) guessed. -@todo: How the exercises are stored, accessed, who can edit what +First, write header of the job to the configuration file. -### Creating a new exercise +```{.yml} +submission: + job-id: hello-word-job + hw-groups: + - group1 +``` -@todo Localized assignments, default settings +Basically it means, that the job _hello-world-job_ needs to be run on workers +that belong to the `group_1` hardware group . Reference files are downloaded +from the default location configured in API (such as +`http://localhost:9999/exercises`) if not stated explicitly otherwise. Job +execution log will not be saved to result archive. -### Runtime environments and hardware groups +Next the tasks have to be constructed under _tasks_ section. In this demo job, +every task depends only on previous one. The first task has input file +_source.c_ (if submitted by user) already available in working directory, so +just call the GCC. Compilation is run in sandbox as any other external program +and should have relaxed time and memory limits. In this scenario, worker +defaults are used. If compilation fails, the whole job is immediately terminated +(because the _fatal-failure_ bit is set). Because _bound-directories_ option in +sandbox limits section is mostly shared between all tasks, it can be set in +worker configuration instead of job configuration (suppose this for following +tasks). For configuration of workers please contact your administrator. -@todo read this later and see if it still makes sense +```{.yml} +- task-id: "compilation" + type: "initiation" + fatal-failure: true + cmd: + bin: "/usr/bin/gcc" + args: + - "source.c" + - "-o" + - "helloworld" + sandbox: + name: "isolate" + limits: + - hw-group-id: group1 + chdir: ${EVAL_DIR} + bound-directories: + - src: ${SOURCE_DIR} + dst: ${EVAL_DIR} + mode: RW +``` -ReCodEx is designed to utilize a rather diverse set of workers -- there can be -differences in many aspects, such as the actual hardware running the worker -(which impacts the results of measuring) or installed compilers, interpreters -and other tools needed for evaluation. To address these two examples in -particular, we assign runtime environments and hardware groups to exercises. +The compiled program is executed with time and memory limit set and the standard +output is redirected to a file. This task depends on _compilation_ task, because +the program cannot be executed without being compiled first. It is important to +mark this task with _execution_ type, so exceeded limits will be reported in +frontend. -The purpose of runtime environments is to specify which tools (and often also -operating system) are required to evaluate a solution of the exercise -- for -example, a C# programming exercise can be evaluated on a Linux worker running -Mono or a Windows worker with the .NET runtime. Such exercise would be assigned -two runtime environments, `Linux+Mono` and `Windows+.NET` (the environment names -are arbitrary strings configured by the administrator). +Time and memory limits set directly for a task have higher priority than worker +defaults. One important constraint is, that these limits cannot exceed limits +set by workers. Worker defaults are present as a safety measure so that a +malformed job configuration cannot block the worker forever. Worker default +limits should be reasonably high, like a gigabyte of memory and several hours of +execution time. For exact numbers please contact your administrator. -A hardware group is a set of workers that run on similar hardware (e.g. a -particular quad-core processor model and a SSD hard drive). Workers are assigned -to these groups by the administrator. If this is done correctly, performance -measurements of a submission should yield the same results. Thanks to this fact, -we can use the same resource limits on every worker in a hardware group. -However, limits can differ between runtime environments -- formally speaking, -limits are a function of three arguments: an assignment, a hardware group and a -runtime environment. +It is important to know that if the output of a program (both standard and +error) is redirected to a file, the sandbox disk quotas apply to that file, as +well as the files created directly by the program. In case the outputs are +ignored, they are redirected to `/dev/null`, which means there is no limit on +the output length (as long as the printing fits in the time limit). -### Reference solutions +```{.yml} +- task-id: "execution_1" + test-id: "A" + type: "execution" + dependencies: + - compilation + cmd: + bin: "helloworld" + sandbox: + name: "isolate" + stdout: ${EVAL_DIR}/out.txt + limits: + - hw-group-id: group1 + chdir: ${EVAL_DIR} + time: 0.5 + memory: 8192 +``` -@todo: how to add one, how to evaluate it +Fetch sample solution from file server. Base URL of file server is in the header +of the job configuration, so only the name of required file (its `sha1sum` in +our case) is necessary. -The task of determining appropriate resource limits for exercises is difficult -to do correctly. To aid exercise authors and group supervisors, ReCodEx supports -assigning reference solutions to exercises. Those are example programs that -should cover the main approaches to the implementation. For example, searching -for an integer in an ordered array can be done with a linear search, or better, -using a binary search. +```{.yml} +- task-id: "fetch_solution_1" + test-id: "A" + dependencies: + - execution + cmd: + bin: "fetch" + args: + - "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b" + - "${SOURCE_DIR}/reference.txt" +``` -Reference solutions can be evaluated on demand, using a selected hardware group. -The evaluation results are stored and can be used later to determine limits. In -our example problem, we could configure the limits so that the linear -search-based program doesn't finish in time on larger inputs, but a binary -search does. +Comparison of results is quite straightforward. It is important to set the task +type to _evaluation_, so that the return code is set to 0 if the program is +correct and 1 otherwise. We do not set our own limits, so the default limits are +used. -Note that separate reference solutions should be supplied for all supported -runtime environments. +```{.yml} +- task-id: "judge_1" + test-id: "A" + type: "evaluation" + dependencies: + - fetch_solution_1 + cmd: + bin: "/usr/bin/diff" + args: + - "out.txt" + - "reference.txt" + sandbox: + name: "isolate" + limits: + - hw-group-id: group1 + chdir: ${EVAL_DIR} +``` -### Exercise assignments -@todo: Creating instances of an exercise for a specific group of users, -capabilities of settings. Editing limits according to the reference -solution. -Evaluation process ------------------- +# The Backend -@todo: How the evaluation process works on the Frontend side. +The backend is the part which is hidden to the user and which has only +one purpose: evaluate user’s solutions of their assignments. -### Uploading files and file storage +@todo: describe the configuration inputs of the Backend -@todo: One by one upload endpoint. Explain different types of the -Uploaded files. +@todo: describe the outputs of the Backend -### Automatic detection of the runtime environment +@todo: describe how the backend receives the inputs and how it +communicates the results -@todo: Users must submit correctly named files – assuming the RTE from -the extensions. +## Components -REST API implementation ------------------------ +Whole backend is not just one service/component, it is quite complex system on its own. -@todo: What is the REST API, what are the basic principles – GET, POST, -Headers, JSON. +@todo: describe the inner parts of the Backend (and refer to the Wiki +for the technical description of the components) -### Authentication and authorization scopes +### Broker -@todo: How authentication works – signed JWT, headers, expiration, -refreshing. Token scopes usage. +@todo: gets stuff done, single point of failure and center point of ReCodEx universe -### HTTP requests handling +### Fileserver -@todo: Router and routes with specific HTTP methods, preflight, required -headers +@todo: stores particular data from frontend and backend, hashing, HTTP API -### HTTP responses format +### Worker -@todo: Describe the JSON structure convention of success and error -responses +@todo: describe a bit of internal structure in general -### Used technologies +@todo: describe how jobs are generally executed -@todo: PHP7 – how it is used for typehints, Nette framework – how it is -used for routing, Presenters actions endpoints, exceptions and -ErrorPresenter, Doctrine 2 – database abstraction, entities and -repositories + conventions, Communication over ZMQ – describe the -problem with the extension and how we reported it and how to treat it in -the future when the bug is solved. Relational database – we use MariaDB, -Doctine enables us to switch the engine to a different engine if needed +### Monitor -### Data model +@todo: not necessary component which can be omitted, proxy-like service -@todo: Describe the code-first approach using the Doctrine entities, how -the entities map onto the database schema (refer to the attached schemas -of entities and relational database models), describe the logical -grouping of entities and how they are related: +## Backend internal communication -- user + settings + logins + ACL -- instance + licenses + groups + group membership -- exercise + assignments + localized assignments + runtime - environments + hardware groups -- submission + solution + reference solution + solution evaluation -- comment threads + comments +@todo: internal backend communication, what communicates with what and why -### API endpoints +The Frontend +============ -@todo: Tell the user about the generated API reference and how the -Swagger UI can be used to access the API directly. +The frontend is the part which is visible to the user of ReCodEx and +which holds the state of the system – the user accounts, their roles in +the system, the database of exercises, the assignments of these +exercises to groups of users (i.e., students), and the solutions and +evaluations of them. -Web Application ---------------- +Frontend is split into three parts: -@todo: What is the purpose of the web application and how it interacts -with the REST API. +- the server-side REST API (“API”) which holds the business logic and + keeps the state of the system consistent -### Used technologies +- the relational database (“DB”) which persists the state of the + system -@todo: Briefly introduce the used technologies like React, Redux and the -build process. For further details refer to the GitHub wiki +- the client side application (“client”) which simplifies access to + the API for the common users -### How to use the application +The centerpiece of this architecture is the API. This component receives +requests from the users and from the Backend, validates them and +modifies the state of the system and persists this modified state in the +DB. -@todo: Describe the user documentation and the FAQ page. +We have created a web application which can communicate with the API +server and present the information received from the server to the user +in a convenient way. The client can be though any application, which can +send HTTP requests and receive the HTTP responses. Users can use general +applications like [cURL](https://github.com/curl/curl/), +[Postman](https://www.getpostman.com/), or create their own specific +client for ReCodEx API. -Backend-Frontend communication protocol -======================================= +Frontend capabilities +--------------------- -@todo: describe the exact methods and respective commands for the -communication +@todo: describe what the frontend is capable of and how it really works, +what are the limitations and how it can be extended -Initiation of a job evaluation ------------------------------- +Terminology +----------- -@todo: How does the Frontend initiate the evaluation and how the Backend -can accept it or decline it +This project was created for the needs of a university and this fact is +reflected into the terminology used throughout the Frontend. A list of +important terms’ definitions follows to make the meaning unambiguous. -Job processing progress monitoring ----------------------------------- +### User and user roles -When evaluating a job the worker sends progress messages on predefined points of -evaluation chain. The sending place can be on very beginning of the job, when -submit archive is downloaded or at the end of each simple task with its state -(completed, failed, skipped). These messages are sent to broker through existing -ZeroMQ connection. Detailed format of messages can be found on [communication -page](https://github.com/ReCodEx/wiki/wiki/Overall-architecture#commands-from-worker-to-broker). +*User* is a person who uses the application. User is granted access to +the application once he or she creates an account directly through the +API or the web application. There are several types of user accounts +depending on the set of permissions – a so called “role” – they have +been granted. Each user receives only the most basic set of permissions +after he or she creates an account and this role can be changed only by +the administrators of the service: -Broker only resends received progress messages to the monitor component via -ZeroMQ socket. The output message format is the same as the input format. +- *Student* is the most basic role. Student can become member of a + group and submit his solutions to his assignments. -Monitor parses received messages to JSON format, which is easy to work with in -JavaScript inside web application. All messages are cached (one queue per job) -and can be obtained multiple times through WebSocket communication channel. The -cache is cleared 5 minutes after receiving last message. +- *Supervisor* can be entitled to manage a group of students. + Supervisor can assign exercises to the students who are members of + his groups and review their solutions submitted to + these assignments. + +- *Super-admin* is a user with unlimited rights. This user can perform + any action in the system. + +There are two implicit changes of roles: -Publishing of the results -------------------------- +- Once a *student* is added to a group as its supervisor, his role is + upgraded to a *supervisor* role. -After job finish the worker packs results directory into single archive and -uploads it to the fileserver through HTTP protocol. The target URL is obtained -from API in headers on job initiation. Then "job done" notification request is -performed to API via broker. Special submissions (reference or asynchronous -submissions) are loaded immediately, other types are loaded on-demand on first -results request. +- Once a *supervisor* is removed from the lasts group where he is a + supervisor then his role is downgraded to a *student* role. -Loading results means fetching archive from fileserver, parsing the main YAML -file generated by worker and saving data to the database. Also, points are -assigned by score calculator. +These mechanisms do not prevent a single user being a supervisor of one +group and student of a different group as supervisors’ permissions are +superset of students’ permissions. +### Login -# User documentation +*Login* is a set of user’s credentials he must submit to verify he can +be allowed to access the system as a specific user. We distinguish two +types of logins: local and external. -@todo: Describe different scenarios of the usage of the Web App +- *Local login* is user’s email address and a password he chooses + during registration. -@todo: Describe the requirements of running the web application (modern web browser, enabled CSS, JavaScript, Cookies & Local storage) +- *External login* is a mapping of a user profile to an account of + some authentication service (e.g., [CAS](https://ldap1.cuni.cz/)). -## Terminology +### Instance -@todo: Describe the terminology: Instance, User, Group, Student, -Supervisor, Admin +*An instance* of ReCodEx is in fact just a set of groups and user +accounts. An instance should correspond to a real entity as a +university, a high-school, an IT company or an HR agency. This approach +enables the system to be shared by multiple independent organizations +without interfering with each other. -## General basics +Usage of the system by the users of an instance can be limited by +possessing a valid license. It is up to the administrators of the system +to determine the conditions under which they will assign licenses to the +instances. -@todo: actions which are available for all users +### Group -@todo: how to solve problems with ReCodEx, first supervisors, then administrators, etc... +*Group* corresponds to a school class or some other unit which gathers +users who will be assigned the same set exercises. Each group can have +multiple supervisors who can manage the students and the list of +assignments. -### First steps in ReCodEx +Groups can form a tree hierarchy of arbitrary depth. This is inspired by the +hierarchy of school classes belonging to the same subject over several school +years. For example, there can be a top level group for a programming class that +contains subgroups for every school year. These groups can then by divided into +actual student groups with respect to lab attendance. Supervisors can create +subgroups of their groups and further manage these subgroups. -You can create an account if you click on the “*Create account*” menu -item in the left sidebar. You can choose between two types of -registration methods – by creating a local account with a specific -password, or pairing your new account with an existing CAS UK account. +### Exercise -If you decide a new “*local*” account using the “*Create ReCodEx -account*” form, you will have to provide your details and choose a -password for your account. You will later sign in using your email -address as your username and the password you select. +*An exercise* consists of textual assignment of a task and a definition +of how a solution to this exercise should be processed and evaluated in +a specific runtime environment (i.e., how to compile a submitted source +code and how to test the correctness of the program). It is a template +which can be instantiated as an *assignment* by a supervisor of a group. -If you decide to use the CAS UK, then we will verify your credentials -and access your name and email stored in the system and create your -account based on this information. You can change your personal -information or email later on the “*Settings*” page. +### Assignment -When creating your account both ways, you must select an instance your -account will belong to by default. The instance you will select will be -most likely your university or other organization you are a member of. +An assignment is an instance of an *exercise* assigned to a specific +*group*. An assignment can modify the text of the task assignment and it +has some additional information which is specific to the group (e.g., a +deadline, the number of points gained for a correct solution, additional +hints for the students in the assignment). The text of the assignment +can be edited and supervisors can translate the assignment into another +language. -To log in, go to the homepage of ReCodEx and in the left sidebar choose -the menu item “*Sign in*”. Then you must enter your credentials into one -of the two forms – if you selected a password during registration, then -you should sign with your email and password in the first form called -“*Sign into ReCodEx*”. If you registered using the Charles University -Authentication Service (CAS), you should put your student’s number and -your CAS password into the second form called “Sign into ReCodEx using -CAS UK”. +### Solution -There are several options you can edit in your user account: +*A solution* is a set of files which a user submits to a given +*assignment*. -- changing your personal information (i.e., name) -- changing your credentials (email and password) -- updating your preferences (e.g., source code viewer/editor settings, - default language) +### Submission -You can access the settings page through the “*Settings*” button right -under your name in the left sidebar. +*A submission* corresponds to a *solution* being evaluated by the +Backend. A single *solution* can be submitted repeatedly (e.g., when the +Backend encounters an error or when the supervisor changes the assignment). -If you don’t use ReCodEx for a whole day, you will be logged out -automatically. However, we recommend you sign out of the application -after you finished your interaction with it. The logout button is placed -in the top section of the left sidebar right under your name. You will -have to expand the sidebar with a button next to the “*ReCodEx*” title -(shown in the picture below). +### Evaluation -### Forgotten password +*An evaluation* is the processed report received from the Backend after +a *submission* is processed. Evaluation contains points given to the +user based on the quality of his solution measured by the Backend and +the settings of the assignment. Supervisors can review the evaluation +and add bonus points (both positive and negative) if the student +deserves some. -If you can’t remember your password and you don’t use CAS UK -authentication, then you can reset your password. You will find a link -saying “*You cannot remember what your password was? Reset your -password.*” under the sign in form. After you click on this link, you -will be asked to submit your email address. An email with a link -containing a special token will be sent to the address you fill in. We -make sure that the person who requested password resetting is really -you. When you click on the link (or you copy & paste it into your web -browser) you will be able to select a new password for your account. The -token is valid only for a couple of minutes, so do not forget to reset -the password as soon as possible, or you will have to request a new link -with a valid token. +### Runtime environment -If you sign in through CAS UK, then please follow the instructions -provided by the administrators of the service described on their -website. +*A runtime environment* defines the used programming language or tools +which are needed to process and evaluate a solution. Examples of a +runtime environment can be: +- *Linux + GCC* +- *Linux + Mono* +- *Windows + .NET 4* +- *Bison + Yacc* -## Student +### Limits -@todo: describe what it means to be a “student” and what are the -student’s rights +A correct *solution* of an *assignment* has to pass all specified tests (mostly +checks that it yields the correct output for various inputs) and typically must +also be effective in some sense. The Backend measures the time and memory +consumption of the solution while running. This consumption of resources can be +*limited* and the solution will receive fewer points if it exceeds the given +limits in some test cases defined by the *exercise*. -### Join group and start solving assignments +User management +--------------- -@todo: How to join a specific group +@todo: roles and their rights, adding/removing different users, how the +role of a specific user changes -@todo: Where can the user see groups description and details, what -information is available. +Instances and hierarchy of groups +--------------------------------- -@todo: Where the student can find the list of the assignment he is -expected to solve, what is the first and second deadline. +@todo: What is an instance, how to create one, what are the licenses and +how do they work. Why can the groups form hierarchies and what are the +benefits – what it means to be an admin of a group, hierarchy of roles +in the group hierarchy. -@todo: How does a student submit his solution through the web app +Exercises database +------------------ -@todo: When the results are ready and what the results mean and what to -do about them, when the user is convinced, that his solution is correct -although the results say different +@todo: How the exercises are stored, accessed, who can edit what -@todo: Describe the comments thread behavior (public/private comments), -who else can see the comments, how notifications work (*not implemented -yet*!). +### Creating a new exercise +@todo Localized assignments, default settings -## Group supervisor +### Runtime environments and hardware groups -@todo: describe what it means to be a “supervisor” of a group and what -are the supervisors rights +@todo read this later and see if it still makes sense -### Create groups and manage them +ReCodEx is designed to utilize a rather diverse set of workers -- there can be +differences in many aspects, such as the actual hardware running the worker +(which impacts the results of measuring) or installed compilers, interpreters +and other tools needed for evaluation. To address these two examples in +particular, we assign runtime environments and hardware groups to exercises. -@todo: How does a user become a supervisor of a group? +The purpose of runtime environments is to specify which tools (and often also +operating system) are required to evaluate a solution of the exercise -- for +example, a C# programming exercise can be evaluated on a Linux worker running +Mono or a Windows worker with the .NET runtime. Such exercise would be assigned +two runtime environments, `Linux+Mono` and `Windows+.NET` (the environment names +are arbitrary strings configured by the administrator). -@todo: How to add a specific student to a given group +A hardware group is a set of workers that run on similar hardware (e.g. a +particular quad-core processor model and a SSD hard drive). Workers are assigned +to these groups by the administrator. If this is done correctly, performance +measurements of a submission should yield the same results. Thanks to this fact, +we can use the same resource limits on every worker in a hardware group. +However, limits can differ between runtime environments -- formally speaking, +limits are a function of three arguments: an assignment, a hardware group and a +runtime environment. -### Assigning exercises +### Reference solutions -@todo: Describe how to access the database of the exercises and what are -the possibilities of assignment setup – availability, deadlines, points, -score configuration, limits +@todo: how to add one, how to evaluate it -@todo: How can I assign some exercises only to some students of the group? Describe how to achieve this using subgroups +The task of determining appropriate resource limits for exercises is difficult +to do correctly. To aid exercise authors and group supervisors, ReCodEx supports +assigning reference solutions to exercises. Those are example programs that +should cover the main approaches to the implementation. For example, searching +for an integer in an ordered array can be done with a linear search, or better, +using a binary search. -### Students' solutions management +Reference solutions can be evaluated on demand, using a selected hardware group. +The evaluation results are stored and can be used later to determine limits. In +our example problem, we could configure the limits so that the linear +search-based program doesn't finish in time on larger inputs, but a binary +search does. -@todo Describe where all the students’ solutions for a given assignment -can be found, where to look for all solutions of a given student, how to -see results of a specific student’s solution’s evaluation result. +Note that separate reference solutions should be supplied for all supported +runtime environments. -@todo Can I assign points to my students’ solutions manually instead of depending on automatic scoring? If and how to change the score of a solution – assignment -settings, setting points, bonus points, accepting a solution (*not -implemented yet!*). Describe how the student and supervisor will still -be able to see the percentage received from the automatic scoring, but -the awarded points will be overridden. +### Exercise assignments -@todo: Describe the comments thread behavior (public/private comments), -who else can see the comments -- same as from the student perspective +@todo: Creating instances of an exercise for a specific group of users, +capabilities of settings. Editing limits according to the reference +solution. -### Creating exercises +Evaluation process +------------------ -@todo: how to create exercise, what has to be provided during creation, who can create exercises +@todo: How the evaluation process works on the Frontend side. -@todo: Describe the form and explain the concept of reference solutions. -How to evaluate the reference solutions for the exercise right now (to -get the up-to-date information). +### Uploading files and file storage + +@todo: One by one upload endpoint. Explain different types of the +Uploaded files. +### Automatic detection of the runtime environment -## Group administrator +@todo: Users must submit correctly named files – assuming the RTE from +the extensions. -@todo: who is this? +REST API implementation +----------------------- -### Creating subgroups and managing supervisors +@todo: What is the REST API, what are the basic principles – GET, POST, +Headers, JSON. -@todo: What it means to create a subgroup and how to do it. +### Authentication and authorization scopes -@todo: who can add another supervisor, what would be the rights of the -second supervisor +@todo: How authentication works – signed JWT, headers, expiration, +refreshing. Token scopes usage. +### HTTP requests handling -## Superadministrator +@todo: Router and routes with specific HTTP methods, preflight, required +headers -Superadmin is user with the most priviledges and as such superadmin should be -quite unique role. Ideally there should be only one of this kind, used with -special caution and adequate security. With this stated it is obvious that -superadmin can perform any action the API is capable of. +### HTTP responses format -### Users management +@todo: Describe the JSON structure convention of success and error +responses -There are only few roles to which users can belong in ReCodEx. Basically there -are only three: _student_, _supervisor_, and _superadmin_. Base role is student -which is assigned to every registered user. Roles are stored in database -alongside other information about user. One user always has only one role at the -time. At first startup of ReCodEx administrator should create his account and -then change role in database by hand. After that manual intervention into -database should never be needed. +### Used technologies -There is a little catch in groups and instances management. Groups can have -admins and supervisors. This setting is valid only per one particular group and -has to be separated from basic role system. This implies that supervisor in one -group can be student in another and simultaneously have global supervisor role. -Changing role from student to supervisor and back is done automatically by -application and should not be managed by hand in database! Previously stated -information can be applied to instances as well, but instances can only have -admins. +@todo: PHP7 – how it is used for typehints, Nette framework – how it is +used for routing, Presenters actions endpoints, exceptions and +ErrorPresenter, Doctrine 2 – database abstraction, entities and +repositories + conventions, Communication over ZMQ – describe the +problem with the extension and how we reported it and how to treat it in +the future when the bug is solved. Relational database – we use MariaDB, +Doctine enables us to switch the engine to a different engine if needed -Roles description: +### Data model -- Student -- Default role which is used for newly created accounts. Student can - join or leave public groups and submit solutions of assigned exercises. -- Supervisor -- Inherits all permissions from student role. Can manage groups to - which he/she belongs to. Supervisor can also view and change groups details, - manage assigned exercises, view students in group and their solutions for - assigned exercises. On top of that supervisor can create/delete groups too, - but only as subgroup of groups he/she belongs to. -- Superadmin -- Inherits all permissions from supervisor role. Most powerful - user in ReCodEx who should be able to do everything which is provided by - application. +@todo: Describe the code-first approach using the Doctrine entities, how +the entities map onto the database schema (refer to the attached schemas +of entities and relational database models), describe the logical +grouping of entities and how they are related: +- user + settings + logins + ACL +- instance + licenses + groups + group membership +- exercise + assignments + localized assignments + runtime + environments + hardware groups +- submission + solution + reference solution + solution evaluation +- comment threads + comments -## Writing job configuration +### API endpoints -To run and evaluate an exercise the backend needs to know the steps how to do -that. This is different for each environment (operation system, programming -language, etc.), so each of the environments needs to have separate -configuration. +@todo: Tell the user about the generated API reference and how the +Swagger UI can be used to access the API directly. -Backend works with a powerful, but quite low level description of simple -connected tasks written in YAML syntax. More about the syntax and general task -overview can be found on [separate -page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned -features was user friendly configuration editor, but due to tight deadline and -team composition it did not make it to the first release. However, writing -configuration in the basic format will be always available and allows users to -use the full expressive power of the system. +Web Application +--------------- -This section walks through creation of job configuration for _hello world_ -exercise. The goal is to compile file _source.c_ and check if it prints `Hello -World!` to the standard output. This is the only test case, let's call it -**A**. +@todo: What is the purpose of the web application and how it interacts +with the REST API. -The problem can be split into several tasks: +### Used technologies -- compile _source.c_ into _helloworld_ with `/usr/bin/gcc` -- run _helloworld_ and save standard output into _out.txt_ -- fetch predefined output (suppose it is already uploaded to fileserver) with - hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_ -- compare _out.txt_ and _reference.txt_ by `/usr/bin/diff` +@todo: Briefly introduce the used technologies like React, Redux and the +build process. For further details refer to the GitHub wiki -The absolute path of tools can be obtained from system administrator. However, -`/usr/bin/gcc` is location, where the GCC binary is available almost everywhere, -so location of some tools can be (professionally) guessed. +### How to use the application -First, write header of the job to the configuration file. +@todo: Describe the user documentation and the FAQ page. -```{.yml} -submission: - job-id: hello-word-job - hw-groups: - - group1 -``` +Backend-Frontend communication protocol +======================================= -Basically it means, that the job _hello-world-job_ needs to be run on workers -that belong to the `group_1` hardware group . Reference files are downloaded -from the default location configured in API (such as -`http://localhost:9999/exercises`) if not stated explicitly otherwise. Job -execution log will not be saved to result archive. +@todo: describe the exact methods and respective commands for the +communication -Next the tasks have to be constructed under _tasks_ section. In this demo job, -every task depends only on previous one. The first task has input file -_source.c_ (if submitted by user) already available in working directory, so -just call the GCC. Compilation is run in sandbox as any other external program -and should have relaxed time and memory limits. In this scenario, worker -defaults are used. If compilation fails, the whole job is immediately terminated -(because the _fatal-failure_ bit is set). Because _bound-directories_ option in -sandbox limits section is mostly shared between all tasks, it can be set in -worker configuration instead of job configuration (suppose this for following -tasks). For configuration of workers please contact your administrator. +Initiation of a job evaluation +------------------------------ -```{.yml} -- task-id: "compilation" - type: "initiation" - fatal-failure: true - cmd: - bin: "/usr/bin/gcc" - args: - - "source.c" - - "-o" - - "helloworld" - sandbox: - name: "isolate" - limits: - - hw-group-id: group1 - chdir: ${EVAL_DIR} - bound-directories: - - src: ${SOURCE_DIR} - dst: ${EVAL_DIR} - mode: RW -``` +@todo: How does the Frontend initiate the evaluation and how the Backend +can accept it or decline it -The compiled program is executed with time and memory limit set and the standard -output is redirected to a file. This task depends on _compilation_ task, because -the program cannot be executed without being compiled first. It is important to -mark this task with _execution_ type, so exceeded limits will be reported in -frontend. +Job processing progress monitoring +---------------------------------- -Time and memory limits set directly for a task have higher priority than worker -defaults. One important constraint is, that these limits cannot exceed limits -set by workers. Worker defaults are present as a safety measure so that a -malformed job configuration cannot block the worker forever. Worker default -limits should be reasonably high, like a gigabyte of memory and several hours of -execution time. For exact numbers please contact your administrator. +When evaluating a job the worker sends progress messages on predefined points of +evaluation chain. The sending place can be on very beginning of the job, when +submit archive is downloaded or at the end of each simple task with its state +(completed, failed, skipped). These messages are sent to broker through existing +ZeroMQ connection. Detailed format of messages can be found on [communication +page](https://github.com/ReCodEx/wiki/wiki/Overall-architecture#commands-from-worker-to-broker). -It is important to know that if the output of a program (both standard and -error) is redirected to a file, the sandbox disk quotas apply to that file, as -well as the files created directly by the program. In case the outputs are -ignored, they are redirected to `/dev/null`, which means there is no limit on -the output length (as long as the printing fits in the time limit). +Broker only resends received progress messages to the monitor component via +ZeroMQ socket. The output message format is the same as the input format. -```{.yml} -- task-id: "execution_1" - test-id: "A" - type: "execution" - dependencies: - - compilation - cmd: - bin: "helloworld" - sandbox: - name: "isolate" - stdout: ${EVAL_DIR}/out.txt - limits: - - hw-group-id: group1 - chdir: ${EVAL_DIR} - time: 0.5 - memory: 8192 -``` +Monitor parses received messages to JSON format, which is easy to work with in +JavaScript inside web application. All messages are cached (one queue per job) +and can be obtained multiple times through WebSocket communication channel. The +cache is cleared 5 minutes after receiving last message. -Fetch sample solution from file server. Base URL of file server is in the header -of the job configuration, so only the name of required file (its `sha1sum` in -our case) is necessary. +Publishing of the results +------------------------- -```{.yml} -- task-id: "fetch_solution_1" - test-id: "A" - dependencies: - - execution - cmd: - bin: "fetch" - args: - - "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b" - - "${SOURCE_DIR}/reference.txt" -``` +After job finish the worker packs results directory into single archive and +uploads it to the fileserver through HTTP protocol. The target URL is obtained +from API in headers on job initiation. Then "job done" notification request is +performed to API via broker. Special submissions (reference or asynchronous +submissions) are loaded immediately, other types are loaded on-demand on first +results request. + +Loading results means fetching archive from fileserver, parsing the main YAML +file generated by worker and saving data to the database. Also, points are +assigned by score calculator. -Comparison of results is quite straightforward. It is important to set the task -type to _evaluation_, so that the return code is set to 0 if the program is -correct and 1 otherwise. We do not set our own limits, so the default limits are -used. -```{.yml} -- task-id: "judge_1" - test-id: "A" - type: "evaluation" - dependencies: - - fetch_solution_1 - cmd: - bin: "/usr/bin/diff" - args: - - "out.txt" - - "reference.txt" - sandbox: - name: "isolate" - limits: - - hw-group-id: group1 - chdir: ${EVAL_DIR} -``` From 273161711f16fe06aef189f159482eb8c8f4f8c9 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Thu, 12 Jan 2017 13:54:24 +0100 Subject: [PATCH 3/7] Intro structure updated --- Rewritten-docs.md | 153 +++++++++++++++++++++------------------------- 1 file changed, 70 insertions(+), 83 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index ef77ff6..4c4e2ed 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -115,7 +115,21 @@ currently used at the university (CodEx), so its flaws and weaknesses can be addressed. Furthermore, many teachers desire to use and test the new system and they are willing to consult ideas or problems during development with us. -### Intended usage +## Current system + +Currently used grading solution at the Faculty of Mathematics and Physics of +the Charles University in Prague which was implemented in 2006 by a group +of students. It is called [CodEx -- The Code Examiner](http://codex.ms.mff.cuni.cz/project/) +and it has been used with some improvements since then. The original plan was +to use the system only for basic programming courses, but there was a demand +for adapting it for many different subjects. + +CodEx is based on dynamic analysis. It features a web-based interface, where +supervisors can assign exercises to their students and the students have a time +window to submit their solutions. Each solution is compiled and run in sandbox +(MO-Eval). The metrics which are checked are: correctness of the output, time +and memory limits. It supports programs written in C, C++, C#, Java, Pascal, +Python and Haskell. The whole system is intended to help both teachers (supervisors) and students. To achieve this, it is crucial to keep in mind the typical usage scenarios of @@ -123,26 +137,7 @@ the system and to try to make these tasks as simple as possible. The system has a database of users. Each user is assigned a role, which corresponds to his/her privileges. There are user groups reflecting the -structure of lectured courses. Groups can be hierarchically ordered to reflect -additional metadata such as the academic year. For example, a reasonable group -hierarchy could look like this: - -``` -Summer term 2016 -|-- Language C# and .NET platform -|   |-- Labs Monday 10:30 -|   `-- Labs Thursday 9:00 -|-- Programming I -|   |-- Labs Monday 14:00 - ... -``` - -In this example, students are members of the leaf groups and the higher level -nodes are just for keeping related groups together. The structure can be -modified and altered to fit specific needs of the university or any other -organization, even a flat structure is possible. One user can be a member of -multiple groups and have a different role in each of them (a student can attend -labs for several courses while also teaching one). +structure of lectured courses. A database of exercises (algorithmic problems) is another part of the project. Each exercise consists of a text describing the problem in multiple language @@ -155,12 +150,50 @@ points, a configuration for calculating the score, a maximum number of submissions, and a list of supported runtime environments (e.g. programming languages) including specific time and memory limits for each one. -Typical use cases for supported user roles are illustrated on following UML -diagram: +Typical use cases for supported user roles are following: + +- **student** + - join a group + - get assignments in group + - submit solution to assignment + - view solution results +- **supervisor** + - create exercise + - assign exercise to group, modify assignment + - view all results in group + - alter automatic solution grading +- **administrator** + - create groups + - alter user privileges + - check system logs, upgrades and other management + +Current system is old, but robust. There were no major security incidents +during its production usage. However, from today's perspective there are +several drawbacks. The main ones are: -![System use case diagram](https://github.com/ReCodEx/wiki/raw/master/images/System_use_case.png) +- **web interface** -- The web interface is simple and fully functional. But + rapid development in web technologies opens new horizons of how web interface + can be made. +- **web API** -- CodEx offers a very limited XML API based on outdated + technologies that is not sufficient for users who would like to create custom + interfaces such as a command line tool or mobile application. +- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system + calls and blocking the bad ones. This can be easily done for single-threaded + applications, but proves difficult with multi-threaded ones. In present day, + parallelism is a very important area of computing, so there is requirement to + test multi-threaded applications too. +- **instances** -- Different ways of CodEx usage scenarios requires separate + instances (Programming I and II, Java, C#, etc.). This configuration is not + user friendly (students have to register in each instance separately) and + burdens administrators with unnecessary work. CodEx architecture does not + allow sharing hardware between instances, which results in an inefficient use + of hardware for evaluation. +- **task extensibility** -- There is a need to test and evaluate complicated + programs for classes such as Parallel programming or Compiler principles, + which have a more difficult evaluation chain than simple + compilation/execution/evaluation provided by CodEx. -#### Exercise evaluation chain +### Exercise evaluation chain The most important part of the system is evaluation of solutions submitted by students. Concepts of consecutive steps from source code to final results @@ -203,27 +236,23 @@ from administrators and supervisors. The ideas were gathered mostly from our personal experience with the system and from meetings with faculty staff involved with the current system. -For clear arrangement all the requirements and wishes are presented grouped by -categories. +In general, CodEx features should be preserved, so only differences are +presented here. For clear arrangement all the requirements and wishes are +presented grouped by categories. ### System features System features represents directly accessible functionality to users of the -system. They describe the evaluation system in general and also university +system. They describe the evaluation system in general and also university addons (mostly administrative features). #### End user requirements -- users have their own accounts in the system -- system users can be members of multiple groups (reflecting courses or labs) +- group hierarchy -- @todo: copy text from +[here](https://github.com/ReCodEx/wiki/wiki/Rewritten-docs/87e4bcd39a4fca3eadbb4748e9a3b6ced2bd7150#intended-usage) - there is a database of exercises; teachers can create exercises including textual description, sample inputs and correct reference outputs (for example "sum all numbers from given file and write the result to the standard output") -- there is a list of assigned exercises in each group and an interface to submit - a solution; teachers can assign an existing exercise to their class with some - specific properties set (deadlines, etc.) -- users can see a list of submitted solutions for each assignment with - corresponding results - teachers can specify way of computation grading points which will be awarded to the students depending on the quality of his/her solution for each assignment extra @@ -237,7 +266,7 @@ addons (mostly administrative features). - localization of all texts (UI and exercises) - Markdown support for creating exercise texts - tagging exercises in database and search by these tags -- comments, comments, comments (exercises, tests, solutions, ...) +- comments of exercises, tests and solutions - plagiarism detection #### Administrative requirements @@ -263,12 +292,12 @@ addons (mostly administrative features). another tool and perform additional tests - use of modern technologies with state-of-the-art compilers -### Technical details +### Nonfunctional requirements -Technical details are requirements of technical character with no direct mapping -to visible parts of system. In ideal word, users should not know about these if -they work properly, but would be at least annoyed if these requirements were not -met. Most notably they are these ones: +Nonfunctional requirements are requirements of technical character with no +direct mapping to visible parts of system. In ideal word, users should not know +about these if they work properly, but would be at least annoyed if these +requirements were not met. Most notably they are these ones: - user interface of the system accessible on users' computers without installation of any kind of additional software @@ -300,48 +329,6 @@ This is not a complete list of available evaluators, but only a few projects which are used these days and can be an inspiration for our project. Each project from the list has a brief description and some key features mentioned. -### CodEx - -Currently used grading solution at the Faculty of Mathematics and Physics of -the Charles University in Prague which was implemented in 2006 by a group -of students. It is called [CodEx -- The Code Examiner](http://codex.ms.mff.cuni.cz/project/) -and it has been used with some improvements since then. The original plan was -to use the system only for basic programming courses, but there was a demand -for adapting it for many different subjects. - -CodEx is based on dynamic analysis. It features a web-based interface, where -supervisors can assign exercises to their students and the students have a time -window to submit their solutions. Each solution is compiled and run in sandbox -(MO-Eval). The metrics which are checked are: correctness of the output, time -and memory limits. It supports programs written in C, C++, C#, Java, Pascal, -Python and Haskell. - -Current system is old, but robust. There were no major security incidents -during its production usage. However, from today's perspective there are -several drawbacks. The main ones are: - -- **web interface** -- The web interface is simple and fully functional. But - rapid development in web technologies opens new horizons of how web interface - can be made. -- **web API** -- CodEx offers a very limited XML API based on outdated - technologies that is not sufficient for users who would like to create custom - interfaces such as a command line tool or mobile application. -- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system - calls and blocking the bad ones. This can be easily done for single-threaded - applications, but proves difficult with multi-threaded ones. In present day, - parallelism is a very important area of computing, so there is requirement to - test multi-threaded applications too. -- **instances** -- Different ways of CodEx usage scenarios requires separate - instances (Programming I and II, Java, C#, etc.). This configuration is not - user friendly (students have to register in each instance separately) and - burdens administrators with unnecessary work. CodEx architecture does not - allow sharing hardware between instances, which results in an inefficient use - of hardware for evaluation. -- **task extensibility** -- There is a need to test and evaluate complicated - programs for classes such as Parallel programming or Compiler principles, - which have a more difficult evaluation chain than simple - compilation/execution/evaluation provided by CodEx. - ### Progtest [Progtest](https://progtest.fit.cvut.cz/) is private project of [FIT From b5e2750e15ab495d33507806ca31c0252afb49b2 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Thu, 12 Jan 2017 15:18:14 +0100 Subject: [PATCH 4/7] Structure --- Rewritten-docs.md | 110 +++++++++++++++++++++++++--------------------- 1 file changed, 59 insertions(+), 51 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 9e809cd..70ed342 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -85,10 +85,10 @@ outputs ones; provides good real world experience, but requires extensive security measures). This project focuses on the machine-controlled part of source code evaluation. -First, general concepts of grading systems are observed, new requirements are -specified and project with similar functionality are examined. Also, problems of -the software previously used at Charles University in Prague are briefly -discussed. With acquired knowledge from such projects in production, we set up +First, general concepts of grading systems are observed and problems of the +software previously used at Charles University in Prague are briefly discussed. +Then new requirements are specified and projects with similar functionality are +examined. With acquired knowledge from such projects in production, we set up goals for the new evaluation system, designed the architecture and implemented a fully operational solution based on dynamic evaluation. The system is now ready for production testing at the university. @@ -110,7 +110,10 @@ consists of following basic steps: 4. compare program outputs with predefined values 5. award the code with a numeric score -The project has a great starting point -- there is an old grading system +The whole system is intended to help both teachers (supervisors) and students. +To achieve this, it is crucial to keep in mind the typical usage scenarios of +the system and to try to make these tasks as simple as possible. To fulfil this +task, the project has a great starting point -- there is an old grading system currently used at the university (CodEx), so its flaws and weaknesses can be addressed. Furthermore, many teachers desire to use and test the new system and they are willing to consult ideas or problems during development with us. @@ -131,10 +134,6 @@ window to submit their solutions. Each solution is compiled and run in sandbox and memory limits. It supports programs written in C, C++, C#, Java, Pascal, Python and Haskell. -The whole system is intended to help both teachers (supervisors) and students. -To achieve this, it is crucial to keep in mind the typical usage scenarios of -the system and to try to make these tasks as simple as possible. - The system has a database of users. Each user is assigned a role, which corresponds to his/her privileges. There are user groups reflecting the structure of lectured courses. @@ -155,18 +154,58 @@ Typical use cases for supported user roles are following: - **student** - join a group - get assignments in group - - submit solution to assignment - - view solution results + - submit solution to assignment -- upload one source file and trigger + evaluation process + - view solution results -- which parts succeeded and failed, total number of + acquired points, bonus points - **supervisor** - - create exercise - - assign exercise to group, modify assignment + - create exercise -- create description text and evaluation configuration + (for each programming environment), upload testing inputs and outputs + - assign exercise to group -- choose exercise and set deadlines, number of + allowed submissions, weights of all testing cases and amount of points for + correct solutions + - modify assignment - view all results in group - - alter automatic solution grading + - check automatic solution grading -- view submitted source and optionally + set bonus points - **administrator** - create groups - - alter user privileges + - alter user privileges -- make supervisor accounts - check system logs, upgrades and other management +### Exercise evaluation chain + +The most important part of the system is evaluation of solutions submitted by +students. Concepts of consecutive steps from source code to final results +is described in more detail below to give readers solid overview of what have to +happen during evaluation process. + +First thing users have to do is to submit their solutions through web user +interface. The system checks assignment invariants (deadlines, count of +submissions, ...) and stores submitted file. The runtime environment is +automatically detected based on input file and a suitable evaluation +configuration variant is chosen (one exercise can have multiple variants, for +example C and Java languages). This exercise configuration is then used for +taking care of evaluation process. + +There is a pool of uniform worker engines dedicated to evaluation jobs. Incoming +jobs are kept in a queue until a free worker picks them. Worker is capable of +sequential evaluation of jobs, one at a time. + +The worker obtains the solution and its evaluation configuration, parses it and +starts executing the contained instructions. It is crucial to keep the worker +computer secure and stable, so a sandboxed environment is used for dealing with +unknown source code. When the execution is finished, results are saved and the +submitter is notified. + +The output of the worker contains data about the evaluation, such as time and +memory spent on running the program for each test input and whether its output +was correct. The system then calculates a numeric score from this data, which is +presented to the student. If the solution is wrong (incorrect output, uses too +much memory,..), error messages are also displayed to the submitter. + +### Weaknesses + Current system is old, but robust. There were no major security incidents during its production usage. However, from today's perspective there are several drawbacks. The main ones are: @@ -193,37 +232,6 @@ several drawbacks. The main ones are: which have a more difficult evaluation chain than simple compilation/execution/evaluation provided by CodEx. -### Exercise evaluation chain - -The most important part of the system is evaluation of solutions submitted by -students. Concepts of consecutive steps from source code to final results -is described in more detail below to give readers solid overview of what have to -happen during evaluation process. - -First thing users have to do is to submit their solutions through some user -interface. Then, the system checks assignment invariants (deadlines, count of -submissions, ...) and stores submitted files. The runtime environment is -automatically detected based on input files and a suitable evaluation -configuration variant is chosen (one exercise can have multiple variants, for -example C and Java languages). This exercise configuration is then used for -taking care of evaluation process. - -There is a pool of worker computers dedicated to evaluation jobs. Each one of -them can support different environments and programming languages to allow -testing programs for as many platforms as possible. Incoming jobs are scheduled -to a worker that is capable of running the job. - -The worker obtains the solution and its evaluation configuration, parses it and -starts executing the contained instructions. It is crucial to keep the worker -computer secure and stable, so a sandboxed environment is used for dealing with -unknown source code. When the execution is finished, results are saved and the -submitter is notified. - -The output of the worker contains data about the evaluation, such as time and -memory spent on running the program for each test input and whether its output -was correct. The system then calculates a numeric score from this data, which is -presented to the student. If the solution is wrong (incorrect output, uses too -much memory,..), error messages are also displayed to the submitter. ## Requirements @@ -292,9 +300,9 @@ addons (mostly administrative features). another tool and perform additional tests - use of modern technologies with state-of-the-art compilers -### Nonfunctional requirements +### Non-functional requirements -Nonfunctional requirements are requirements of technical character with no +Non-functional requirements are requirements of technical character with no direct mapping to visible parts of system. In ideal word, users should not know about these if they work properly, but would be at least annoyed if these requirements were not met. Most notably they are these ones: @@ -317,14 +325,14 @@ extendable, so everyone can develop their own feature. This also means that widely used programming languages and techniques should be used, so users can quickly understand the code and make changes. + +## Related work + To find out the current state in the field of automatic grading systems we did a short market survey on the field of automatic grading systems at universities, programming contests, and possibly other places where similar tools are available. - -## Related work - This is not a complete list of available evaluators, but only a few projects which are used these days and can be an inspiration for our project. Each project from the list has a brief description and some key features mentioned. From 74a14d22a4edb690e9283cf469abe9a5f431f523 Mon Sep 17 00:00:00 2001 From: Teyras Date: Thu, 12 Jan 2017 15:44:01 +0100 Subject: [PATCH 5/7] reworked requirements section --- Rewritten-docs.md | 134 ++++++++++++++++++++++++++++------------------ 1 file changed, 82 insertions(+), 52 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 70ed342..51a2737 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -256,42 +256,69 @@ addons (mostly administrative features). #### End user requirements -- group hierarchy -- @todo: copy text from -[here](https://github.com/ReCodEx/wiki/wiki/Rewritten-docs/87e4bcd39a4fca3eadbb4748e9a3b6ced2bd7150#intended-usage) -- there is a database of exercises; teachers can create exercises including - textual description, sample inputs and correct reference outputs (for example - "sum all numbers from given file and write the result to the standard output") -- teachers can specify way of computation grading points which will be awarded - to the students depending on the quality of his/her solution for each - assignment extra -- teachers can view detailed data about their students (users of a their groups) - including all submitted solutions; also, each of the solution can be manually - reviewed, commented and assigned additional points (positive or negative) -- one particular solution can be marked as accepted (used for grading this - assignment); otherwise, the solution with most points is used -- teacher can edit student solution and privately resubmit it; optionally saving - all results (including temporary ones) -- localization of all texts (UI and exercises) -- Markdown support for creating exercise texts -- tagging exercises in database and search by these tags -- comments of exercises, tests and solutions -- plagiarism detection +- _group hierarchy_ -- creating an arbitrarily nested tree structure should be + supported to allow keeping related groups together, such as in the example + below. A group hierarchy also allows archiving data from past courses. + +``` + Summer term 2016 + |-- Language C# and .NET platform + | |-- Labs Monday 10:30 + | `-- Labs Thursday 9:00 + |-- Programming I + | |-- Labs Monday 14:00 + ... +``` + +- _a database of exercises_ -- teachers should be able to create exercises + including textual description, sample inputs and correct reference outputs + (for example "sum all numbers from given file and write the result to the + standard output") and to browse this database +- _customizable grading system_ -- teachers need to specify the way of + computation of the final score, which will be awarded to the student's + submissions depending on their quality +- _viewing student details_ -- teachers should be able to view detailed data + about their students (members of their groups), including all submitted + solutions; +- _awarding additional points_ -- adding (or subtracting) points from the final + score of a submission by a supervisor must be supported +- _marking a solution as accepted_ -- the system should allow marking one + particular solution as accepted (used for grading the assignment) by the + supervisor +- _solution resubmission_ -- teachers should be able edit student's solutions + and privately resubmit them, optionally saving all results (including + temporary ones); this feature can be used to quickly fix errors in the + solution +- _localization_ -- all texts (UI and exercises) should be translatable +- _formatted exercise texts_ -- Markdown or another lightweight markup language + should be supported for formatting exercise texts +- _exercise tags_ -- the system should support tagging exercises searching by + these tags +- _comments_ -- adding both private and public comments to exercises, tests and + solutions should be supported +- _plagiarism detection_ #### Administrative requirements -- users can use an intuitive user interface for interaction with the system, - mainly for viewing assigned exercises, uploading their own solutions to the - assignments, and viewing the results of the solutions after an automatic - evaluation is finished; the two wanted interfaces are web and command-line - based -- user privilege separation (at least two roles -- _student_ and _supervisor_) -- logging in through a university authentication system (e.g. LDAP) -- SIS (university information system) integration for fetching personal user - data -- safe environment in which the students' solutions are executed -- support for multiple programming environments at once to avoid unacceptable - workload for administrator (maintain separate installation for every course) - and high hardware occupation +- _pluggable user interface_ -- the system should allow using an alternative + user interface, such as a command line client; implementation of such clients + should be as straightforward as possible +- _privilege separation_ -- there should be at least two roles -- _student_ and + _supervisor_. Cases when a student of a course is also a teacher of another + lab must be handled correctly +- _alternate authentication methods_ -- logging in through a university + authentication system (e.g. LDAP) and potentially other services, such as + OAuth, should be supported +- _querying SIS_ -- loading user data from the university information system + should be supported +- _sandboxing_ -- there should be a safe environment in which the students' + solutions are executed to prevent system failures due to malicious code being + submitted; the sandboxed environment should have the least possible impact on + measurement results (most importantly on measured times) +- _heterogenous worker pool_ -- there must be support for submission evaluation + in multiple programming environments in a single installation to avoid + unacceptable workload for the administrator (maintaining a separate + installation for every course) and high hardware occupation - advanced low-level evaluation flow configuration with high-level abstraction layer for ordinary configuration cases; the configuration should be able to express more complicated flows than just compiling a source code and running @@ -303,33 +330,36 @@ addons (mostly administrative features). ### Non-functional requirements Non-functional requirements are requirements of technical character with no -direct mapping to visible parts of system. In ideal word, users should not know -about these if they work properly, but would be at least annoyed if these -requirements were not met. Most notably they are these ones: - -- user interface of the system accessible on users' computers without - installation of any kind of additional software -- easy implementation of different user interfaces -- be ready for workload hundreds of students and tens of supervisors -- automated installation of all components -- source code with permissive license allowing further development; this also - applies on used libraries and frameworks -- multi-platform worker supporting at least two major operating systems +direct mapping to visible parts of the system. In an ideal world, users should +not know about these features if they work properly, but would be at least +annoyed if they did not. + +- _no installation_ -- the primary user interface of the system must be accessible + on users' computers without the need to install any additional software +- _performace_ -- the system must be ready for at least hundreds of students and + tens of supervisors using it at once +- _automated deployment_ -- all of the components of the system must be easy to + deploy in an automated fashion +- _open source licensing_ -- the source code should be released under a + permissive license allowing further development; this also applies to used + libraries and frameworks +- _multi-platform worker_ -- worker machines running Linux, Windows and + potentially other operating systems must be supported ### Conclusion The survey shows that there are a lot of different requirements and wishes for -the new system. When the system is ready it is likely that there will be new +the new system. When the system is ready, it is likely that there will be new ideas of how to use the system and thus the system must be designed to be easily -extendable, so everyone can develop their own feature. This also means that -widely used programming languages and techniques should be used, so users can -quickly understand the code and make changes. - +extendable, so that these new ideas can be easily implemented, either by us or +community members. This also means that widely used programming languages and +techniques should be used, so that users can quickly understand the code and +make changes. ## Related work -To find out the current state in the field of automatic grading systems we did a -short market survey on the field of automatic grading systems at universities, +To find out the current state in the field of automatic grading systems, we did +a short market survey on the field of automatic grading systems at universities, programming contests, and possibly other places where similar tools are available. From 1b906c0de12b2cd7fccb69fc8b8d8b6fc60979d8 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Thu, 12 Jan 2017 16:01:54 +0100 Subject: [PATCH 6/7] Typos --- Rewritten-docs.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 51a2737..3710fb1 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -267,7 +267,7 @@ addons (mostly administrative features). | `-- Labs Thursday 9:00 |-- Programming I | |-- Labs Monday 14:00 - ... + ... ``` - _a database of exercises_ -- teachers should be able to create exercises @@ -315,7 +315,7 @@ addons (mostly administrative features). solutions are executed to prevent system failures due to malicious code being submitted; the sandboxed environment should have the least possible impact on measurement results (most importantly on measured times) -- _heterogenous worker pool_ -- there must be support for submission evaluation +- _heterogeneous worker pool_ -- there must be support for submission evaluation in multiple programming environments in a single installation to avoid unacceptable workload for the administrator (maintaining a separate installation for every course) and high hardware occupation @@ -334,10 +334,11 @@ direct mapping to visible parts of the system. In an ideal world, users should not know about these features if they work properly, but would be at least annoyed if they did not. -- _no installation_ -- the primary user interface of the system must be accessible - on users' computers without the need to install any additional software -- _performace_ -- the system must be ready for at least hundreds of students and - tens of supervisors using it at once +- _no installation_ -- the primary user interface of the system must be + accessible on users' computers without the need to install any additional + software +- _performance_ -- the system must be ready for at least hundreds of students + and tens of supervisors using it at once - _automated deployment_ -- all of the components of the system must be easy to deploy in an automated fashion - _open source licensing_ -- the source code should be released under a From 90f69798bf3fa9cef5b513415f50043c143b2e73 Mon Sep 17 00:00:00 2001 From: Teyras Date: Thu, 12 Jan 2017 16:10:47 +0100 Subject: [PATCH 7/7] some nitpicks --- Rewritten-docs.md | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 3710fb1..a178076 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -120,12 +120,12 @@ they are willing to consult ideas or problems during development with us. ## Current system -Currently used grading solution at the Faculty of Mathematics and Physics of -the Charles University in Prague which was implemented in 2006 by a group -of students. It is called [CodEx -- The Code Examiner](http://codex.ms.mff.cuni.cz/project/) -and it has been used with some improvements since then. The original plan was -to use the system only for basic programming courses, but there was a demand -for adapting it for many different subjects. +The grading solution currently used at the Faculty of Mathematics and Physics of +the Charles University in Prague was implemented in 2006 by a group of students. +It is called [CodEx -- The Code Examiner](http://codex.ms.mff.cuni.cz/project/) +and it has been used with some improvements since then. The original plan was to +use the system only for basic programming courses, but there was a demand for +adapting it for many different subjects. CodEx is based on dynamic analysis. It features a web-based interface, where supervisors can assign exercises to their students and the students have a time @@ -182,7 +182,7 @@ happen during evaluation process. First thing users have to do is to submit their solutions through web user interface. The system checks assignment invariants (deadlines, count of -submissions, ...) and stores submitted file. The runtime environment is +submissions, ...) and stores the submitted file. The runtime environment is automatically detected based on input file and a suitable evaluation configuration variant is chosen (one exercise can have multiple variants, for example C and Java languages). This exercise configuration is then used for @@ -277,9 +277,8 @@ addons (mostly administrative features). - _customizable grading system_ -- teachers need to specify the way of computation of the final score, which will be awarded to the student's submissions depending on their quality -- _viewing student details_ -- teachers should be able to view detailed data - about their students (members of their groups), including all submitted - solutions; +- _viewing student details_ -- teachers should be able to view the details of + their students (members of their groups), including all submitted solutions - _awarding additional points_ -- adding (or subtracting) points from the final score of a submission by a supervisor must be supported - _marking a solution as accepted_ -- the system should allow marking one @@ -931,10 +930,10 @@ protocol between these two logical parts will be described as well. ## Implementation analysis When developing a project like ReCodEx there has to be some discussion over -implementation details and how to solve some particular problems properly. -This discussion is a never ending story which is done through the whole -development process. Some of the most important implementation problems or -interesting observations will be discussed in this chapter. +implementation details and how to solve some particular problems properly. This +discussion is a never ending story which goes on through the whole development +process. Some of the most important implementation problems or interesting +observations will be discussed in this chapter. ### General communication