From 6cd4235bf7e69018101403736632cb7cb699b01a Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Fri, 6 Jan 2017 18:06:07 +0100 Subject: [PATCH 01/46] Beginning of concept analysis --- Rewritten-docs.md | 88 ++++++++++++++++++++++++++++++----------------- 1 file changed, 56 insertions(+), 32 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 17b84d6..7d00eab 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -394,21 +394,21 @@ group, how points are assigned for solutions, ... @todo: merge with next chapter (Solution concept analysis) -None of the existing systems we came across is capable of all the required -features of the new system. There is no grading system which is designed to -support a complicated evaluation pipeline, so this part is an unexplored field -and has to be designed with caution. Also, no project is modern and extensible -so it could be used as a base for ReCodEx. After considering all these facts, -it was clear that a new system has to be written from scratch. This implies, -that only a subset of all the features will be implemented in the first version, -the other in the following ones. +None of the existing projects we came across is capable of all the required +features for the new system. There is no grading system which is designed to +support a complicated evaluation pipeline, so this unexplored field has to be +designed with caution. Also, no project is modern and extensible enough so it +could be used as a base for the new system. After considering all these facts, +it is clear that a new system has to be written from scratch. This implies, that +only a subset of all the features will be implemented in the first version, the +others coming in the following releases. Gathered features are categorized based on priorities for the whole system. The highest priority has main functionality similar to current CodEx. It is a base line to be useful in production environment, but a new design allows to easily develop further. On top of that, most of ideas from faculty staff belongs to -second priority bucket, which will be implemented as part of the project. Most -advanced tasks from this category are advanced low-level evaluation +second priority bucket, which will be implemented as part of the project. The +most complicated tasks from this category are advanced low-level evaluation configuration format, using modern tools, connecting to a university systems and merging separate system instances into single one. Other tasks are scheduled for next releases after successful project defense. Namely, these are high-level @@ -418,30 +418,49 @@ side) and command-line submit tool. Plagiarism detection is not likely to be part of any release in near future unless someone other makes the engine. The detection problem is too hard to be solved as part of this project. -We named the project as **ReCodEx -- ReCodEx Code Examiner**. The name should -point to the old CodEx, but also reflect the new approach to solve issues. -**Re** as part of the name means redesigned, rewritten, renewed, or restarted. +We named the new project as **ReCodEx -- ReCodEx Code Examiner**. The name +should point to the old CodEx, but also reflect the new approach to solve +issues. **Re** as part of the name means redesigned, rewritten, renewed, or +restarted. At this point there is a clear idea how the new system will be used and what are the major enhancements for future releases. With this in mind, the overall -architecture can be sketched. From the previous research, we set up several -goals, which the new system should have. They mostly reflect drawbacks of the -current version of CodEx and some reasonable wishes of university users. Most -notable features are following: +architecture can be sketched. From the previous research, several goals are set +up for the new project. They mostly reflect drawbacks of the current version of +CodEx and some reasonable wishes of university users. Most notable features are +following: - modern HTML5 web frontend written in JavaScript using a suitable framework - REST API implemented in PHP, communicating with database, evaluation backend and a file server - evaluation backend implemented as a distributed system on top of a message - queue framework (ZeroMQ) with master-worker architecture -- worker with basic support of the Windows environment (without sandbox, no - general purpose suitable tool available yet) + queue framework (ZeroMQ) with master-worker architecture +- multi-platform worker supporting Linux and Windows environment (latter + without sandbox, no general purpose suitable tool available yet) - evaluation procedure configured in a YAML file, compound of small tasks connected into an arbitrary oriented acyclic graph - -## Solution concepts analysis +The reasons supporting these decisions are explained in the rest of analysis +chapter. Also a lot of smaller design choices are mentioned including possible +options, what is picked to implement and why. But first, discuss basic concepts +of the system. + +### Basic concepts + +The system is designed as a web application. The requirements say, that user +interface must be accessible from students' computers without installation +additional software. This immediately implies that users are connected to the +internet, so it is used as communication medium. Today, there are two main ways +of designing graphical user interface -- native app or web page. Make nice and +multi-platform application with graphical interface is almost impossible because +of large number of different environments. Also, these applications often +requires installation or at least downloading its files (sources or binaries). +However, creating web application is easier, because every personal computer has +a internet browser installed. Also, browsers supports sort of unified and +standardized environment of HTML5 and JavaScript. CodEx is also web application +and everybody seems happy about it. There are other communicating channels every +user have available like email or git, but they are totally inappropriate for +designing user interfaces on top of them. @todo: what problems were solved on abstract and high levels, how they can be solved and what was the final solution @@ -457,6 +476,7 @@ notable features are following: - judges, discuss what they possibly can do and what it can be used for (returning for instance 2 numbers instead of 1 and why we return just one) - discuss points assigned to solution, why are there bonus points, explain minimal point threshold - discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why +- where is kept the state (MariaDB) - and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks @@ -520,7 +540,7 @@ is a database of exercises which can be assigned to the groups and the users of these groups can submit their solutions for these assignments. The frontend will initiate evaluation of these solutions by the backend and it will store the results afterwards. The results will be visible to authorized users and the -results will be awarded with points according to the score given by the Backend +results will be awarded with points according to the score given by the backend in the evaluation process. The supervisors of the groups can edit the parameters of the assignments, review the solutions and the evaluations in detail and award the solutions with bonus points (both positive and negative) and discuss about @@ -541,13 +561,13 @@ of the users. The frontend developed as part of this project is a web application created with the needs of the Faculty of Mathematics and Physics of the Charles university in -Prague in mind. The users are the students and their teachers, groups -correspond to the different courses, the teachers are the supervisors of these -groups. We believe that this model is applicable to the needs of other -universities, schools, and IT companies, which can use the same system for their -needs. It is also possible to develop their own frontend with their own user -management system for their specific needs and use the possibilities of the -Backend without any changes, as was mentioned in the previous paragraphs. +Prague in mind. The users are the students and their teachers, groups correspond +to the different courses, the teachers are the supervisors of these groups. We +believe that this model is applicable to the needs of other universities, +schools, and IT companies, which can use the same system for their needs. It is +also possible to develop their own frontend with their own user management +system for their specific needs and use the possibilities of the backend without +any changes, as was mentioned in the previous paragraphs. In the latter parts of the documentation, both of the backend and frontend parts will be introduced separately and covered in more detail. The communication @@ -645,7 +665,11 @@ interesting observations will be discussed in this chapter. ### Backend communication -@todo: what type of communication within backend could be used, mention some frameworks, queue managers, protocols, which was considered +@todo: what type of communication within backend could be used, mention some +frameworks, queue managers, protocols, which was considered - why we don't use +CORBA, RabbitMQ and why is ZeroMQ great + +@todo: maybe merge frontend communication into this ### Broker From 533db9662e675c41777a1edf1c741f07bba0f0f2 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Fri, 6 Jan 2017 18:29:10 +0100 Subject: [PATCH 02/46] Sandboxing --- Rewritten-docs.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 17b84d6..36ef770 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -759,7 +759,20 @@ Previous description implies that there is gap between detection of last access #### Sandboxing -@todo: sandboxing, what possibilites are out there (Linux, Windows), what are general and really needed features, mention isolate, what are isolate features +There are numerous ways how to approach sandboxing on different platforms, +describing all possible approaches is out of scope of this document. Instead of +that have a look at some of the features which are certainly needed for ReCodEx +and propose some particular sandboxes implementations on linux or Windows. + +General purpose of sandbox is safely execute software in any form, from scripts to binaries. Various sandboxes differ in how safely are they and what limiting features they have. Ideal situation is that sandbox will have numerous options and corresponding features which will allow administrators to setup environment as they like and which will not allow user programs to somehow damage executing machine in any way possible. + +For ReCodEx and its evaluation there is need for at least these features: execution time and memory limitation, disk operations limit, disk accessibility restrictions and network restrictions. All these features if combined and implemented well are giving pretty safe sandbox which can be used for all kinds of users solutions and should be able to restrict and stop any standard way of attacks or errors. + +Linux systems have quite extent support of sandboxing in kernel, there were introduced and implemented kernel namespaces and cgroups which combined can limit hardware resources (cpu, memory) and separate executing program into its own namespace (pid, network). These two features comply sandbox requirement for ReCodEx so there were two options, either find existing solution or implement new one. Luckily existing solution was found and its name is **isolate**. Isolate does not use all possible kernel features but only subset which is still enough to be used by ReCodEx. + +The opposite situation is in Windows world, there is limited support in its kernel which makes sandboxing a bit trickier. Windows kernel only has ways how to restrict privileges of a process through restriction of internal access tokens. Monitoring of hardware resources is not possible but used resources can be obtained through newly created job objects. But find sandbox which can do all things needed for ReCodEx seems to be impossible. There are numerous sandboxes for Windows but they all are focused on different things in a lot of cases they serves as safe environment for malicious programs, viruses in particular. Or they are designed as a separate filesystem namespace for installing a lot of temporarily used programs. From all these we can mention Sandboxie, Comodo Internet Security, Cuckoo sandbox and many others. None of these is fitted as sandbox solution for ReCodEx. With this being said we can safely state that designing and implementing new general sandbox for Windows is out of scope of this project. + +New general sandbox for Windows is out of bussiness but what about more specialized solution used for instance only for C#. CLR as a virtual machine and runtime environment has a pretty good security support for restrictions and separation which is also transfered to C#. This makes it quite easy to implement simple sandbox within C# but suprisingly there cannot be found some well known general purpose implementations. As said in previous paragraph implementing our own solution is out of scope of project there is simple not enough time. But C# sandbox is quite good topic for another project for example semestral project for C# course so it might be written and integrated in future. ### Fileserver From 3485a160709ad80a2720bc4a04f1aa0291c3f876 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Fri, 6 Jan 2017 18:30:59 +0100 Subject: [PATCH 03/46] Format --- Rewritten-docs.md | 177 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 154 insertions(+), 23 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 306f846..5d56a87 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -699,11 +699,60 @@ ZeroMQ is possible to provide in-process messages working on the same principles as network communication which is quite handy and solves problems with threads synchronization and such. -At this point we have worker with two internal parts listening one and execution one. Implementation of first one is quite straighforward and clear. So lets discuss what should be happening in execution subsystem. Jobs as work units can quite vary and do completely different things, that means configuration and worker has to be prepared for this kind of generality. Configuration and its solution was already discussed above, implementation in worker is then quite also quite straightforward. Worker has internal structures to which loads and which stores metadata given in configuration. Whole job is mapped to job metadata structure and tasks are mapped to either external ones or internal ones (internal commands has to be defined within worker), both are different whether they are executed in sandbox or as internal worker commands. - -Another division of tasks is by task-type field in configuration. This field can have four values: initiation, execution, evaluation and inner. All was discussed and described above in configuration analysis. What is important to worker is how to behave if execution of task with some particular type fails. There are two possible situations execution fails due to bad user solution or due to some internal error. If execution fails on internal error solution cannot be declared overly as failed. User should not be punished for bad configuration or some network error. This is where task types are useful. Generally initiation, execution and evaluation are tasks which are somehow executing code which was given by users who submitted solution of exercise. If this kinds of tasks fail it is probably connected with bad user solution and can be evaluated. But if some inner task fails solution should be re-executed, in best case scenario on different worker. That is why if inner task fails it is sent back to broker which will reassign job to another worker. More on this subject should be discussed in broker assigning algorithms section. - -There is also question about working directory or directories of job, which directories should be used and what for. There is one simple answer on this every job will have only one specified directory which will contain every file with which worker will work in the scope of whole job execution. This is of course nonsense there has to be some logical division. The least which must be done are two folders one for internal temporary files and second one for evaluation. The directory for temporary files is enough to comprehend all kind of internal work with filesystem but only one directory for whole evaluation is somehow not enough. Users solutions are downloaded in form of zip archives so why these should be present during execution or why the results and files which should be uploaded back to fileserver should be cherry picked from the one big directory? The answer is of course another logical division into subfolders. The solution which was chosen at the end is to have folders for downloaded archive, decompressed solution, evaluation directory in which user solution is executed and then folders for temporary files and for results and generally files which should be uploaded back to fileserver with solution results. Of course there has to be hierarchy which separate folders from different workers on the same machines. That is why paths to directories are in format: ${DEFAULT}/${FOLDER}/${WORKER_ID}/${JOB_ID} where default means default working directory of whole worker, folder is particular directory for some purpose (archives, evaluation...). Mentioned division of job directories proved to be flexible and detailed enough, everything is in logical units and where it is supposed to be which means that searching through this system should be easy. In addition if solutions of users have access only to evaluation directory then they do not have access to unnecessary files which is better for overall security of whole ReCodEx. +At this point we have worker with two internal parts listening one and execution +one. Implementation of first one is quite straighforward and clear. So lets +discuss what should be happening in execution subsystem. Jobs as work units can +quite vary and do completely different things, that means configuration and +worker has to be prepared for this kind of generality. Configuration and its +solution was already discussed above, implementation in worker is then quite +also quite straightforward. Worker has internal structures to which loads and +which stores metadata given in configuration. Whole job is mapped to job +metadata structure and tasks are mapped to either external ones or internal ones +(internal commands has to be defined within worker), both are different whether +they are executed in sandbox or as internal worker commands. + +Another division of tasks is by task-type field in configuration. This field can +have four values: initiation, execution, evaluation and inner. All was discussed +and described above in configuration analysis. What is important to worker is +how to behave if execution of task with some particular type fails. There are +two possible situations execution fails due to bad user solution or due to some +internal error. If execution fails on internal error solution cannot be declared +overly as failed. User should not be punished for bad configuration or some +network error. This is where task types are useful. Generally initiation, +execution and evaluation are tasks which are somehow executing code which was +given by users who submitted solution of exercise. If this kinds of tasks fail +it is probably connected with bad user solution and can be evaluated. But if +some inner task fails solution should be re-executed, in best case scenario on +different worker. That is why if inner task fails it is sent back to broker +which will reassign job to another worker. More on this subject should be +discussed in broker assigning algorithms section. + +There is also question about working directory or directories of job, which +directories should be used and what for. There is one simple answer on this +every job will have only one specified directory which will contain every file +with which worker will work in the scope of whole job execution. This is of +course nonsense there has to be some logical division. The least which must be +done are two folders one for internal temporary files and second one for +evaluation. The directory for temporary files is enough to comprehend all kind +of internal work with filesystem but only one directory for whole evaluation is +somehow not enough. Users solutions are downloaded in form of zip archives so +why these should be present during execution or why the results and files which +should be uploaded back to fileserver should be cherry picked from the one big +directory? The answer is of course another logical division into subfolders. The +solution which was chosen at the end is to have folders for downloaded archive, +decompressed solution, evaluation directory in which user solution is executed +and then folders for temporary files and for results and generally files which +should be uploaded back to fileserver with solution results. Of course there has +to be hierarchy which separate folders from different workers on the same +machines. That is why paths to directories are in format: +${DEFAULT}/${FOLDER}/${WORKER_ID}/${JOB_ID} where default means default working +directory of whole worker, folder is particular directory for some purpose +(archives, evaluation...). Mentioned division of job directories proved to be +flexible and detailed enough, everything is in logical units and where it is +supposed to be which means that searching through this system should be easy. In +addition if solutions of users have access only to evaluation directory then +they do not have access to unnecessary files which is better for overall +security of whole ReCodEx. As we discovered above worker has job directories but users who are writing and managing job configurations do not know where they are (on some particular @@ -720,9 +769,31 @@ description of variable. #### Evaluation -After successful arrival of job, worker has to prepare new execution environment, then solution archive has to be downloaded from fileserver and extracted. Job configuration is located within these files and loaded into internal structures and executed. After that results are uploaded back to fileserver. These steps are the basic ones which are really necessary for whole execution and have to be executed in this precise order. - -Interesting problem is with supplementary files (inputs, sample outputs). There are two approaches which can be observed. Supplementary files can be downloaded either on the start of the execution or during execution. If the files are downloaded at the beginning execution does not really started at this point and if there are problems with network worker find it right away and can abort execution without executing single task. Slight problems can arise if some of the files needs to have same name (e.g. solution assumes that input is `input.txt`), in this scenario downloaded files cannot be renamed at the beginning but during execution which is somehow impractical and not easily observed. Second solution of this problem when files are downloaded on the fly has quite opposite problem, if there are problems with network worker will find it during execution when for instance almost whole execution is done, this is also not ideal solution if we care about burnt hardware resources. On the other hand using this approach users have quite advanced control of execution flow and know what files exactly are available during execution which is from users perspective probably more appealing then the first solution. Based on that downloading of supplementary files using 'fetch' tasks during execution was chosen and implemented. +After successful arrival of job, worker has to prepare new execution +environment, then solution archive has to be downloaded from fileserver and +extracted. Job configuration is located within these files and loaded into +internal structures and executed. After that results are uploaded back to +fileserver. These steps are the basic ones which are really necessary for whole +execution and have to be executed in this precise order. + +Interesting problem is with supplementary files (inputs, sample outputs). There +are two approaches which can be observed. Supplementary files can be downloaded +either on the start of the execution or during execution. If the files are +downloaded at the beginning execution does not really started at this point and +if there are problems with network worker find it right away and can abort +execution without executing single task. Slight problems can arise if some of +the files needs to have same name (e.g. solution assumes that input is +`input.txt`), in this scenario downloaded files cannot be renamed at the +beginning but during execution which is somehow impractical and not easily +observed. Second solution of this problem when files are downloaded on the fly +has quite opposite problem, if there are problems with network worker will find +it during execution when for instance almost whole execution is done, this is +also not ideal solution if we care about burnt hardware resources. On the other +hand using this approach users have quite advanced control of execution flow and +know what files exactly are available during execution which is from users +perspective probably more appealing then the first solution. Based on that +downloading of supplementary files using 'fetch' tasks during execution was +chosen and implemented. #### Caching mechanism @@ -748,7 +819,19 @@ within cron which is able to delete files which were unused for some time. Together with worker fetching feature cleaner completes machine specific caching system. -Cleaner as mentioned is simple script which is executed regularly as cron job. If there is caching system like it was introduced in paragraph above there are little possibilities how cleaner should be implemented. On various filesystems there is usually support for two particular timestamps, `last access time` and `last modification time`. Files in cache are once downloaded and then just copied, this means that last modification time is set only once on creation of file and last access time should be set every time on copy. This imply last access time is what is needed here. But last modification time is widely used by operating systems, on the other hand last access time is not by default. More on this subject can be found [here](https://en.wikipedia.org/wiki/Stat_%28system_call%29#Criticism_of_atime). For proper cleaner functionality filesystem which is used by worker for caching has to have last access time for files enabled. +Cleaner as mentioned is simple script which is executed regularly as cron job. +If there is caching system like it was introduced in paragraph above there are +little possibilities how cleaner should be implemented. On various filesystems +there is usually support for two particular timestamps, `last access time` and +`last modification time`. Files in cache are once downloaded and then just +copied, this means that last modification time is set only once on creation of +file and last access time should be set every time on copy. This imply last +access time is what is needed here. But last modification time is widely used by +operating systems, on the other hand last access time is not by default. More on +this subject can be found +[here](https://en.wikipedia.org/wiki/Stat_%28system_call%29#Criticism_of_atime). +For proper cleaner functionality filesystem which is used by worker for caching +has to have last access time for files enabled. Having cleaner as separated component and caching itself handled in worker is kind of blurry and is not clearly observable that it works without any race @@ -775,11 +858,21 @@ delete files. Implementation in cleaner follows: - there is a loop going through all files and even directories in specified cache folder - last access time of file or folder is detected - - last access time is subtracted from reference timestamp into difference - - difference is compared against specified maximal file age, if difference - is greater, file or folder is deleted - -Previous description implies that there is gap between detection of last access time and deleting file within cleaner. In the gap there can be worker which will access file and the file is anyway deleted but this is fine, file is deleted but worker has it copied. Another problem can be with two workers downloading the same file, but this is also not a problem file is firstly downloaded to working folder and after that copied to cache. And even if something else unexpectedly fails and because of that fetch task will fail during execution even that should be fine. Because fetch tasks should have 'inner' task type which implies that fail in this task will stop all execution and job will be reassigned to another worker. It should be like the last salvation in case everything else goes wrong. + - last access time is subtracted from reference timestamp into + difference + - difference is compared against specified maximal file age, if + difference is greater, file or folder is deleted + +Previous description implies that there is gap between detection of last access +time and deleting file within cleaner. In the gap there can be worker which will +access file and the file is anyway deleted but this is fine, file is deleted but +worker has it copied. Another problem can be with two workers downloading the +same file, but this is also not a problem file is firstly downloaded to working +folder and after that copied to cache. And even if something else unexpectedly +fails and because of that fetch task will fail during execution even that should +be fine. Because fetch tasks should have 'inner' task type which implies that +fail in this task will stop all execution and job will be reassigned to another +worker. It should be like the last salvation in case everything else goes wrong. #### Sandboxing @@ -788,15 +881,53 @@ describing all possible approaches is out of scope of this document. Instead of that have a look at some of the features which are certainly needed for ReCodEx and propose some particular sandboxes implementations on linux or Windows. -General purpose of sandbox is safely execute software in any form, from scripts to binaries. Various sandboxes differ in how safely are they and what limiting features they have. Ideal situation is that sandbox will have numerous options and corresponding features which will allow administrators to setup environment as they like and which will not allow user programs to somehow damage executing machine in any way possible. - -For ReCodEx and its evaluation there is need for at least these features: execution time and memory limitation, disk operations limit, disk accessibility restrictions and network restrictions. All these features if combined and implemented well are giving pretty safe sandbox which can be used for all kinds of users solutions and should be able to restrict and stop any standard way of attacks or errors. - -Linux systems have quite extent support of sandboxing in kernel, there were introduced and implemented kernel namespaces and cgroups which combined can limit hardware resources (cpu, memory) and separate executing program into its own namespace (pid, network). These two features comply sandbox requirement for ReCodEx so there were two options, either find existing solution or implement new one. Luckily existing solution was found and its name is **isolate**. Isolate does not use all possible kernel features but only subset which is still enough to be used by ReCodEx. - -The opposite situation is in Windows world, there is limited support in its kernel which makes sandboxing a bit trickier. Windows kernel only has ways how to restrict privileges of a process through restriction of internal access tokens. Monitoring of hardware resources is not possible but used resources can be obtained through newly created job objects. But find sandbox which can do all things needed for ReCodEx seems to be impossible. There are numerous sandboxes for Windows but they all are focused on different things in a lot of cases they serves as safe environment for malicious programs, viruses in particular. Or they are designed as a separate filesystem namespace for installing a lot of temporarily used programs. From all these we can mention Sandboxie, Comodo Internet Security, Cuckoo sandbox and many others. None of these is fitted as sandbox solution for ReCodEx. With this being said we can safely state that designing and implementing new general sandbox for Windows is out of scope of this project. - -New general sandbox for Windows is out of bussiness but what about more specialized solution used for instance only for C#. CLR as a virtual machine and runtime environment has a pretty good security support for restrictions and separation which is also transfered to C#. This makes it quite easy to implement simple sandbox within C# but suprisingly there cannot be found some well known general purpose implementations. As said in previous paragraph implementing our own solution is out of scope of project there is simple not enough time. But C# sandbox is quite good topic for another project for example semestral project for C# course so it might be written and integrated in future. +General purpose of sandbox is safely execute software in any form, from scripts +to binaries. Various sandboxes differ in how safely are they and what limiting +features they have. Ideal situation is that sandbox will have numerous options +and corresponding features which will allow administrators to setup environment +as they like and which will not allow user programs to somehow damage executing +machine in any way possible. + +For ReCodEx and its evaluation there is need for at least these features: +execution time and memory limitation, disk operations limit, disk accessibility +restrictions and network restrictions. All these features if combined and +implemented well are giving pretty safe sandbox which can be used for all kinds +of users solutions and should be able to restrict and stop any standard way of +attacks or errors. + +Linux systems have quite extent support of sandboxing in kernel, there were +introduced and implemented kernel namespaces and cgroups which combined can +limit hardware resources (cpu, memory) and separate executing program into its +own namespace (pid, network). These two features comply sandbox requirement for +ReCodEx so there were two options, either find existing solution or implement +new one. Luckily existing solution was found and its name is **isolate**. +Isolate does not use all possible kernel features but only subset which is still +enough to be used by ReCodEx. + +The opposite situation is in Windows world, there is limited support in its +kernel which makes sandboxing a bit trickier. Windows kernel only has ways how +to restrict privileges of a process through restriction of internal access +tokens. Monitoring of hardware resources is not possible but used resources can +be obtained through newly created job objects. But find sandbox which can do all +things needed for ReCodEx seems to be impossible. There are numerous sandboxes +for Windows but they all are focused on different things in a lot of cases they +serves as safe environment for malicious programs, viruses in particular. Or +they are designed as a separate filesystem namespace for installing a lot of +temporarily used programs. From all these we can mention Sandboxie, Comodo +Internet Security, Cuckoo sandbox and many others. None of these is fitted as +sandbox solution for ReCodEx. With this being said we can safely state that +designing and implementing new general sandbox for Windows is out of scope of +this project. + +New general sandbox for Windows is out of bussiness but what about more +specialized solution used for instance only for C#. CLR as a virtual machine and +runtime environment has a pretty good security support for restrictions and +separation which is also transfered to C#. This makes it quite easy to implement +simple sandbox within C# but suprisingly there cannot be found some well known +general purpose implementations. As said in previous paragraph implementing our +own solution is out of scope of project there is simple not enough time. But C# +sandbox is quite good topic for another project for example semestral project +for C# course so it might be written and integrated in future. ### Fileserver From adb8d03102491a1509e19f748120701c4f460ecb Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Fri, 6 Jan 2017 22:25:17 +0100 Subject: [PATCH 04/46] Restruct --- Rewritten-docs.md | 55 ++++++++++++++++++++++++----------------------- 1 file changed, 28 insertions(+), 27 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 5d56a87..efab2a9 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -386,14 +386,6 @@ recruiters and also some universities. # Analysis -## ReCodEx goals - -@todo: improve and extend this chapter - analysis of user requirements and way we -solve them; exercise is a template for assignment, users are in groups, what is -group, how points are assigned for solutions, ... - -@todo: merge with next chapter (Solution concept analysis) - None of the existing projects we came across is capable of all the required features for the new system. There is no grading system which is designed to support a complicated evaluation pipeline, so this unexplored field has to be @@ -445,7 +437,7 @@ chapter. Also a lot of smaller design choices are mentioned including possible options, what is picked to implement and why. But first, discuss basic concepts of the system. -### Basic concepts +## Basic concepts The system is designed as a web application. The requirements say, that user interface must be accessible from students' computers without installation @@ -462,22 +454,31 @@ and everybody seems happy about it. There are other communicating channels every user have available like email or git, but they are totally inappropriate for designing user interfaces on top of them. -@todo: what problems were solved on abstract and high levels, how they can be solved and what was the final solution +@todo: what type of users there should be, why they are needed + +@todo: groups, they can be public and private and why is that, what it solves, explain and discuss threshold and other group features + +@todo: explain instances why they are useful what they solve and also discuss licenses concept + +@todo: exercise, what is it why we need it + +@todo: explain why there is exercise and assignment division, what means what and how they are used + +@todo: extended execution pipeline (not just compilation/execution/evaluation) and why it is needed + +@todo: progress state, how it can be done and displayed to user, why random messages + +@todo: how to display generally all outputs of executed programs to user (supervisor, student), what students can or cannot see and why + +@todo: judges, discuss what they possibly can do and what it can be used for (returning for instance 2 numbers instead of 1 and why we return just one) + +@todo: discuss points assigned to solution, why are there bonus points, explain minimal point threshold + +@todo: discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why + +@todo: where is kept the state (MariaDB) -- which problems are they? ... these ones below: -- what type of users there should be, why they are needed -- explain why there is exercise and assignment division, what means what and how they are used -- explain instances why they are useful what they solve and also discuss licenses concept -- groups, they can be public and private and why is that, what it solves, - explain and discuss threshold and other group features -- extended execution pipeline (not just compilation/execution/evaluation) and why it is needed -- progress state, how it can be done and displayed to user, why random messages -- how to display generally all outputs of executed programs to user (supervisor, student), what students can or cannot see and why -- judges, discuss what they possibly can do and what it can be used for (returning for instance 2 numbers instead of 1 and why we return just one) -- discuss points assigned to solution, why are there bonus points, explain minimal point threshold -- discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why -- where is kept the state (MariaDB) -- and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks +@todo: and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks ## Structure of the project @@ -570,17 +571,17 @@ system for their specific needs and use the possibilities of the backend without any changes, as was mentioned in the previous paragraphs. In the latter parts of the documentation, both of the backend and frontend parts -will be introduced separately and covered in more detail. The communication +will be introduced separately and covered in more detail. The communication protocol between these two logical parts will be described as well. -### Evaluation unit executed on backend +### Evaluation unit executed by ReCodEx One of the bigger requests for the new system is to support a complex configuration of execution pipeline. The idea comes from lecturers of Compiler principles class who want to migrate their semi-manual evaluation process to CodEx. Unfortunately, CodEx is not capable of such complicated exercise setup. -None of evaluation systems we found is can handle such task, so design from +None of evaluation systems we found can handle such task, so design from scratch is needed. There are two main approaches to design a complex execution configuration. It From 974f4ee467b2dbc3710eb3b871cb9760bc601376 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Sat, 7 Jan 2017 14:28:15 +0100 Subject: [PATCH 05/46] Add 2 paragraphs :D --- Rewritten-docs.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index efab2a9..6aa60b7 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -454,7 +454,33 @@ and everybody seems happy about it. There are other communicating channels every user have available like email or git, but they are totally inappropriate for designing user interfaces on top of them. -@todo: what type of users there should be, why they are needed +The application interacts with users. From the project assignment is clear, that +the system has to keep personalized data about users and adapt presented content +according to this knowledge. User data cannot be publicly visible, so that +implies necessity of user authentication. There are several way of +auhtentication user in web applications. HTTP basic authentication has a few +drawbacks like sending plain credentials in every request or no logout option, +so it is not recommended to use. Using cookies is possible, but it is +susceptible to various types of attacks including stealing and also brings state +into stateless protocols as REST. Another option is using tokens, OAuth2 or JWT +(JSON Web Token). Nowadays it is widely used and has only a few cons (need to +make extra effort to mitigate XSS attacks). Additional option is usage of +one-time passwords. These can be time or counter based and are mostly used for +two-factor authentication. Since ReCodEx does not need to have military grade +security, JWT tokens are used for authentication. However, two-factor +authentication may come in next releases. + +User data also includes a privilege level. From the assignment it is required to +have at least two roles, _student_ and _supervisor_. However, it is wise to add +_administrator_ level, which takes care of the system as a whole and is +responsible for core setup, monitoring, updates and so on. Student role has the +least power, basically can just view assignments and submit solutions. +Supervisors have more authority, so they can create exercises and assignments, +view results of students etc. From the university organization, one possible +level could be introduced, _course guarantor_. However, from real experience all +duties related with lecturing of labs are already associtated with supervisors, +so this role seems not so useful. In addition, no one requested more than three +level privilege scheme. @todo: groups, they can be public and private and why is that, what it solves, explain and discuss threshold and other group features From 9071b6b4fdae423862a46510e7afea2550e96308 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Sat, 7 Jan 2017 23:10:22 +0100 Subject: [PATCH 06/46] More analyis (groups) --- Rewritten-docs.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 6aa60b7..c5ddead 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -482,7 +482,29 @@ duties related with lecturing of labs are already associtated with supervisors, so this role seems not so useful. In addition, no one requested more than three level privilege scheme. -@todo: groups, they can be public and private and why is that, what it solves, explain and discuss threshold and other group features +School labs are lessons for some students lead by one (or sometimes two or +three) supervisors. Students have the same homeworks and supervisors are +evaluating them. This organization has to be carried into the new system. +Counterpart to real labs are virtual groups. This concept was already discussed +in previous chapter including need for hierarchical structure of groups. Right +for attending labs has only a person, who is student of the university and is +recorded in university information system. To allow restriction of group members +in ReCodEx, there two type of groups -- _public_ and _private_. Public groups +are open for every registered users, but to become a member of private group one +of its supervisors have to add that user. This could be done automatically at +beginning of the term with data from information system, but unfortunately there +is no such API yet. However, creating this API is now considered by university +leadership. Another just as good solution for restricting membership of a group +is to allow anyone join the group with supplementary confirmation of +supervisors. It has no additional benefits, so approach with public and private +groups is implemented. + +Supervisors using CodEx in their labs usually set minimum amout of points +required to get a credit. These points can be get by solving assigned exercises. +To visualy show users if they already have enough points, ReCodEx groups +supports setting this limit. There are two equal ways how to set a limit -- +absolute value or relative value to maximum. The latter way seems nicer, so it +is implemented. The relative value is set in percents and is called threashold. @todo: explain instances why they are useful what they solve and also discuss licenses concept From e25fbd43cb617f9ec432a0295d52bc73406642a5 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Sat, 7 Jan 2017 23:52:57 +0100 Subject: [PATCH 07/46] Reorder, will fill later --- Rewritten-docs.md | 86 ++++++++++++++++++++++------------------------- 1 file changed, 41 insertions(+), 45 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index c5ddead..08648a0 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -712,13 +712,52 @@ This discussion is a never ending story which is done through the whole development process. Some of the most important implementation problems or interesting observations will be discussed in this chapter. -### Backend communication +### General communication @todo: what type of communication within backend could be used, mention some frameworks, queue managers, protocols, which was considered - why we don't use CORBA, RabbitMQ and why is ZeroMQ great -@todo: maybe merge frontend communication into this +Frontend communication follows the choice, that ReCodEx should be primary a web +application. The communication protocol has to reflect client-server +architecture. There are several options: + +- *TCP sockets* -- TCP sockets give a reliable means of a full-duplex + communication. All major operating systems support this protocol and there are + libraries which simplify the implementation. On the other side, it is not + possible to initiate a TCP socket from a web browser. +- *WebSockets* -- The WebSocket standard is built on top of TCP. It enables a + web browser to connect to a server over a TCP socket. WebSockets are + implemented in recent versions of all modern web browsers and there are + libraries for several programming languages like Python or JavaScript (running + in Node.js). Encryption of the communication over a WebSocket is supported as + a standard. +- *HTTP protocol* -- The HTTP protocol is a state-less protocol implemented on + top of the TCP protocol. The communication between the client and server + consists of a requests sent by the client and responses to these requests sent + back by the sever. The client can send as many requests as needed and it may + ignore the responses from the server, but the server must respond only to the + requests of the client and it cannot initiate communication on its own. + End-to-end encryption can be achieved easily using SSL (HTTPS). + +We chose the HTTP(S) protocol because of the simple implementation in all sorts +of operating systems and runtime environments on both the client and the server +side. + +The API of the server should expose basic CRUD (Create, Read, Update, Delete) +operations. There are some options on what kind of messages to send over the +HTTP: + +- SOAP -- a protocol for exchanging XML messages. It is very robust and complex. +- REST -- is a stateless architecture style, not a protocol or a technology. It + relies on HTTP (but not necessarily) and its method verbs (e.g., GET, POST, + PUT, DELETE). It can fully implement the CRUD operations. + +Even though there are some other technologies we chose the REST style over the +HTTP protocol. It is widely used, there are many tools available for development +and testing, and it is understood by programmers so it should be easy for a new +developer with some experience in client-side applications to get to know with +the ReCodEx API and develop a client application. ### Broker @@ -1041,49 +1080,6 @@ the monitor. To solve this, messages for each job are hold 5 minutes after reception of last message. The client gets all already received messages at time of connection with no message loss. - -### Frontend communication - -The first thing we need to address is the communication protocol of this -client-server architecture. There are several options: - -- *TCP sockets* -- TCP sockets give a reliable means of a full-duplex - communication. All major operating systems support this protocol and there are - libraries which simplify the implementation. On the other side, it is not - possible to initiate a TCP socket from a web browser. -- *WebSockets* -- The WebSocket standard is built on top of TCP. It enables a - web browser to connect to a server over a TCP socket. WebSockets are - implemented in recent versions of all modern web browsers and there are - libraries for several programming languages like Python or JavaScript (running - in Node.js). Encryption of the communication over a WebSocket is supported as - a standard. -- *HTTP protocol* -- The HTTP protocol is a state-less protocol implemented on - top of the TCP protocol. The communication between the client and server - consists of a requests sent by the client and responses to these requests sent - back by the sever. The client can send as many requests as needed and it may - ignore the responses from the server, but the server must respond only to the - requests of the client and it cannot initiate communication on its own. - End-to-end encryption can be achieved easily using SSL (HTTPS). - -We chose the HTTP(S) protocol because of the simple implementation in all sorts -of operating systems and runtime environments on both the client and the server -side. - -The API of the server should expose basic CRUD (Create, Read, Update, Delete) -operations. There are some options on what kind of messages to send over the -HTTP: - -- SOAP -- a protocol for exchanging XML messages. It is very robust and complex. -- REST -- is a stateless architecture style, not a protocol or a technology. It - relies on HTTP (but not necessarily) and its method verbs (e.g., GET, POST, - PUT, DELETE). It can fully implement the CRUD operations. - -Even though there are some other technologies we chose the REST style over the -HTTP protocolo. It is widely used, there are many tools available for -development and testing, and it is understood by programmers so it should be -easy for a new developer with some experience in client-side applications to get -to know with the ReCodEx API and develop a client application. - ### API server The API server must handle HTTP requests and manage the state of the application From 66ee27d1852f2c6441a8129a1ff2eab7be7cff6f Mon Sep 17 00:00:00 2001 From: Teyras Date: Sun, 8 Jan 2017 00:10:35 +0100 Subject: [PATCH 08/46] broker analysis --- Rewritten-docs.md | 76 ++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 69 insertions(+), 7 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index efab2a9..6e4687d 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -674,11 +674,73 @@ CORBA, RabbitMQ and why is ZeroMQ great ### Broker -@todo: assigning of jobs to workers, which are possible algorithms, queues, which one was chosen - -@todo: how can jobs be sent over zeromq, mainly mention that files can be transported, but it is not feasible - -@todo: making action and reaction over zeromq more general and easily extensible, mention reactor and why is needed and what it solves +The broker is responsible for keeping track of available workers and +distributing jobs that it receives from the frontend between them. + +#### Worker management + +@todo initialization - broker is fixed, workers connect to it + +@todo heartbeating - workers send ping, the inverse is possible, too (doesn't +really matter) + +#### Scheduling + +Jobs should be scheduled in a way that ensures that they will be processed +without unnecessary waiting. This depends on the fairness of the scheduling +algorithm (no worker machine should be overloaded). + +The design of such scheduling algorithm is complicated by the requirements on +the diversity of workers -- they can differ in operating systems, available +software, computing power and many other aspects. + +We decided to keep the details of connected workers hidden from the frontend, +which should lead to a better separation of responsibilities and flexibility. +Therefore, the frontend needs a way of communicating its requirements on the +machine that processes a job without knowing anything about the available +workers. A key-value structure is suitable for representing such requirements. + +With respect to these constraints, and because the analysis and design of a more +sophisticated solution was declared out of scope of our project assignment, a +rather simple scheduling algorithm was chosen. The broker shall maintain a queue +of available workers. When assigning a job, it traverses this queue and chooses +the first machine that matches the requirements of the job. This machine is then +moved to the end of the queue. + +Presented algorithm results in a simple round-robin load balancing strategy, +which should be sufficient for small-scale deployments (such as a single +university). However, with a large amount of jobs, some workers will easily +become overloaded. The implementation must allow for a simple replacement of the +load balancing strategy so that this problem can be solved in the near future. + +#### Forwarding jobs + +Information about a job can be divided in two disjoint parts -- what the worker +needs to know to process it and what the broker needs to forward it to the +correct worker. It remains to be decided how this information will be +transferred to its destination. + +It is technically possible to transfer all the data required by the worker at +once through the broker. This package could contain submitted files, test +data, requirements on the worker, etc. A drawback of this solution is that +both submitted files and test data can be rather large. Furthermore, it is +likely that test data would be transferred many times. + +Because of these facts, we decided to store data required by the worker using a +shared storage space and only send a link to this data through the broker. This +approach leads to a more efficient network and resource utilization (the broker +doesn't have to process data that it doesn't need), but also makes the job +submission flow more complicated. + +#### Further requirements + +The broker can be viewed as a central point of the backend. While it has only +two primary, closely related responsibilities, other requirements have arisen +(forwarding messages about job evaluation progress back to the frontend) and +will arise in the future. To facilitate such requirements, its architecture +should allow simply adding new communication flows. It should also be as +asynchronous as possible to enable efficient communication with external +services, for example via HTTP. ### Worker @@ -693,7 +755,7 @@ this in all kind of projects. This means that worker should be able to send ping messages even during execution. So worker has to be divided into two separate parts, the one which will handle communication with broker and the another which will execute jobs. The easiest solution is to have these parts in separate -threads which somehow tightly communicates with each other. For inner process +threads which somehow tightly communicates with each other. For inter process communication there can be used numerous technologies, from shared memory to condition variables or some kind of in-process messages. Already used library ZeroMQ is possible to provide in-process messages working on the same principles @@ -763,7 +825,7 @@ represent particular folders. Marks or signs can have form of some kind of special strings which can be called variables. These variables then can be used everywhere where filesystems paths are used within configuration file. This will solve problem with specific worker environment and specific hierarchy of -directories. Final form of variables is ${...} where triple dot is textual +directories. Final form of variables is \${...} where triple dot is textual description. This format was used because of special dollar sign character which cannot be used within filesystem path, braces are there only to border textual description of variable. From 26f1c1b26982a20ad6c4d9290cafe6a68504172a Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Sun, 8 Jan 2017 12:14:14 +0100 Subject: [PATCH 09/46] Backend communication --- Rewritten-docs.md | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 80f1394..14f2776 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -714,9 +714,40 @@ interesting observations will be discussed in this chapter. ### General communication -@todo: what type of communication within backend could be used, mention some -frameworks, queue managers, protocols, which was considered - why we don't use -CORBA, RabbitMQ and why is ZeroMQ great +Overall design of the project is discussed above. There are bunch of components +with their own responsibility. Important thing to design is communication of +these components. All we can count with is that they are connected by network. + +To choose a suitable protocol, there are some additional requirements that +should be met: + +- reliability -- if a message is sent between components, the protocol has to + ensure that it is received by target component +- working over IP protocol +- multi-platform and multi-language usage + +TCP/IP protocol meets these conditions, however it is quite low level and +working with it usually requires working with platform dependent non-object API. +Often way to reflect these reproaches is to use some framework which provides +better abstraction and more suitable API. We decided to go this way, so the +following options are considered: + +- CORBA -- Corba is a well known framework for remote object invocation. There + are multiple implementations for almost every known programming language. It + fits nicely into object oriented programming environment. +- RabbitMQ -- RabbitMQ is a messaging framework written in Erlang. It has + bindings to huge number of languages and large community. Also, it is capable + of routing requests, which could be handy feature for job loadbalancing. +- ZeroMQ -- ZeroMQ is another messaging framework, but instead of creating + separate service this is a small library which can be embedded into own + projects. It is written in C++ with huge number of bindings. + +We like CORBA, but our system should be more loosely-coupled, so (asynchronous) +messaging is better approach in our minds. RabbitMQ seems nice with great +advantage of routing capability, but it is quite heavy service written in +language no one from the team knows, so we do not like it much. ZeroMQ is the +best option for us. However, all of the three options would have been possible +to use. Frontend communication follows the choice, that ReCodEx should be primary a web application. The communication protocol has to reflect client-server From c701f2329ebb93780eb3110000d0f08f124e147f Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Sun, 8 Jan 2017 14:14:58 +0100 Subject: [PATCH 10/46] Instances and licenses --- Rewritten-docs.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 14f2776..928b7c9 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -504,7 +504,22 @@ required to get a credit. These points can be get by solving assigned exercises. To visualy show users if they already have enough points, ReCodEx groups supports setting this limit. There are two equal ways how to set a limit -- absolute value or relative value to maximum. The latter way seems nicer, so it -is implemented. The relative value is set in percents and is called threashold. +is implemented. The relative value is set in percents and is called threshold. + +Our university has a few partner grammar schools. There were an idea, that they +could use CodEx for teaching informatics classes. To make the setup simple for +them, all the software and hardware would be provided by university and hosted +in their datacentre. However, CodEx were not prepared to support this kind of +usage and no one had time to manage a separate instance. With ReCodEx it is +possible to offer hosted environment as a service to other subjects. The concept +we figured out is based on user and group separation inside the system. There +are multiple _instances_ in the system, which means unit of separation. Each +instance has own set of users and groups, exercises can be optionally shared. +Evaluation backend is common for all instances. To keep track of active +instances and paying customers, each instance must have a valid _license_ to +allow users submit their solutions. License is granted for defined period of +time and can be revoked in advance if the subject do not keep approved terms and +conditions. @todo: explain instances why they are useful what they solve and also discuss licenses concept From 3ddd7b1ad4ab4059ec6106544b6081ad8ea668c0 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Sun, 8 Jan 2017 14:32:02 +0100 Subject: [PATCH 11/46] WIP: progress --- Rewritten-docs.md | 171 ++++++++++++++++++++++++---------------------- 1 file changed, 88 insertions(+), 83 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 14f2776..3d5fbab 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -512,9 +512,95 @@ is implemented. The relative value is set in percents and is called threashold. @todo: explain why there is exercise and assignment division, what means what and how they are used -@todo: extended execution pipeline (not just compilation/execution/evaluation) and why it is needed +### Evaluation unit executed by ReCodEx + +One of the bigger requests for the new system is to support a complex +configuration of execution pipeline. The idea comes from lecturers of Compiler +principles class who want to migrate their semi-manual evaluation process to +CodEx. Unfortunately, CodEx is not capable of such complicated exercise setup. +None of evaluation systems we found can handle such task, so design from +scratch is needed. + +There are two main approaches to design a complex execution configuration. It +can be composed of small amount of relatively big components or much more small +tasks. Big components are easy to write and whole configuration is reasonably +small. The components are designed for current problems, so it is not scalable +enough for pleasant future usage. This can be solved by introducing small set of +single-purposed tasks which can be composed together. The whole configuration is +then quite bigger, but with great adaptation ability for new conditions and also +less amount of work programming them. For better user experience, configuration +generators for some common cases can be introduced. + +ReCodEx target is to be continuously developed and used for many years, so the +smaller tasks are the right choice. Observation of CodEx system shows that +only a few tasks are needed. In extreme case, only one task is enough -- execute +a binary. However, for better portability of configurations along different +systems it is better to implement reasonable subset of operations directly +without calling system provided binaries. These operations are copy file, create +new directory, extract archive and so on, altogether called internal tasks. +Another benefit from custom implementation of these tasks is guarantied safety, +so no sandbox needs to be used as in external tasks case. + +For a job evaluation, the tasks needs to be executed sequentially in a specified +order. The idea of running independent tasks in parallel is bad because exact +time measurement needs controlled environment on target computer with +minimization of interrupts by other processes. It seems that connecting tasks +into directed acyclic graph (DAG) can handle all possible problem cases. None of +the authors, supervisors and involved faculty staff can think of a problem that +cannot be decomposed into tasks connected in a DAG. The goal of evaluation is +to satisfy as many tasks as possible. During execution there are sometimes +multiple choices of next task. To control that, each task can have a priority, +which is used as a secondary ordering criterion. For better understanding, here +is a small example. + +![Task serialization](https://github.com/ReCodEx/wiki/raw/master/images/Assignment_overview.png) + +The _job root_ task is imaginary single starting point of each job. When the +_CompileA_ task is finished, the _RunAA_ task is started (or _RunAB_, but should +be deterministic by position in configuration file -- tasks stated earlier +should be executed earlier). The task priorities guaranties, that after +_CompileA_ task all dependent tasks are executed before _CompileB_ task (they +have higher priority number). For example this is useful to control which files +are present in a working directory at every moment. To sum up, there are 3 +ordering criteria: dependencies, then priorities and finally position of task in +configuration. Together, they define a unambiguous linear ordering of all tasks. + +For grading there are several important tasks. First, tasks executing submitted +code need to be checked for time and memory limits. Second, outputs of judging +tasks need to be checked for correctness (represented by return value or by data +on standard output) and should not fail on time or memory limits. This division +can be transparent for backend, each task is executed the same way. But frontend +must know which tasks from whole job are important and what is their kind. It is +reasonable, to keep this piece of information alongside the tasks in job +configuration, so each task can have a label about its purpose. Unlabeled tasks +have an internal type _inner_. There are four categories of tasks: + +- _initiation_ -- setting up the environment, compiling code, etc.; for users + failure means error in their sources which are not compatible with running it + with examination data +- _execution_ -- running the user code with examination data, must not exceed + time and memory limits; for users failure means wrong design, slow data + structures, etc. +- _evaluation_ -- comparing user and examination outputs; for user failure means + that the program does not compute the right results +- _inner_ -- no special meaning for frontend, technical tasks for fetching and + copying files, creating directories, etc. + +Each job is composed of multiple tasks of these types which are semantically +grouped into tests. A test can represent one set of examination data for user +code. To mark the grouping, another task label can be used. Each test must have +exactly one _evaluation_ task (to show success or failure to users) and +arbitrary number of tasks with other types. + +### Evaluation progress state + +Users surely want to know progress state of their submitted solution this kind of functionality comes particularly handy in long duration exercises. Because of reporting progress users have immediate knowledge if anything goes wrong, not mention psychological effect that whole system and its parts are working and doing something. That is why this feature was considered from beginning but there are multiple ways how to look at it in particular. + +The very first idea would be to provide progress state based on done messages from compilation, execution and evaluation. Which is something what a lot of evaluation systems are providing. These information are high level enough for users and they probably know what is going on and executing right now. If compilation fails users know that their solution is not compilable, if execution fails there were some problems with their program. The clarity of this kind of progress state is nice and understandable. But as we learnt ReCodEx has to have more advanced execution pipeline there can be more compilations or more executions. And in addition parts of the system which ensure execution of users solutions do not have to precisely know what they are executing at the moment. This kind of information may be meaningless for them. -@todo: progress state, how it can be done and displayed to user, why random messages +That is why another solution of progress state was considered. As we know right now one of the best ways how to ensure generality is to have jobs with single-purpose tasks. These tasks can be anything, some internal operation or execution of external and sandboxed program. Based on this there is one very simple solution how to provide general progress state which should be independent on task types. We know that job has some number of tasks which has to be executed so we can send state info after execution of every task. And that is how we get percentual completion of an execution. Yes, it is kind of boring and standard way but on top of that there can be built something else and more appealing to users. + +So displaying progress to users can be done numerous ways. We have percentual completion which is of course begging for simple solution which is displaying user only the percentage. ... todo done this @todo: how to display generally all outputs of executed programs to user (supervisor, student), what students can or cannot see and why @@ -623,87 +709,6 @@ will be introduced separately and covered in more detail. The communication protocol between these two logical parts will be described as well. -### Evaluation unit executed by ReCodEx - -One of the bigger requests for the new system is to support a complex -configuration of execution pipeline. The idea comes from lecturers of Compiler -principles class who want to migrate their semi-manual evaluation process to -CodEx. Unfortunately, CodEx is not capable of such complicated exercise setup. -None of evaluation systems we found can handle such task, so design from -scratch is needed. - -There are two main approaches to design a complex execution configuration. It -can be composed of small amount of relatively big components or much more small -tasks. Big components are easy to write and whole configuration is reasonably -small. The components are designed for current problems, so it is not scalable -enough for pleasant future usage. This can be solved by introducing small set of -single-purposed tasks which can be composed together. The whole configuration is -then quite bigger, but with great adaptation ability for new conditions and also -less amount of work programming them. For better user experience, configuration -generators for some common cases can be introduced. - -ReCodEx target is to be continuously developed and used for many years, so the -smaller tasks are the right choice. Observation of CodEx system shows that -only a few tasks are needed. In extreme case, only one task is enough -- execute -a binary. However, for better portability of configurations along different -systems it is better to implement reasonable subset of operations directly -without calling system provided binaries. These operations are copy file, create -new directory, extract archive and so on, altogether called internal tasks. -Another benefit from custom implementation of these tasks is guarantied safety, -so no sandbox needs to be used as in external tasks case. - -For a job evaluation, the tasks needs to be executed sequentially in a specified -order. The idea of running independent tasks in parallel is bad because exact -time measurement needs controlled environment on target computer with -minimization of interrupts by other processes. It seems that connecting tasks -into directed acyclic graph (DAG) can handle all possible problem cases. None of -the authors, supervisors and involved faculty staff can think of a problem that -cannot be decomposed into tasks connected in a DAG. The goal of evaluation is -to satisfy as many tasks as possible. During execution there are sometimes -multiple choices of next task. To control that, each task can have a priority, -which is used as a secondary ordering criterion. For better understanding, here -is a small example. - -![Task serialization](https://github.com/ReCodEx/wiki/raw/master/images/Assignment_overview.png) - -The _job root_ task is imaginary single starting point of each job. When the -_CompileA_ task is finished, the _RunAA_ task is started (or _RunAB_, but should -be deterministic by position in configuration file -- tasks stated earlier -should be executed earlier). The task priorities guaranties, that after -_CompileA_ task all dependent tasks are executed before _CompileB_ task (they -have higher priority number). For example this is useful to control which files -are present in a working directory at every moment. To sum up, there are 3 -ordering criteria: dependencies, then priorities and finally position of task in -configuration. Together, they define a unambiguous linear ordering of all tasks. - -For grading there are several important tasks. First, tasks executing submitted -code need to be checked for time and memory limits. Second, outputs of judging -tasks need to be checked for correctness (represented by return value or by data -on standard output) and should not fail on time or memory limits. This division -is transparent for backend, each task is executed the same way. But frontend -must know which tasks from whole job are important and what is their kind. It is -reasonable, to keep this piece of information alongside the tasks in job -configuration, so each task can have a label about its purpose. Unlabeled tasks -have an internal type _inner_. There are four categories of tasks: - -- _initiation_ -- setting up the environment, compiling code, etc.; for users - failure means error in their sources which are not compatible with running it - with examination data -- _execution_ -- running the user code with examination data, must not exceed - time and memory limits; for users failure means wrong design, slow data - structures, etc. -- _evaluation_ -- comparing user and examination outputs; for user failure means - that the program does not compute the right results -- _inner_ -- no special meaning for frontend, technical tasks for fetching and - copying files, creating directories, etc. - -Each job is composed of multiple tasks of these types which are semantically -grouped into tests. A test can represent one set of examination data for user -code. To mark the grouping, another task label can be used. Each test must have -exactly one _evaluation_ task (to show success or failure to users) and -arbitrary number of tasks with other types. - - ## Implementation analysis When developing a project like ReCodEx there has to be some discussion over From 9c50d3464fec34fc7c5e6c345df331f12cffd1cd Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Sun, 8 Jan 2017 17:22:10 +0100 Subject: [PATCH 12/46] Progress --- Rewritten-docs.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 383ab65..59a09ce 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -615,7 +615,9 @@ The very first idea would be to provide progress state based on done messages fr That is why another solution of progress state was considered. As we know right now one of the best ways how to ensure generality is to have jobs with single-purpose tasks. These tasks can be anything, some internal operation or execution of external and sandboxed program. Based on this there is one very simple solution how to provide general progress state which should be independent on task types. We know that job has some number of tasks which has to be executed so we can send state info after execution of every task. And that is how we get percentual completion of an execution. Yes, it is kind of boring and standard way but on top of that there can be built something else and more appealing to users. -So displaying progress to users can be done numerous ways. We have percentual completion which is of course begging for simple solution which is displaying user only the percentage. ... todo done this +So displaying progress to users can be done numerous ways. We have percentual completion which is of course begging for simple solution which is displaying only the percentage or some kind of standard graphical progress bar. But that is too mainstream lets try something else. Very good idea is to have some kind of puzzled image or images which will be composed together according to progress. Nice way but kind of challenging if we do not have designer around. Another original solution is to have database of random kind-of-funny statements which will be displayed every time task is completed. It is easy enough for implementation and even for making up these messages and it is quite new and original. That is why this last solution was chosen for displaying progress state. + +### Results of evaluation @todo: how to display generally all outputs of executed programs to user (supervisor, student), what students can or cannot see and why @@ -625,6 +627,8 @@ So displaying progress to users can be done numerous ways. We have percentual co @todo: discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why +### Persistence + @todo: where is kept the state (MariaDB) @todo: and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks From c420975242bb84eebc3e8f18b7b32cd79896451b Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Sun, 8 Jan 2017 17:23:09 +0100 Subject: [PATCH 13/46] format --- Rewritten-docs.md | 49 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 7 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 59a09ce..57fa6bd 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -609,13 +609,47 @@ arbitrary number of tasks with other types. ### Evaluation progress state -Users surely want to know progress state of their submitted solution this kind of functionality comes particularly handy in long duration exercises. Because of reporting progress users have immediate knowledge if anything goes wrong, not mention psychological effect that whole system and its parts are working and doing something. That is why this feature was considered from beginning but there are multiple ways how to look at it in particular. - -The very first idea would be to provide progress state based on done messages from compilation, execution and evaluation. Which is something what a lot of evaluation systems are providing. These information are high level enough for users and they probably know what is going on and executing right now. If compilation fails users know that their solution is not compilable, if execution fails there were some problems with their program. The clarity of this kind of progress state is nice and understandable. But as we learnt ReCodEx has to have more advanced execution pipeline there can be more compilations or more executions. And in addition parts of the system which ensure execution of users solutions do not have to precisely know what they are executing at the moment. This kind of information may be meaningless for them. - -That is why another solution of progress state was considered. As we know right now one of the best ways how to ensure generality is to have jobs with single-purpose tasks. These tasks can be anything, some internal operation or execution of external and sandboxed program. Based on this there is one very simple solution how to provide general progress state which should be independent on task types. We know that job has some number of tasks which has to be executed so we can send state info after execution of every task. And that is how we get percentual completion of an execution. Yes, it is kind of boring and standard way but on top of that there can be built something else and more appealing to users. - -So displaying progress to users can be done numerous ways. We have percentual completion which is of course begging for simple solution which is displaying only the percentage or some kind of standard graphical progress bar. But that is too mainstream lets try something else. Very good idea is to have some kind of puzzled image or images which will be composed together according to progress. Nice way but kind of challenging if we do not have designer around. Another original solution is to have database of random kind-of-funny statements which will be displayed every time task is completed. It is easy enough for implementation and even for making up these messages and it is quite new and original. That is why this last solution was chosen for displaying progress state. +Users surely want to know progress state of their submitted solution this kind +of functionality comes particularly handy in long duration exercises. Because of +reporting progress users have immediate knowledge if anything goes wrong, not +mention psychological effect that whole system and its parts are working and +doing something. That is why this feature was considered from beginning but +there are multiple ways how to look at it in particular. + +The very first idea would be to provide progress state based on done messages +from compilation, execution and evaluation. Which is something what a lot of +evaluation systems are providing. These information are high level enough for +users and they probably know what is going on and executing right now. If +compilation fails users know that their solution is not compilable, if execution +fails there were some problems with their program. The clarity of this kind of +progress state is nice and understandable. But as we learnt ReCodEx has to have +more advanced execution pipeline there can be more compilations or more +executions. And in addition parts of the system which ensure execution of users +solutions do not have to precisely know what they are executing at the moment. +This kind of information may be meaningless for them. + +That is why another solution of progress state was considered. As we know right +now one of the best ways how to ensure generality is to have jobs with +single-purpose tasks. These tasks can be anything, some internal operation or +execution of external and sandboxed program. Based on this there is one very +simple solution how to provide general progress state which should be +independent on task types. We know that job has some number of tasks which has +to be executed so we can send state info after execution of every task. And that +is how we get percentual completion of an execution. Yes, it is kind of boring +and standard way but on top of that there can be built something else and more +appealing to users. + +So displaying progress to users can be done numerous ways. We have percentual +completion which is of course begging for simple solution which is displaying +only the percentage or some kind of standard graphical progress bar. But that is +too mainstream lets try something else. Very good idea is to have some kind of +puzzled image or images which will be composed together according to progress. +Nice way but kind of challenging if we do not have designer around. Another +original solution is to have database of random kind-of-funny statements which +will be displayed every time task is completed. It is easy enough for +implementation and even for making up these messages and it is quite new and +original. That is why this last solution was chosen for displaying progress +state. ### Results of evaluation @@ -631,6 +665,7 @@ So displaying progress to users can be done numerous ways. We have percentual co @todo: where is kept the state (MariaDB) + @todo: and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks From 568464e12af5d7f35a5a2c7f74b9bda0ad6d03e7 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Sun, 8 Jan 2017 18:48:24 +0100 Subject: [PATCH 14/46] Exercises described --- Rewritten-docs.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 57fa6bd..e5e9a69 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -499,7 +499,7 @@ is to allow anyone join the group with supplementary confirmation of supervisors. It has no additional benefits, so approach with public and private groups is implemented. -Supervisors using CodEx in their labs usually set minimum amout of points +Supervisors using CodEx in their labs usually set minimum amount of points required to get a credit. These points can be get by solving assigned exercises. To visualy show users if they already have enough points, ReCodEx groups supports setting this limit. There are two equal ways how to set a limit -- @@ -521,11 +521,21 @@ allow users submit their solutions. License is granted for defined period of time and can be revoked in advance if the subject do not keep approved terms and conditions. -@todo: explain instances why they are useful what they solve and also discuss licenses concept - -@todo: exercise, what is it why we need it - -@todo: explain why there is exercise and assignment division, what means what and how they are used +The main work for the system is to evaluate programming exercises. The exercise +is quite similar to homework assignment during school labs. When a homework is +assigned, two things are important to know for users: + +- description of the problem +- metadata -- when and whom to submit solutions, grading scale, penalties, etc. + +To reflect this idea teachers and students are already familiar with, we decided +to keep separation between problem itself (_exercise_) and its _assignment_. +Exercise only describes one problem and provides testing data with description +of how to evaluate it. In fact, it is template for assignments. Assignment then +contains data from its exercise and additional metadata, which can be different +for every assignment of the same exercise. This separation is natural for all +users, in CodEx it is implemented in similar way and no other considerable +solution was found. ### Evaluation unit executed by ReCodEx From 0e4f85e7b974f1fb3978aa557d52efd696820418 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Sun, 8 Jan 2017 19:28:07 +0100 Subject: [PATCH 15/46] Typos --- Rewritten-docs.md | 79 ++++++++++++++++++++++++----------------------- 1 file changed, 40 insertions(+), 39 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index e5e9a69..5735bc3 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -145,7 +145,7 @@ by instructed privileged users. Assigning an exercise to a group means to choose one of the available exercises and specifying additional properties. An assignment has a deadline (optionally a second deadline), a maximum amount of points, a configuration for calculating the final score, a maximum number of -submissions, and a list of supported runtime environements (e.g., programming +submissions, and a list of supported runtime environments (e.g., programming languages) including specific time and memory limits for each one. Typical use cases for supported user roles are illustrated on following UML @@ -221,7 +221,7 @@ addons (mostly administrative features). specific properties set (deadlines, etc.) - user can see a list of submitted solutions for each assignment with corresponding results -- teachers can specify way of computation grading points which will be awarted +- teachers can specify way of computation grading points which will be awarded to the students depending on the quality of his/her solution for each assignment extra - teachers can view detailed data about their students (users of a their groups) @@ -283,7 +283,7 @@ quickly understand the code and make changes. To find out the current state in the field of automatic grading systems we did a short market survey on the field of automatic grading systems at universities, -programming contests, and possibly other places where similat tools are +programming contests, and possibly other places where similar tools are available. @@ -458,7 +458,7 @@ The application interacts with users. From the project assignment is clear, that the system has to keep personalized data about users and adapt presented content according to this knowledge. User data cannot be publicly visible, so that implies necessity of user authentication. There are several way of -auhtentication user in web applications. HTTP basic authentication has a few +authentication user in web applications. HTTP basic authentication has a few drawbacks like sending plain credentials in every request or no logout option, so it is not recommended to use. Using cookies is possible, but it is susceptible to various types of attacks including stealing and also brings state @@ -478,30 +478,30 @@ least power, basically can just view assignments and submit solutions. Supervisors have more authority, so they can create exercises and assignments, view results of students etc. From the university organization, one possible level could be introduced, _course guarantor_. However, from real experience all -duties related with lecturing of labs are already associtated with supervisors, +duties related with lecturing of labs are already associated with supervisors, so this role seems not so useful. In addition, no one requested more than three level privilege scheme. School labs are lessons for some students lead by one (or sometimes two or -three) supervisors. Students have the same homeworks and supervisors are -evaluating them. This organization has to be carried into the new system. -Counterpart to real labs are virtual groups. This concept was already discussed -in previous chapter including need for hierarchical structure of groups. Right -for attending labs has only a person, who is student of the university and is -recorded in university information system. To allow restriction of group members -in ReCodEx, there two type of groups -- _public_ and _private_. Public groups -are open for every registered users, but to become a member of private group one -of its supervisors have to add that user. This could be done automatically at -beginning of the term with data from information system, but unfortunately there -is no such API yet. However, creating this API is now considered by university -leadership. Another just as good solution for restricting membership of a group -is to allow anyone join the group with supplementary confirmation of -supervisors. It has no additional benefits, so approach with public and private -groups is implemented. +three) supervisors. Students have the same homework and supervisors are +evaluating its solutions. This organization has to be carried into the new +system. Counterpart to real labs are virtual groups. This concept was already +discussed in previous chapter including need for hierarchical structure of +groups. Right for attending labs has only a person, who is student of the +university and is recorded in university information system. To allow +restriction of group members in ReCodEx, there two type of groups -- _public_ +and _private_. Public groups are open for every registered users, but to become +a member of private group one of its supervisors have to add that user. This +could be done automatically at beginning of the term with data from information +system, but unfortunately there is no such API yet. However, creating this API +is now considered by university leadership. Another just as good solution for +restricting membership of a group is to allow anyone join the group with +supplementary confirmation of supervisors. It has no additional benefits, so +approach with public and private groups is implemented. Supervisors using CodEx in their labs usually set minimum amount of points required to get a credit. These points can be get by solving assigned exercises. -To visualy show users if they already have enough points, ReCodEx groups +To visually show users if they already have enough points, ReCodEx groups supports setting this limit. There are two equal ways how to set a limit -- absolute value or relative value to maximum. The latter way seems nicer, so it is implemented. The relative value is set in percents and is called threshold. @@ -1010,7 +1010,7 @@ worker) and how they can be accessed and written into configuration. For this kind of task we have to introduce some kind of marks or signs which will represent particular folders. Marks or signs can have form of some kind of special strings which can be called variables. These variables then can be used -everywhere where filesystems paths are used within configuration file. This will +everywhere where filesystem paths are used within configuration file. This will solve problem with specific worker environment and specific hierarchy of directories. Final form of variables is \${...} where triple dot is textual description. This format was used because of special dollar sign character which @@ -1129,7 +1129,7 @@ worker. It should be like the last salvation in case everything else goes wrong. There are numerous ways how to approach sandboxing on different platforms, describing all possible approaches is out of scope of this document. Instead of that have a look at some of the features which are certainly needed for ReCodEx -and propose some particular sandboxes implementations on linux or Windows. +and propose some particular sandboxes implementations on Linux or Windows. General purpose of sandbox is safely execute software in any form, from scripts to binaries. Various sandboxes differ in how safely are they and what limiting @@ -1169,15 +1169,15 @@ sandbox solution for ReCodEx. With this being said we can safely state that designing and implementing new general sandbox for Windows is out of scope of this project. -New general sandbox for Windows is out of bussiness but what about more +New general sandbox for Windows is out of business but what about more specialized solution used for instance only for C#. CLR as a virtual machine and runtime environment has a pretty good security support for restrictions and -separation which is also transfered to C#. This makes it quite easy to implement -simple sandbox within C# but suprisingly there cannot be found some well known -general purpose implementations. As said in previous paragraph implementing our -own solution is out of scope of project there is simple not enough time. But C# -sandbox is quite good topic for another project for example semestral project -for C# course so it might be written and integrated in future. +separation which is also transferred to C#. This makes it quite easy to +implement simple sandbox within C# but surprisingly there cannot be found some +well known general purpose implementations. As said in previous paragraph +implementing our own solution is out of scope of project there is simple not +enough time. But C# sandbox is quite good topic for another project for example +term project for C# course so it might be written and integrated in future. ### Fileserver @@ -1187,7 +1187,7 @@ for C# course so it might be written and integrated in future. @todo: what can be stored on fileserver -@todo: how can jobs be stored on fileserver, mainly mention that it is nonsence to store inputs and outputs within job archive +@todo: how can jobs be stored on fileserver, mainly mention that it is nonsense to store inputs and outputs within job archive ### Monitor @@ -1267,7 +1267,7 @@ and we were able to develop all the features we needed without learning to use a new technology. Since the number of features was quite high and needed to meet a strict deadline. This does not mean that we would find all the other technologies superior to PHP in all other aspects - PHP 7 is a mature language -with a huge comunity and a wide range of tools, libraries, and frameworks. +with a huge community and a wide range of tools, libraries, and frameworks. We decided to use an ORM framework to manage the database, namely the widely used PHP ORM Doctrine 2. This framework has a robust abstraction layer DBAL so @@ -1283,19 +1283,20 @@ patterns used in this framework (e.g., dependency injection, authentication, routing). There is a good extension for the Nette framework which makes usage of Doctrine 2 very straightforward. -@todo: what database can be used, how it is mapped and used within code - -@todo: authentication, some possibilities and describe used jwt +@todo: authentication, some possibilities and describe used jwt (mentioned in +basic concepts, maybe elaborate more or just remove this item from todo list) @todo: solution of forgotten password, why this in particular @todo: rest api is used for report of backend state and errors, describe why and other possibilities (separate component) -@todo: what files are stored in api, why there are duplicates among api and fileserver +@todo: mail reports - to users, admins -@todo: why are there instances and for which they can be used for, describe licences and its implementation +@todo: what files are stored in api, why there are duplicates among api and fileserver -@todo: groups and hierarchy, describe arbitrary nesting which should be possible within instance and how it is implemented and how it could be implemented +@todo: groups and hierarchy, describe arbitrary nesting which should be possible +within instance and how it is implemented and how it could be implemented +(describe only implementation if it is something what to say) @todo: where is stored which workers can be used by supervisors and which runtimes are available, describe possibilities and why is not implemented automatic solution @@ -1344,7 +1345,7 @@ for the technical description of the components) ### Monitor -@todo: not necessary component which can be ommited, proxy-like service +@todo: not necessary component which can be omitted, proxy-like service ## Backend internal communication From 320dff7817b0f4ca78591d7233f94b6c7faca8d1 Mon Sep 17 00:00:00 2001 From: Teyras Date: Sun, 8 Jan 2017 19:36:15 +0100 Subject: [PATCH 16/46] more broker analysis --- Rewritten-docs.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 5735bc3..6b648ea 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -866,10 +866,24 @@ distributing jobs that it receives from the frontend between them. #### Worker management -@todo initialization - broker is fixed, workers connect to it - -@todo heartbeating - workers send ping, the inverse is possible, too (doesn't -really matter) +It is intended for the broker to be a fixed part of the backend infrastructure +to which workers connect at will. Thanks to this design, workers can be added +and removed when necessary (and possibly in an automated fashion), without +changing the configuration of the broker. An alternative solution would be +configuring a list of workers before startup, thus making them passive in the +communication (in the sense that they just wait for incoming jobs instead of +connecting to the broker). However, this approach comes with a notable +administration overhead -- in addition to starting a worker, the administrator +would have to update the worker list. + +Worker management must also take into account the possibility of worker +disconnection, either because of a network or software failure (or termination). +A common way to detect such events in distributed systems is to periodically +send short messages to other nodes and expect a response. When these messages +stop arriving, we presume that the other node encountered a failure. Both the +broker and workers can be made responsible for initiating these exchanges and it +seems that there are no differences stemming from this choice. We decided that +the workers will be the active party that initiates the exchange. #### Scheduling From bc46d8dcf67d8fa28bde47fefb1f952e20a1179d Mon Sep 17 00:00:00 2001 From: Jan Buchar Date: Mon, 9 Jan 2017 08:54:08 +0100 Subject: [PATCH 17/46] fileserver analysis introduction --- Rewritten-docs.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 6b648ea..dc56e24 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1195,7 +1195,14 @@ term project for C# course so it might be written and integrated in future. ### Fileserver -@todo: fileserver and why is separated +The fileserver provides access to a shared storage space that contains files +submitted by students, supplementary files such as test inputs and outputs and +results of evaluation. This functionality can be easily separated from the rest +of the backend features, which led to designing the fileserver as a +standalone component. Such design helps encapsulate the details of how the files +are stored (e.g. on a file system, in a database or using a cloud storage +service), while also making it possible to share the storage between multiple +ReCodEx frontends. @todo: mention hashing on fileserver and why this approach was chosen From a3fc1c69a7de8dd3127ae35684be5f8fd11e720c Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Mon, 9 Jan 2017 16:58:04 +0100 Subject: [PATCH 18/46] Leave out fileserver hashing from worker desc --- Rewritten-docs.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index dc56e24..fa0fa21 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1061,10 +1061,9 @@ chosen and implemented. #### Caching mechanism -As described in fileserver section stored supplementary files have special -filenames which reflects hashes of their content. As such there are no -duplicates stored in fileserver. Worker can use feature too and caches these -files for some while and saves precious bandwidth. This means there has to be +Worker can use caching mechanism based on files from fileserver under one +condition, provided files has to have unique name. If uniqueness is fulfilled +then precious bandwidth can be saved using cache. This means there has to be system which can download file, store it in cache and after some time of inactivity delete it. Because there can be multiple worker instances on some particular server it is not efficient to have this system in every worker on its @@ -1076,7 +1075,7 @@ exactly needed. But mainly it would be single-failure component if it would stop working it is quite problem. So there was chosen another solution which assumes worker has access to specified cache folder, to this folder worker can download supplementary files and copy them from here. This means every worker has the -\possibility to maintain downloads to cache, but what is worker not able to +possibility to maintain downloads to cache, but what is worker not able to properly do is deletion of unused files after some time. For that single-purpose component is introduced which is called 'cleaner'. It is simple script executed within cron which is able to delete files which were unused for some time. From ee4a07836e12f325556c6cf16407d49282bbf700 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Mon, 9 Jan 2017 19:00:55 +0100 Subject: [PATCH 19/46] analysis - outputs concept --- Rewritten-docs.md | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index fa0fa21..1b57d50 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -663,7 +663,35 @@ state. ### Results of evaluation -@todo: how to display generally all outputs of executed programs to user (supervisor, student), what students can or cannot see and why +There are lot of things which deserves discussion concerning results of +evaluation, how they should be displayed, what should be visible or not and also +some what kind of reward for users solutions should be chosen. + +At first lets focus on all kinds of outputs from executed programs within job. +Out of discussion is that supervisors should be able to view almost all outputs +from solutions if they choose them to be visible and recorded. This feature is +critical in debugging either whole exercises or users solutions. But should it +be default behaviour to record every output? Absolutely not, supervisor should +have choice to turn it on but defaults has to be not record it. Even without +this functionality can be file base around whole ReCodEx system quite large and +on top of that outputs from executed programs can be sometimes very extensive. +Storing this amount of data is purely nonsense to every solution. But if +requested by supervisor then this feature should be available. + +Question what should regular users see from execution of their solution is more +interesting. Simple answer is of course that they should not see anything which +is partly true. Outputs from their programs can be anything and users can +somehow analyse inputs or even redirect them to output. So outputs from +execution should not be visible at all or under very special circumstances. But +what about compilation or other kinds of initiations. Well, this is another +story, it really depends on the particular case. But generally it is quite +harmless to display user some kind of compilation error which can help a lot +during troubleshooting. Of course again this kind of functionality should be +configurable by supervisors and disabled by default. There is also the last kind +of tasks which can output some information which is evaluation tasks. Output of +these tasks is somehow important to whole system and again can contain some +information about inputs or reference outputs. So outputs of evaluation tasks +should not also be visible to regular users. @todo: judges, discuss what they possibly can do and what it can be used for (returning for instance 2 numbers instead of 1 and why we return just one) From 5d68a7197f341bb703671f1e9800c6f793991efb Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Mon, 9 Jan 2017 22:37:29 +0100 Subject: [PATCH 20/46] Persistence --- Rewritten-docs.md | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index fa0fa21..4c4d1af 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -665,18 +665,44 @@ state. @todo: how to display generally all outputs of executed programs to user (supervisor, student), what students can or cannot see and why -@todo: judges, discuss what they possibly can do and what it can be used for (returning for instance 2 numbers instead of 1 and why we return just one) - @todo: discuss points assigned to solution, why are there bonus points, explain minimal point threshold @todo: discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why ### Persistence -@todo: where is kept the state (MariaDB) - - -@todo: and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks +Previous parts of analysis show that the system has to keep some state. This +could be user settings, group membership, evaluated assignments and so on. The +data have to be kept across restart, so persistence is important decision +factor. There are several ways how to save structured data: + +- plain files +- NoSQL database +- relational database + +Another important factor is amount and size of stored data. Our guess is about +1000 users, 100 exercises, 200 assignments per year and 400000 unique solutions +per year. The data are mostly structured and there are a lot of them with the +same format. For example, there is a thousand of users and each one has the same +values -- name, email, age, etc. These kind of data are relatively small, name +and email are short strings, age is an integer. Considering this, relational +databases or formatted plain files (CSV for example) fits best for them. +However, the data often have to support find operation, so they have to be +sorted and allow random access for resolving cross references. Also, addition a +deletion of entries should take reasonable time (at most logaritmic time +complexity to number of saved values). This practicaly excludes plain files, so +relational database is used instead. + +On the other hand, there are some data with no such great structure and much +larger size. These can be evaluation logs, sample input files for exercises or +submited sources by students. Saving this kind of data into relational database +is not suitable, but it is better to keep them as ordinary files or store them +into some kind of NoSQL database. Since they are already files and does not need +to be backed up in multiple copies, it is easier to keep them as ordinary files +in filesystem. Also, this solution is more lightweight and does not require +additional dependencies on third-party software. File can be identified using +its filesystem path or unique index stored as value in relational database. Both +approaches are equaly good, final decission depends on actual case. ## Structure of the project From 73bea479815935e7d73980ace13a4f1b44c411c4 Mon Sep 17 00:00:00 2001 From: Teyras Date: Mon, 9 Jan 2017 23:03:37 +0100 Subject: [PATCH 21/46] fileserver analysis --- Rewritten-docs.md | 48 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 12 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 0778df7..ee74a1c 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1250,18 +1250,42 @@ term project for C# course so it might be written and integrated in future. The fileserver provides access to a shared storage space that contains files submitted by students, supplementary files such as test inputs and outputs and -results of evaluation. This functionality can be easily separated from the rest -of the backend features, which led to designing the fileserver as a -standalone component. Such design helps encapsulate the details of how the files -are stored (e.g. on a file system, in a database or using a cloud storage -service), while also making it possible to share the storage between multiple -ReCodEx frontends. - -@todo: mention hashing on fileserver and why this approach was chosen - -@todo: what can be stored on fileserver - -@todo: how can jobs be stored on fileserver, mainly mention that it is nonsense to store inputs and outputs within job archive +results of evaluation. In other words, it acts as an intermediate node for data +passed between the frontend and the backend. This functionality can be easily +separated from the rest of the backend features, which led to designing the +fileserver as a standalone component. Such design helps encapsulate the details +of how the files are stored (e.g. on a file system, in a database or using a +cloud storage service), while also making it possible to share the storage +between multiple ReCodEx frontends. + +For early releases of the system, we chose to store all files on the file system +-- it is the least complicated solution (in terms of implementation complexity) +and the storage backend can be rather easily migrated to a different technology. + +One of the facts we learned from CodEx is that many exercises share test input +and output files, and also that these files can be rather large (hundreds of +megabytes). A direct consequence of this is that we cannot add these files to +submission archives that are to be downloaded by workers -- the combined size of +the archives would quickly exceed gigabytes, which is impractical. Another +conclusion we made is that a way to deal with duplicate files must be +introduced. + +A simple solution to this problem is storing supplementary files under the +hashes of their content. This ensures that every file is stored only once. On +the other hand, it makes it more difficult to understand what the content of a +file is at a glance, which might prove problematic for the administrator. + +A notable part of the fileserver's work is done by a web server (e.g. listening +to HTTP requests and caching recently accessed files in memory for faster +access). What remains to be implemented is handling requests that upload files +-- student submissions should be stored in archives to facilitate simple +downloading and supplementary exercise files need to be stored under their +hashes. + +We decided to use Python and the Flask web framework. This combination makes it +possible to express the logic in ~100 SLOC and also provides means to run the +fileserver as a standalone service (without a web server), which is useful for +development. ### Monitor From 7fa925fdae9ed35640039f49771abf5e94869e8f Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Tue, 10 Jan 2017 10:41:47 +0100 Subject: [PATCH 22/46] More pictures! --- Rewritten-docs.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index ee74a1c..7218d81 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -812,6 +812,16 @@ server side based on the inputs from the user. This decoupling gives us the ability to create multiple client side tools which may address different needs of the users. +One possible configuration of ReCodEx system is ilustrated on following picture, +where thete is one shared backend with three workers and two separate instances +of whole frontend. This configuration may be suitable for MFF UK -- basic +programming course and KSP competition. But maybe even sharing web API and +fileserver with only custom instances of client (web app or own implementation) +is more likely to be used. Note, that connections between components are not +fully accurate. + +![Overall architecture](https://github.com/ReCodEx/wiki/blob/master/images/Overall_Architecture.png) + The frontend developed as part of this project is a web application created with the needs of the Faculty of Mathematics and Physics of the Charles university in Prague in mind. The users are the students and their teachers, groups correspond @@ -913,6 +923,12 @@ and testing, and it is understood by programmers so it should be easy for a new developer with some experience in client-side applications to get to know with the ReCodEx API and develop a client application. +To sum up, chosen ways of communication inside the ReCodEx system are captured +in the following image. Red connections are through ZeroMQ sockets, blue are +through WebSockets and green are through HTTP(S). + +![Communication schema](https://github.com/ReCodEx/wiki/raw/master/images/Backend_Connections.png) + ### Broker The broker is responsible for keeping track of available workers and From 2192da93c0429ada7587640f1e2d47de72725b28 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Tue, 10 Jan 2017 12:30:13 +0100 Subject: [PATCH 23/46] More todos done --- Rewritten-docs.md | 62 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 14 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 7218d81..4e729c6 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -667,7 +667,7 @@ There are lot of things which deserves discussion concerning results of evaluation, how they should be displayed, what should be visible or not and also some what kind of reward for users solutions should be chosen. -At first lets focus on all kinds of outputs from executed programs within job. +At first let us focus on all kinds of outputs from executed programs within job. Out of discussion is that supervisors should be able to view almost all outputs from solutions if they choose them to be visible and recorded. This feature is critical in debugging either whole exercises or users solutions. But should it @@ -683,19 +683,53 @@ interesting. Simple answer is of course that they should not see anything which is partly true. Outputs from their programs can be anything and users can somehow analyse inputs or even redirect them to output. So outputs from execution should not be visible at all or under very special circumstances. But -what about compilation or other kinds of initiations. Well, this is another -story, it really depends on the particular case. But generally it is quite -harmless to display user some kind of compilation error which can help a lot -during troubleshooting. Of course again this kind of functionality should be -configurable by supervisors and disabled by default. There is also the last kind -of tasks which can output some information which is evaluation tasks. Output of -these tasks is somehow important to whole system and again can contain some -information about inputs or reference outputs. So outputs of evaluation tasks -should not also be visible to regular users. - -@todo: discuss points assigned to solution, why are there bonus points, explain minimal point threshold - -@todo: discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why +that is not so straighforward for compilation or other kinds of initiation. +Well, this is another story, it really depends on the particular case. But +generally it is quite harmless to display user some kind of compilation error +which can help a lot during troubleshooting. Of course again this kind of +functionality should be configurable by supervisors and disabled by default. +There is also the last kind of tasks which can output some information which is +evaluation tasks. Output of these tasks is somehow important to whole system and +again can contain some information about inputs or reference outputs. So outputs +of evaluation tasks should not be visible to regular users too. + +The overall concept of grading solutions was presented earlier. To briefly +remind that, backend returns only exact measured values (used time and memory, +return code of the judging task, ...) and on top of that one value is computed. +The way of this computation can be very different across supervisors, so it has +to be easily extendable. The best way is to provide interface, which can be +implemented and any sort of magic can return the final value. + +We found out several computational possibilities. There is basic arithmetic, +weighted arithmetic, geometric and harmonic mean of results of each test (the +result is boolean succeeded/failed, optionaly has weight), some kind of +interpolation of used amount of time for each test, the same with used memory +amount and surely many others. To keep the project simple, we decided to design +apropriate interface and implement only weighted arithmetic mean computation, +which is used in about 90% of all assignments. Of course, diferent scheme can be +chosen for every assignment and also configured -- for example test weights can +be specified for implemented weighted arithmetic mean. Advanced ways of +computation can be implemented on demand when is a real demand for them. + +To avoid assigning points for insufficient solutions (like only printing "File +error" which is the valid answer in two tests), a minimal point threshold can be +specified. It he solution is to get less points than specified, it will get zero +points instead. This functionality can be empedded into grading computation +algoritm itself, but it would have to be present in each implementation +separately, which is a bit ugly. So, this feature is separated from point +computation. + +Automatic grading cannot reflect all aspects of submitted code. For example, +structuring the code, number and quality of comments and so on. To allow +supervisors bring these manually checked things into grading, there is a concept +of bonus points. They can be positive or negative. Generaly the solution with +the most assigned points is marked for grading that particular solution. +However, if supervisor is not satisfied with student solution (really bad code, +cheating, ...) he/she assigns the student negative bonus points. But to prevent +chosing another solution with more points by the system or even submitting the +same code again which is worth more points by students, supervisor can mark a +particular solution as marked and used for grading instead of solution with the +most points. ### Persistence From a7d95d2811666c35084233bf84faf3ffb03c1f70 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Tue, 10 Jan 2017 14:33:45 +0100 Subject: [PATCH 24/46] Typographic improvement --- Rewritten-docs.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 4e729c6..d2c2e2d 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1113,14 +1113,14 @@ and then folders for temporary files and for results and generally files which should be uploaded back to fileserver with solution results. Of course there has to be hierarchy which separate folders from different workers on the same machines. That is why paths to directories are in format: -${DEFAULT}/${FOLDER}/${WORKER_ID}/${JOB_ID} where default means default working -directory of whole worker, folder is particular directory for some purpose -(archives, evaluation...). Mentioned division of job directories proved to be -flexible and detailed enough, everything is in logical units and where it is -supposed to be which means that searching through this system should be easy. In -addition if solutions of users have access only to evaluation directory then -they do not have access to unnecessary files which is better for overall -security of whole ReCodEx. +`${DEFAULT}/${FOLDER}/${WORKER_ID}/${JOB_ID}` where default means default +working directory of whole worker, folder is particular directory for some +purpose (archives, evaluation, ...). Mentioned division of job directories +proved to be flexible and detailed enough, everything is in logical units and +where it is supposed to be which means that searching through this system should +be easy. In addition if solutions of users have access only to evaluation +directory then they do not have access to unnecessary files which is better for +overall security of whole ReCodEx. As we discovered above worker has job directories but users who are writing and managing job configurations do not know where they are (on some particular @@ -1130,7 +1130,7 @@ represent particular folders. Marks or signs can have form of some kind of special strings which can be called variables. These variables then can be used everywhere where filesystem paths are used within configuration file. This will solve problem with specific worker environment and specific hierarchy of -directories. Final form of variables is \${...} where triple dot is textual +directories. Final form of variables is `${...}` where triple dot is textual description. This format was used because of special dollar sign character which cannot be used within filesystem path, braces are there only to border textual description of variable. From ce7c198529c01382cbb4310001c04e456076b8af Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Tue, 10 Jan 2017 14:36:35 +0100 Subject: [PATCH 25/46] Better pdf fit --- Rewritten-docs.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index d2c2e2d..f19a497 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -846,16 +846,6 @@ server side based on the inputs from the user. This decoupling gives us the ability to create multiple client side tools which may address different needs of the users. -One possible configuration of ReCodEx system is ilustrated on following picture, -where thete is one shared backend with three workers and two separate instances -of whole frontend. This configuration may be suitable for MFF UK -- basic -programming course and KSP competition. But maybe even sharing web API and -fileserver with only custom instances of client (web app or own implementation) -is more likely to be used. Note, that connections between components are not -fully accurate. - -![Overall architecture](https://github.com/ReCodEx/wiki/blob/master/images/Overall_Architecture.png) - The frontend developed as part of this project is a web application created with the needs of the Faculty of Mathematics and Physics of the Charles university in Prague in mind. The users are the students and their teachers, groups correspond @@ -866,6 +856,16 @@ also possible to develop their own frontend with their own user management system for their specific needs and use the possibilities of the backend without any changes, as was mentioned in the previous paragraphs. +One possible configuration of ReCodEx system is ilustrated on following picture, +where thete is one shared backend with three workers and two separate instances +of whole frontend. This configuration may be suitable for MFF UK -- basic +programming course and KSP competition. But maybe even sharing web API and +fileserver with only custom instances of client (web app or own implementation) +is more likely to be used. Note, that connections between components are not +fully accurate. + +![Overall architecture](https://github.com/ReCodEx/wiki/blob/master/images/Overall_Architecture.png) + In the latter parts of the documentation, both of the backend and frontend parts will be introduced separately and covered in more detail. The communication protocol between these two logical parts will be described as well. From 5d9c6f13294ede40690b522a5b04bf6ee0736e3e Mon Sep 17 00:00:00 2001 From: Teyras Date: Tue, 10 Jan 2017 15:49:57 +0100 Subject: [PATCH 26/46] rest api analysis --- Rewritten-docs.md | 71 ++++++++++++++++++++++++++++++----------------- 1 file changed, 46 insertions(+), 25 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index f19a497..3849903 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -457,18 +457,10 @@ designing user interfaces on top of them. The application interacts with users. From the project assignment is clear, that the system has to keep personalized data about users and adapt presented content according to this knowledge. User data cannot be publicly visible, so that -implies necessity of user authentication. There are several way of -authentication user in web applications. HTTP basic authentication has a few -drawbacks like sending plain credentials in every request or no logout option, -so it is not recommended to use. Using cookies is possible, but it is -susceptible to various types of attacks including stealing and also brings state -into stateless protocols as REST. Another option is using tokens, OAuth2 or JWT -(JSON Web Token). Nowadays it is widely used and has only a few cons (need to -make extra effort to mitigate XSS attacks). Additional option is usage of -one-time passwords. These can be time or counter based and are mostly used for -two-factor authentication. Since ReCodEx does not need to have military grade -security, JWT tokens are used for authentication. However, two-factor -authentication may come in next releases. +implies necessity of user authentication. The application also has to support +multiple ways of authentication (university authentication systems, a company +LDAP server, an OAuth server...) and permit adding more security measures in the +future, such as two-factor authentication. User data also includes a privilege level. From the assignment it is required to have at least two roles, _student_ and _supervisor_. However, it is wise to add @@ -1402,6 +1394,9 @@ We considered several technologies which could be used: servers. It is a suitable technology for this kind of a project. It has all the features we need when some additional extensions are installed (to support LDAP or ZeroMQ). +- Ruby on Rails, Python (Django), etc. -- popular web technologies that appeared + in the last decade. Both support ZeroMQ and LDAP via extensions and have large + developer communities. - ASP.NET (C#), JSP (Java) -- these technologies are very robust and are used to create server technologies in many big enterpises. Both can run on Windows and Linux servers (ASP.NET using the .NET Core). @@ -1418,21 +1413,45 @@ technologies superior to PHP in all other aspects - PHP 7 is a mature language with a huge community and a wide range of tools, libraries, and frameworks. We decided to use an ORM framework to manage the database, namely the widely -used PHP ORM Doctrine 2. This framework has a robust abstraction layer DBAL so -the database engine is not very important and it can be changed without any need -for changing the code. We chose an open-source database MariaDB. +used PHP ORM Doctrine 2. Using an ORM tool means we do not have to write SQL +queries by hand. Instead, we work with persistent objects, which provides a +higher level of abstraction. Doctrine also has a robust database abstraction +layer so the database engine is not very important and it can be changed without +any need for changing the code. MariaDB was chosen as the storage backend. To speed up the development process of the PHP server application we decided to -use an MVC framework. After evaluating and trying several frameworks, such as -Lumen, Laravel, and Symfony, we ended up using the framework Nette. This -framework is very common in the Czech Republic -- its main developer is a -well-known Czech programmer David Grudl -- and we were already familiar with the -patterns used in this framework (e.g., dependency injection, authentication, -routing). There is a good extension for the Nette framework which makes usage of -Doctrine 2 very straightforward. - -@todo: authentication, some possibilities and describe used jwt (mentioned in -basic concepts, maybe elaborate more or just remove this item from todo list) +use a web framework. After evaluating and trying several frameworks, such as +Lumen, Laravel, and Symfony, we ended up using Nette. This framework is very +common in Czech Republic -- its lead developer is a well-known Czech programmer +David Grudl -- and we were already familiar with the patterns used in this +framework, such as dependency injection, authentication, routing. These concepts +are useful even when developing a REST application, which might be a surprise +considering that Nette focuses on "traditional" web applications. There is also +a Nette extension which makes integartion of Doctrine 2 very straightforward. + +#### Request handling + +@todo Nette Router, how we exploit presenters to act as api endpoints + +#### Authentication + +Because Nette is focused on building web applications that render a new +page for (almost) every request, it uses PHP sessions (based on cookies) for +authentication. This method is unsuitable for REST APIs where clients do not +typically store cookies. However, it is common that RESTful services provide +access tokens that are then sent with every request by the client. + +JWT (JSON web tokens), an open standard for access tokens, was chosen for our +authentication implementation. Support libraries exist for all major languages +used in web developments that facilitate straightforward usage. The tokens use +asymmetric cryptography for signing, which provides a satisfactory level of +security. + +To implement JWT in Nette, we have to implement some of its security-related +interfaces such as IAuthenticator and IUserStorage, which is rather easy +thanks to the simple authentication flow. Replacing these services in a Nette +application is also straightforward, thanks to its dependency injection +container implementation. @todo: solution of forgotten password, why this in particular @@ -1446,6 +1465,8 @@ basic concepts, maybe elaborate more or just remove this item from todo list) within instance and how it is implemented and how it could be implemented (describe only implementation if it is something what to say) +@todo permission handling, roles, etc. + @todo: where is stored which workers can be used by supervisors and which runtimes are available, describe possibilities and why is not implemented automatic solution @todo: on demand loading of students submission, in-time loading of every other submission, why From e378fc5aebb2792e44cb0c94bfe629cdeeeacf8f Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Tue, 10 Jan 2017 16:37:28 +0100 Subject: [PATCH 27/46] Improvements --- Rewritten-docs.md | 96 ++++++++++++++++++++++++----------------------- 1 file changed, 49 insertions(+), 47 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index f19a497..cad10b4 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -665,33 +665,34 @@ state. There are lot of things which deserves discussion concerning results of evaluation, how they should be displayed, what should be visible or not and also -some what kind of reward for users solutions should be chosen. +what kind of reward for users solutions should be chosen. At first let us focus on all kinds of outputs from executed programs within job. Out of discussion is that supervisors should be able to view almost all outputs from solutions if they choose them to be visible and recorded. This feature is critical in debugging either whole exercises or users solutions. But should it be default behaviour to record every output? Absolutely not, supervisor should -have choice to turn it on but defaults has to be not record it. Even without -this functionality can be file base around whole ReCodEx system quite large and -on top of that outputs from executed programs can be sometimes very extensive. -Storing this amount of data is purely nonsense to every solution. But if -requested by supervisor then this feature should be available. - -Question what should regular users see from execution of their solution is more -interesting. Simple answer is of course that they should not see anything which -is partly true. Outputs from their programs can be anything and users can -somehow analyse inputs or even redirect them to output. So outputs from +have a choice to turn it on, but discarding the outputs has to be the default +option. Even without this functionality a file base around whole ReCodEx system +can become quite large and on top of that outputs from executed programs can be +sometimes very extensive. Storing this amount of data is inefficient and +unnecessary to most of the solutions. However, on supervisor request this +feature should be available. + +More interesting question is what should regular users see from execution of +their solution. Simple answer is of course that they should not see anything +which is partly true. Outputs from their programs can be anything and users can +somehow analyze inputs or even redirect them to output. So outputs from execution should not be visible at all or under very special circumstances. But -that is not so straighforward for compilation or other kinds of initiation. -Well, this is another story, it really depends on the particular case. But -generally it is quite harmless to display user some kind of compilation error -which can help a lot during troubleshooting. Of course again this kind of -functionality should be configurable by supervisors and disabled by default. -There is also the last kind of tasks which can output some information which is -evaluation tasks. Output of these tasks is somehow important to whole system and -again can contain some information about inputs or reference outputs. So outputs -of evaluation tasks should not be visible to regular users too. +that is not so straightforward for compilation or other kinds of initiation, +where it really depends on the particular case. Generally it is quite harmless +to display user some kind of compilation error which can help a lot during +troubleshooting. Of course again this kind of functionality should be +configurable by supervisors and disabled by default. There is also the last kind +of tasks which can output some information which is evaluation tasks. Output of +these tasks is somehow important to whole system and again can contain some +information about inputs or reference outputs. So outputs of evaluation tasks +should not be visible to regular users too. The overall concept of grading solutions was presented earlier. To briefly remind that, backend returns only exact measured values (used time and memory, @@ -702,19 +703,20 @@ implemented and any sort of magic can return the final value. We found out several computational possibilities. There is basic arithmetic, weighted arithmetic, geometric and harmonic mean of results of each test (the -result is boolean succeeded/failed, optionaly has weight), some kind of +result is logical value succeeded/failed, optionally with weight), some kind of interpolation of used amount of time for each test, the same with used memory amount and surely many others. To keep the project simple, we decided to design -apropriate interface and implement only weighted arithmetic mean computation, -which is used in about 90% of all assignments. Of course, diferent scheme can be -chosen for every assignment and also configured -- for example test weights can -be specified for implemented weighted arithmetic mean. Advanced ways of -computation can be implemented on demand when is a real demand for them. +appropriate interface and implement only weighted arithmetic mean computation, +which is used in about 90% of all assignments. Of course, different scheme can +be chosen for every assignment and also can be configured -- for example +specifying test weights for implemented weighted arithmetic mean. Advanced ways +of computation can be implemented on demand when there is a real demand for +them. To avoid assigning points for insufficient solutions (like only printing "File error" which is the valid answer in two tests), a minimal point threshold can be -specified. It he solution is to get less points than specified, it will get zero -points instead. This functionality can be empedded into grading computation +specified. It the solution is to get less points than specified, it will get +zero points instead. This functionality can be embedded into grading computation algoritm itself, but it would have to be present in each implementation separately, which is a bit ugly. So, this feature is separated from point computation. @@ -722,14 +724,14 @@ computation. Automatic grading cannot reflect all aspects of submitted code. For example, structuring the code, number and quality of comments and so on. To allow supervisors bring these manually checked things into grading, there is a concept -of bonus points. They can be positive or negative. Generaly the solution with -the most assigned points is marked for grading that particular solution. +of bonus points. They can be positive or negative. Generally the solution with +the most assigned points is marked for grading that particular assignment. However, if supervisor is not satisfied with student solution (really bad code, -cheating, ...) he/she assigns the student negative bonus points. But to prevent -chosing another solution with more points by the system or even submitting the -same code again which is worth more points by students, supervisor can mark a -particular solution as marked and used for grading instead of solution with the -most points. +cheating, ...) he/she assigns the student negative bonus points. To prevent +overriding this decision by system choosing another solution with more points or +even student submitting the same code again which evaluates to more points, +supervisor can mark a particular solution as marked and used for grading instead +of solution with the most points. ### Persistence @@ -752,19 +754,19 @@ databases or formatted plain files (CSV for example) fits best for them. However, the data often have to support find operation, so they have to be sorted and allow random access for resolving cross references. Also, addition a deletion of entries should take reasonable time (at most logaritmic time -complexity to number of saved values). This practicaly excludes plain files, so +complexity to number of saved values). This practically excludes plain files, so relational database is used instead. On the other hand, there are some data with no such great structure and much larger size. These can be evaluation logs, sample input files for exercises or -submited sources by students. Saving this kind of data into relational database +submitted sources by students. Saving this kind of data into relational database is not suitable, but it is better to keep them as ordinary files or store them into some kind of NoSQL database. Since they are already files and does not need to be backed up in multiple copies, it is easier to keep them as ordinary files in filesystem. Also, this solution is more lightweight and does not require additional dependencies on third-party software. File can be identified using its filesystem path or unique index stored as value in relational database. Both -approaches are equaly good, final decission depends on actual case. +approaches are equally good, final decision depends on actual case. ## Structure of the project @@ -856,10 +858,10 @@ also possible to develop their own frontend with their own user management system for their specific needs and use the possibilities of the backend without any changes, as was mentioned in the previous paragraphs. -One possible configuration of ReCodEx system is ilustrated on following picture, -where thete is one shared backend with three workers and two separate instances -of whole frontend. This configuration may be suitable for MFF UK -- basic -programming course and KSP competition. But maybe even sharing web API and +One possible configuration of ReCodEx system is illustrated on following +picture, where there is one shared backend with three workers and two separate +instances of whole frontend. This configuration may be suitable for MFF UK -- +basic programming course and KSP competition. But maybe even sharing web API and fileserver with only custom instances of client (web app or own implementation) is more likely to be used. Note, that connections between components are not fully accurate. @@ -1068,7 +1070,7 @@ as network communication which is quite handy and solves problems with threads synchronization and such. At this point we have worker with two internal parts listening one and execution -one. Implementation of first one is quite straighforward and clear. So lets +one. Implementation of first one is quite straightforward and clear. So lets discuss what should be happening in execution subsystem. Jobs as work units can quite vary and do completely different things, that means configuration and worker has to be prepared for this kind of generality. Configuration and its @@ -1342,7 +1344,7 @@ development. Users want to view real time evaluation progress of their solution. It can be easily done with established double-sided connection stream, but it is hard to achieve with web technologies. HTTP protocol works differently on separate -requests basis with no longterm connection. However, there is widely used +requests basis with no long term connection. However, there is widely used technology to solve this problem, WebSocket protocol. Working with WebSocket protocol from the backend is possible, but not ideal from @@ -1376,7 +1378,7 @@ following picture. ![Message flow inside montior](https://raw.githubusercontent.com/ReCodEx/wiki/master/images/Monitor_arch.png) The message channel inputing the monitor uses ZeroMQ as main message framework -used by backend. This decision keeps rest of backend avare of used +used by backend. This decision keeps rest of backend aware of used communication protocol and related libraries. Output channel is WebSocket as a protocol for sending messages to web browsers. In Python, there are several WebSocket libraries. The most popular one is `websockets` in cooperation with @@ -1403,8 +1405,8 @@ We considered several technologies which could be used: the features we need when some additional extensions are installed (to support LDAP or ZeroMQ). - ASP.NET (C#), JSP (Java) -- these technologies are very robust and are used to - create server technologies in many big enterpises. Both can run on Windows and - Linux servers (ASP.NET using the .NET Core). + create server technologies in many big enterprises. Both can run on Windows + and Linux servers (ASP.NET using the .NET Core). - JavaScript (Node.js) -- it is a quite new technology and it is being used to create REST APIs lately. Applications running on Node.js are quite performant and the number of open-source libraries available on the Internet is very From 2df1d04278038a61ff58ac892bbee560d02584aa Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Tue, 10 Jan 2017 17:33:18 +0100 Subject: [PATCH 28/46] forgotten password --- Rewritten-docs.md | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 81465da..8adbd9c 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -529,6 +529,36 @@ for every assignment of the same exercise. This separation is natural for all users, in CodEx it is implemented in similar way and no other considerable solution was found. +### Forgotten password + +With authentication and some sort of dealing with passwords is related problem +with forgotten passwords. People easily forget them and there has to be some +kind of mechanism to retrieve new password or change old one. Problem is it +cannot be done in totally secure way but we can at least come quite close to it. +But lets start from the beginning there are ways to handle that which are +absolutely not secure and recommendable, for example sending old password +through email. Better solution but still not secure is to generate new one and +again send it through email. This solution was provided in CodEx, users had to +write email to administrator who generated new password and sent it back to +sender. This simple solution could be also automated but administrator had quite +a big control over whole process which might come in handy there could be some +additional checkups for example, but on the other hand it can be quite time +consuming. + +Probably the best solution which is quite used and fairly secure is following. +Lets consider only case in which all users have to fill email addresses into +system and these addresses are safely in the hands of the right users. At the +beginning user finds out that he/she does not remember password after that user +requests password reset and fill in his/her unique identifier, it might be email +or unique nickname. Based on matched user account system generates unique access +token and sends it user email address. Tokens should be time limited and usable +only once so they cannot be misused. User then takes token or address which was +provided in email and go to system in appropriate section, where new password +can be set. After that user can sign in with his new password. As stated this +solution is quite safe and user can handle it on its own administrator does not +have to worry about it. That is why this approach was chosen to be used in +ReCodEx. + ### Evaluation unit executed by ReCodEx One of the bigger requests for the new system is to support a complex @@ -1455,8 +1485,6 @@ thanks to the simple authentication flow. Replacing these services in a Nette application is also straightforward, thanks to its dependency injection container implementation. -@todo: solution of forgotten password, why this in particular - @todo: rest api is used for report of backend state and errors, describe why and other possibilities (separate component) @todo: mail reports - to users, admins From 8bb0317a6afab4ac5b2e06ff63902bc8652cba3e Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Tue, 10 Jan 2017 17:59:34 +0100 Subject: [PATCH 29/46] Update --- Rewritten-docs.md | 52 +++++++++++++++++++++++------------------------ 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 8adbd9c..91afb33 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -531,33 +531,33 @@ solution was found. ### Forgotten password -With authentication and some sort of dealing with passwords is related problem -with forgotten passwords. People easily forget them and there has to be some -kind of mechanism to retrieve new password or change old one. Problem is it -cannot be done in totally secure way but we can at least come quite close to it. -But lets start from the beginning there are ways to handle that which are -absolutely not secure and recommendable, for example sending old password -through email. Better solution but still not secure is to generate new one and +With authentication and some sort of dealing with passwords is related a problem +with forgotten credentials, especilly passwords. People easily forget them and +there has to be some kind of mechanism to retrieve a new password or change the +old one. Problem is that it cannot be done in totally secure way, but we can at +least come quite close to it. First, there are absolutely not secure and +recommendable ways to handle that, for example sending the old password through +email. A better, but still not secure solution is to generate a new one and again send it through email. This solution was provided in CodEx, users had to -write email to administrator who generated new password and sent it back to -sender. This simple solution could be also automated but administrator had quite -a big control over whole process which might come in handy there could be some -additional checkups for example, but on the other hand it can be quite time -consuming. - -Probably the best solution which is quite used and fairly secure is following. -Lets consider only case in which all users have to fill email addresses into -system and these addresses are safely in the hands of the right users. At the -beginning user finds out that he/she does not remember password after that user -requests password reset and fill in his/her unique identifier, it might be email -or unique nickname. Based on matched user account system generates unique access -token and sends it user email address. Tokens should be time limited and usable -only once so they cannot be misused. User then takes token or address which was -provided in email and go to system in appropriate section, where new password -can be set. After that user can sign in with his new password. As stated this -solution is quite safe and user can handle it on its own administrator does not -have to worry about it. That is why this approach was chosen to be used in -ReCodEx. +write an email to administrator, who generated a new password and sent it back +to the sender. This simple solution could be also automated, but administrator +had quite a big control over whole process. This might come in handy if there +could be some additional checkups for example, but on the other hand it can be +quite time consuming. + +Probably the best solution which is often used and is fairly secure is +following. Let us consider only case in which all users have to fill their +email addresses into the system and these addresses are safely in the hands of +the right users. When user finds out that he/she does not remember a password, +he/she requests a password reset and fill in his/her unique identifier; it might +be email or unique nickname. Based on matched user account the system generates +unique access token and sends it user via email address. This token should be +time limited and usable only once, so it cannot be misused. User then takes the +token or URL address which is provided in the email and go to the system's +appropriate section, where new password can be set. After that user can sign in +with his/her new password. As previously stated, this solution is quite safe and +user can handle it on its own, so administrator does not have to worry about it. +That is the main reason why this approach was chosen to be used in ReCodEx. ### Evaluation unit executed by ReCodEx From 31e660e3cc43c5680969b7c5c90b57eba9e367b3 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Tue, 10 Jan 2017 19:15:05 +0100 Subject: [PATCH 30/46] WIP: backend failure reporting --- Rewritten-docs.md | 39 ++++++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 91afb33..593f12d 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -532,32 +532,32 @@ solution was found. ### Forgotten password With authentication and some sort of dealing with passwords is related a problem -with forgotten credentials, especilly passwords. People easily forget them and +with forgotten credentials, especially passwords. People easily forget them and there has to be some kind of mechanism to retrieve a new password or change the old one. Problem is that it cannot be done in totally secure way, but we can at -least come quite close to it. First, there are absolutely not secure and -recommendable ways to handle that, for example sending the old password through -email. A better, but still not secure solution is to generate a new one and -again send it through email. This solution was provided in CodEx, users had to -write an email to administrator, who generated a new password and sent it back -to the sender. This simple solution could be also automated, but administrator -had quite a big control over whole process. This might come in handy if there -could be some additional checkups for example, but on the other hand it can be -quite time consuming. +least come quite close to it. First, there are absolutely not secure and +recommendable ways how to handle that, for example sending the old password +through email. A better, but still not secure solution is to generate a new one +and again send it through email. This solution was provided in CodEx, users had +to write an email to administrator, who generated a new password and sent it +back to the sender. This simple solution could be also automated, but +administrator had quite a big control over whole process. This might come in +handy if there could be some additional checkups for example, but on the other +hand it can be quite time consuming. Probably the best solution which is often used and is fairly secure is -following. Let us consider only case in which all users have to fill their +following. Let us consider only case in which all users have to fill their email addresses into the system and these addresses are safely in the hands of -the right users. When user finds out that he/she does not remember a password, +the right users. When user finds out that he/she does not remember a password, he/she requests a password reset and fill in his/her unique identifier; it might be email or unique nickname. Based on matched user account the system generates -unique access token and sends it user via email address. This token should be +unique access token and sends it to user via email address. This token should be time limited and usable only once, so it cannot be misused. User then takes the token or URL address which is provided in the email and go to the system's appropriate section, where new password can be set. After that user can sign in with his/her new password. As previously stated, this solution is quite safe and user can handle it on its own, so administrator does not have to worry about it. -That is the main reason why this approach was chosen to be used in ReCodEx. +That is the main reason why this approach was chosen to be used. ### Evaluation unit executed by ReCodEx @@ -1485,8 +1485,6 @@ thanks to the simple authentication flow. Replacing these services in a Nette application is also straightforward, thanks to its dependency injection container implementation. -@todo: rest api is used for report of backend state and errors, describe why and other possibilities (separate component) - @todo: mail reports - to users, admins @todo: what files are stored in api, why there are duplicates among api and fileserver @@ -1497,9 +1495,16 @@ within instance and how it is implemented and how it could be implemented @todo permission handling, roles, etc. +@todo: on demand loading of students submission, in-time loading of every other submission, why + +#### Backend management + +Considering the fact that we have backend as a separate component which has no clue about administrators and uses only logging as some kind of failure reporting. It can be handy to provide this functionality to backend from frontend which manages users. The simplest solution would be again to have separate component with some sort of public interface. It can be for example REST or some other communication which backend can handle. Functionality of this kind of component is then quite easy. When request for report arrives from backend then type is inferred and if it is error which deserves attention of administrator then email is sent to him/her. There can also be errors which are not that important, was somehow solved by backend itself or are only informative, these do not have to be reported by email but only stored in persistent database for further consideration. On top of that separate component can be internal and not exposed to outside network. Disadvantage is that database layer which is used in some particular API instance cannot be used here because multiple instances of API can use one backend. + +@todo: rest api is used for report of backend state and errors, describe why + @todo: where is stored which workers can be used by supervisors and which runtimes are available, describe possibilities and why is not implemented automatic solution -@todo: on demand loading of students submission, in-time loading of every other submission, why ### Web-app From cd071f19743d4b7f1e017039bc512c4a037c6be0 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Tue, 10 Jan 2017 19:43:31 +0100 Subject: [PATCH 31/46] backend error reporting --- Rewritten-docs.md | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 593f12d..44c347f 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1499,9 +1499,37 @@ within instance and how it is implemented and how it could be implemented #### Backend management -Considering the fact that we have backend as a separate component which has no clue about administrators and uses only logging as some kind of failure reporting. It can be handy to provide this functionality to backend from frontend which manages users. The simplest solution would be again to have separate component with some sort of public interface. It can be for example REST or some other communication which backend can handle. Functionality of this kind of component is then quite easy. When request for report arrives from backend then type is inferred and if it is error which deserves attention of administrator then email is sent to him/her. There can also be errors which are not that important, was somehow solved by backend itself or are only informative, these do not have to be reported by email but only stored in persistent database for further consideration. On top of that separate component can be internal and not exposed to outside network. Disadvantage is that database layer which is used in some particular API instance cannot be used here because multiple instances of API can use one backend. - -@todo: rest api is used for report of backend state and errors, describe why +Considering the fact that we have backend as a separate component which has no +clue about administrators and uses only logging as some kind of failure +reporting. It can be handy to provide this functionality to backend from +frontend which manages users. The simplest solution would be again to have +separate component with some sort of public interface. It can be for example +REST or some other communication which backend can handle. Functionality of this +kind of component is then quite easy. When request for report arrives from +backend then type is inferred and if it is error which deserves attention of +administrator then email is sent to him/her. There can also be errors which are +not that important, was somehow solved by backend itself or are only +informative, these do not have to be reported by email but only stored in +persistent database for further consideration. On top of that separate component +can be internal and not exposed to outside network. Disadvantage is that +database layer which is used in some particular API instance cannot be used here +because multiple instances of API can use one backend. + +Another solution which was at the end implemented is to integrate backend +failure reporting feature to API. Problem with previous one is that if job +execution fails backend has to report this error to some particular API server +from which request for evaluation came. This information is essential and has to +be stored there and not in some general component and general error database. +Obviously if there are multiple API servers connected to one backend there has +to be some API server configured in backend as the main one which receives +reports about general backend errors which are not connected to jobs. This +solution was chosen because as stated we have to implement job error reporting +in API and having separate component only for general errors is not feasible. In +the end error reporting should be available under different route which is +secured by basic HTTP authentication, because basic authentication is easy +enough to implement in low-level backend components. That also means this +feature is visible and can be exploited but from our points of view it seems as +appropriate compromise in simplicity. @todo: where is stored which workers can be used by supervisors and which runtimes are available, describe possibilities and why is not implemented automatic solution From c9839e957af3f15b942f7ab13871e0060f12986e Mon Sep 17 00:00:00 2001 From: Teyras Date: Tue, 10 Jan 2017 21:58:12 +0100 Subject: [PATCH 32/46] a bit about file uploads --- Rewritten-docs.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 44c347f..510cad5 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1489,6 +1489,15 @@ container implementation. @todo: what files are stored in api, why there are duplicates among api and fileserver +#### Uploading files + +There are two cases when users need to upload files using the API – submitting +solutions to an assignment and creating a new exercise. In both of these cases, +the final destination of the files is the fileserver. + +However, the fileserver is not publicly accessible, so the files have to be +uploaded through the API. + @todo: groups and hierarchy, describe arbitrary nesting which should be possible within instance and how it is implemented and how it could be implemented (describe only implementation if it is something what to say) From 9d156e9a827dff4e55d03375808f781d01793e02 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Tue, 10 Jan 2017 22:50:14 +0100 Subject: [PATCH 33/46] Backend state in API --- Rewritten-docs.md | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 510cad5..509942f 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1540,8 +1540,25 @@ enough to implement in low-level backend components. That also means this feature is visible and can be exploited but from our points of view it seems as appropriate compromise in simplicity. -@todo: where is stored which workers can be used by supervisors and which runtimes are available, describe possibilities and why is not implemented automatic solution - +Next thing relating backend management is storing its current state. This namely +concerns which workers are available for processing with what hardware and which +languages can be used in exercises. Another step is overall backend state like +how many jobs were processed on some particular worker, workload of broker and +workers, etc. The easiest solution is to manage these information by hand, every +instance of API has to have administrator which would have to fill them. This of +course includes only currently available workers and runtime environments, +backend statistics cannot be provided this way. + +Better solution is to let these information update automatically. This can be +done two ways either it can be provided by backend on-demand if API needs them +or backend will send these information periodically to API. Things like +currently available workers or environments are better to be really up-to-date +so this can provided on-demand if needed. Backend statistics are not necessary +stuff which can be updated periodically. But it really depends on the period of +updates, if it is short enough then even available workers, etc. could be +updated this way and be quite up-to-date. However due to lack of time automatic +refreshing of backend state will not be implemented in early versions but might +be implemented in next releases. ### Web-app From 90eff61aad9b12ab2f864adf295bee0d439ab5e1 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Tue, 10 Jan 2017 23:01:59 +0100 Subject: [PATCH 34/46] Solution loading --- Rewritten-docs.md | 36 +++++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 44c347f..28ed9bd 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1485,18 +1485,40 @@ thanks to the simple authentication flow. Replacing these services in a Nette application is also straightforward, thanks to its dependency injection container implementation. -@todo: mail reports - to users, admins +#### Solution loading + +When a solution evaluation on backend is finished, results are saved to +fileserver and API is notified by broker about this state. Then some further +steps needs to be done before the results can be presented to users. For +example, these steps are parsing of the results, computating score or saving +structured data into database. There are two main possibilities of loading: + +- immediately +- on demand + +They are almost equal, none of them provides any kind of big advantage. Loading +solutions immediately is better, because fetching results from client for the +first time can be a bit faster. On the other hand, loading them on demand when +they are requested for the first time can save some resources when the solution +results are not important and no one is interested in them. + +From this choice, we picked up lazy loading when the results are requested. +However, concept of asynchronous jobs is then introduced. This type of job is +useful for batch submitting of jobs, for example rerunning jobs which failed on +worker hardware issue. These jobs are typically submitted by different user than +author (administator for example), so original authors should be notified. In +this case it is more reasonable to load the results immediately and optionally +send them with the notification. Exactly this is also done, special asynchronous +jobs are loaded immediately with email notification to the original job author. + +From a short time distance it seems that immediate loading of all jobs could +simplify loading code and has no major drawbacks. In the next version of ReCodEx +we will rethink this decission. @todo: what files are stored in api, why there are duplicates among api and fileserver -@todo: groups and hierarchy, describe arbitrary nesting which should be possible -within instance and how it is implemented and how it could be implemented -(describe only implementation if it is something what to say) - @todo permission handling, roles, etc. -@todo: on demand loading of students submission, in-time loading of every other submission, why - #### Backend management Considering the fact that we have backend as a separate component which has no From 55f08f9fb3a4927f654c54fcda6afa4992e51be1 Mon Sep 17 00:00:00 2001 From: Teyras Date: Tue, 10 Jan 2017 22:22:25 +0100 Subject: [PATCH 35/46] more about file uploads --- Rewritten-docs.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 14ee214..e46ec85 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1489,12 +1489,17 @@ container implementation. #### Uploading files -There are two cases when users need to upload files using the API – submitting +There are two cases when users need to upload files using the API -– submitting solutions to an assignment and creating a new exercise. In both of these cases, -the final destination of the files is the fileserver. - -However, the fileserver is not publicly accessible, so the files have to be -uploaded through the API. +the final destination of the files is the fileserver. However, the fileserver is +not publicly accessible, so the files have to be uploaded through the API. + +The files can be either forwarded to the fileserver directly, without any +interference from the API server, or stored and forwarded later. We chose the +second approach, which is harder to implement, but more convenient -- it lets +exercise authors double-check what they upload to the fileserver and solutions +to assignments can be uploaded in a single request, which makes it easy for the +fileserver to create an archive of the solution files. @todo permission handling, roles, etc. From f4119b7b7659219614a49da99386ffceaace8d05 Mon Sep 17 00:00:00 2001 From: Teyras Date: Tue, 10 Jan 2017 23:37:23 +0100 Subject: [PATCH 36/46] request handling analysis --- Rewritten-docs.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index e46ec85..f940469 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1463,7 +1463,23 @@ a Nette extension which makes integartion of Doctrine 2 very straightforward. #### Request handling -@todo Nette Router, how we exploit presenters to act as api endpoints +A typical scenario for handling an API request is matching the HTTP request with +a corresponding handler routine which creates a response object, that is then +sent back to the client, encoded with JSON. The Nette\Application package can be +used to achieve this with Nette, although it is meant to be used mainly in MVP +applications. + +Matching HTTP requests with handlers can be done using standard Nette URL +routing -- we will create a Nette route for each API endpoint. Using the routing +mechanism from Nette logically leads to implementing handler routines as Nette +Presenter actions. Each presenter should serve logically related endpoints. + +The last step is encoding the response as JSON. In Nette\Application, HTTP +responses are returned using the `Presenter::sendResponse()` method. We decided +to write a method that calls `sendResponse` internally and takes care of the +encoding. This method has to be called in every presenter action. An alternative +approach would be using the internal payload object of the presenter, which is +more convenient, but provides us with less control. #### Authentication @@ -1485,11 +1501,9 @@ thanks to the simple authentication flow. Replacing these services in a Nette application is also straightforward, thanks to its dependency injection container implementation. -@todo: what files are stored in api, why there are duplicates among api and fileserver - #### Uploading files -There are two cases when users need to upload files using the API -– submitting +There are two cases when users need to upload files using the API -- submitting solutions to an assignment and creating a new exercise. In both of these cases, the final destination of the files is the fileserver. However, the fileserver is not publicly accessible, so the files have to be uploaded through the API. From 7b643215eb710bde12b6b4495b2c291b6d8e2a27 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Wed, 11 Jan 2017 10:44:56 +0100 Subject: [PATCH 37/46] Typos --- Rewritten-docs.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index f940469..f7a9985 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -775,7 +775,7 @@ and email are short strings, age is an integer. Considering this, relational databases or formatted plain files (CSV for example) fits best for them. However, the data often have to support find operation, so they have to be sorted and allow random access for resolving cross references. Also, addition a -deletion of entries should take reasonable time (at most logaritmic time +deletion of entries should take reasonable time (at most logarithmic time complexity to number of saved values). This practically excludes plain files, so relational database is used instead. @@ -1459,7 +1459,7 @@ David Grudl -- and we were already familiar with the patterns used in this framework, such as dependency injection, authentication, routing. These concepts are useful even when developing a REST application, which might be a surprise considering that Nette focuses on "traditional" web applications. There is also -a Nette extension which makes integartion of Doctrine 2 very straightforward. +a Nette extension which makes integration of Doctrine 2 very straightforward. #### Request handling @@ -1522,7 +1522,7 @@ fileserver to create an archive of the solution files. When a solution evaluation on backend is finished, results are saved to fileserver and API is notified by broker about this state. Then some further steps needs to be done before the results can be presented to users. For -example, these steps are parsing of the results, computating score or saving +example, these steps are parsing of the results, computation of score or saving structured data into database. There are two main possibilities of loading: - immediately @@ -1538,14 +1538,14 @@ From this choice, we picked up lazy loading when the results are requested. However, concept of asynchronous jobs is then introduced. This type of job is useful for batch submitting of jobs, for example rerunning jobs which failed on worker hardware issue. These jobs are typically submitted by different user than -author (administator for example), so original authors should be notified. In +author (administrator for example), so original authors should be notified. In this case it is more reasonable to load the results immediately and optionally send them with the notification. Exactly this is also done, special asynchronous jobs are loaded immediately with email notification to the original job author. From a short time distance it seems that immediate loading of all jobs could simplify loading code and has no major drawbacks. In the next version of ReCodEx -we will rethink this decission. +we will rethink this decision. #### Backend management From 363077f1b4da73da0d4a2eec8e315786be88d7fa Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Wed, 11 Jan 2017 11:19:00 +0100 Subject: [PATCH 38/46] Permissions --- Rewritten-docs.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index f7a9985..0a5ed15 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1515,7 +1515,28 @@ exercise authors double-check what they upload to the fileserver and solutions to assignments can be uploaded in a single request, which makes it easy for the fileserver to create an archive of the solution files. -@todo permission handling, roles, etc. +#### Permissions + +In a system storing user data has to be implemented some kind of permission +checking. Previous chapters implies, that each user has to have a role, which +corresponds to his/her privileges. Our research showed, that three roles are +sufficient -- student, supervisor and administrator. The user role has to be +checked with every request. The good points is, that roles nicely match with +granuality of API endpoints, so the permission checking can be done at the +beginning of each request. That is implemented using PHP annotations, which +allows to specify allowed user roles for each request with very little of code, +but all the business logic is the same, together in one place. + +However, roles cannot cover all cases. For example, if user is a supervisor, it +relates only to groups, where he/she is a supervisor. But using only roles +allows him/her to act as supervisor in all groups in the system. Unfortunately, +this cannot be easily fixed using some annotations, because there are many +different cases when this problem occurs. To fix that, some additional checks +can be performed at the beginning of request processing. Usually it is only one +or two simple conditions. + +With this two concepts together it is possible to easily cover all cases of +permission checking with quite a small amount of code. #### Solution loading From b4b81f52918dd2e91e2d8e86c19e1bfd836767b7 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Wed, 11 Jan 2017 11:22:07 +0100 Subject: [PATCH 39/46] Make PDF compile --- Rewritten-docs.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 0a5ed15..65b1040 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -1465,16 +1465,16 @@ a Nette extension which makes integration of Doctrine 2 very straightforward. A typical scenario for handling an API request is matching the HTTP request with a corresponding handler routine which creates a response object, that is then -sent back to the client, encoded with JSON. The Nette\Application package can be -used to achieve this with Nette, although it is meant to be used mainly in MVP -applications. +sent back to the client, encoded with JSON. The `Nette\Application` package can +be used to achieve this with Nette, although it is meant to be used mainly in +MVP applications. Matching HTTP requests with handlers can be done using standard Nette URL routing -- we will create a Nette route for each API endpoint. Using the routing mechanism from Nette logically leads to implementing handler routines as Nette Presenter actions. Each presenter should serve logically related endpoints. -The last step is encoding the response as JSON. In Nette\Application, HTTP +The last step is encoding the response as JSON. In `Nette\Application`, HTTP responses are returned using the `Presenter::sendResponse()` method. We decided to write a method that calls `sendResponse` internally and takes care of the encoding. This method has to be called in every presenter action. An alternative From ad785cbb8320a0d7b8726e5b8cb50c2499b492ba Mon Sep 17 00:00:00 2001 From: Teyras Date: Wed, 11 Jan 2017 14:43:34 +0100 Subject: [PATCH 40/46] typo --- Rewritten-docs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 65b1040..9adb481 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -173,7 +173,7 @@ may have different environment to allow testing programs in more conditions. Incoming jobs are scheduled to particular worker depending on its capabilities and job requirements. -Job processing itself stars with obtaining source files and job configuration. +Job processing itself starts with obtaining source files and job configuration. The configuration is parsed into small tasks with simple piece of work. Evaluation itself goes in direction of tasks ordering. It is crucial to keep executive computer secure and stable, so isolated sandboxed environment is used From fbcfe96261913a6c44cafd633cff0717f65c4d26 Mon Sep 17 00:00:00 2001 From: Teyras Date: Wed, 11 Jan 2017 14:44:31 +0100 Subject: [PATCH 41/46] wording --- Rewritten-docs.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 9adb481..c23dc31 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -190,7 +190,7 @@ and achieved score (amount of awarded points). ## Requirements -There are bunch of different formal requirements for the system. Some of them +There are many different formal requirements for the system. Some of them are necessary for any system for source code evaluation, some of them are specific for university deployment and some of them arose during the ten year long lifetime of the old system. There are not many ways to improve CodEx @@ -208,18 +208,17 @@ System features represents directly accessible functionality to users of the system. They describe the evaluation system in general and also university addons (mostly administrative features). -#### Pure user requirements +#### End user requirements -- users have their own account in the system +- users have their own accounts in the system - system users can be members of multiple groups (reflecting courses or labs) - there is a database of exercises; teachers can create exercises including textual description, sample inputs and correct reference outputs (for example - "sum all numbers from given file and write the result to the standard - output") -- there is a list of assigned exercises in each group and interface to submit a - solution; teachers can assign an existing exercise to their class with some + "sum all numbers from given file and write the result to the standard output") +- there is a list of assigned exercises in each group and an interface to submit + a solution; teachers can assign an existing exercise to their class with some specific properties set (deadlines, etc.) -- user can see a list of submitted solutions for each assignment with +- users can see a list of submitted solutions for each assignment with corresponding results - teachers can specify way of computation grading points which will be awarded to the students depending on the quality of his/her solution for each From 543794bd490b184e6d47c27eb0da644e4c7cf045 Mon Sep 17 00:00:00 2001 From: Teyras Date: Wed, 11 Jan 2017 22:40:36 +0100 Subject: [PATCH 42/46] introduction and intended usage polishing --- Rewritten-docs.md | 134 ++++++++++++++++++++++++---------------------- 1 file changed, 69 insertions(+), 65 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index c23dc31..7277ece 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -61,22 +61,28 @@ knowledge are more suitable for this practical type of learning than others, and fortunately, programming is one of them. University education system is one of the areas where this knowledge can be -applied. In computer programming, there are several requirements such as the -code being syntactically correct, efficient and easy to read, maintain and -extend. Correctness and efficiency can be tested automatically to help teachers -save time for their research, but reviewing bad design, bad coding habits and -logical mistakes is really hard to automate and requires manpower. - -Checking programs written by students takes a lot of time and requires a lot of -mechanical, repetitive work. The first idea of an automatic evaluation system +applied. In computer programming, there are several requirements a program +should satify, such as the code being syntactically correct, efficient and easy +to read, maintain and extend. + +Checking programs written by students takes time and requires a lot of +mechanical, repetitive work -- reviewing source codes, compiling them and +running them through testing scenarios. It is therefore desirable to automate as +much of this work as possible. The first idea of an automatic evaluation system comes from Stanford University professors in 1965. They implemented a system which evaluated code in Algol submitted on punch cards. In following years, many similar products were written. -There are two basic ways of automatically evaluating code -- statically (check -the code without running it; safe, but not very precise) or dynamically (run the -code on testing inputs with checking the outputs against reference ones; needs -sandboxing, but provides good real world experience). +In today's world, properties like correctness and efficiency can be tested +automatically to a large extent. This fact should be exploited to help teachers +save time for tasks such as examining bad design, bad coding habits and logical +mistakes, which are difficult to perform automatically. + +There are two basic ways of automatically evaluating code -- statically +(checking the sourcecode without running it; safe, but not very precise) or +dynamically (running the code on test inputs and checking the correctness of +outputs ones; provides good real world experience, but requires extensive +security measures). This project focuses on the machine-controlled part of source code evaluation. First, general concepts of grading systems are observed, new requirements are @@ -84,8 +90,8 @@ specified and project with similar functionality are examined. Also, problems of the software previously used at Charles University in Prague are briefly discussed. With acquired knowledge from such projects in production, we set up goals for the new evaluation system, designed the architecture and implemented a -fully operational solution. The system is now ready for production testing at -the university. +fully operational solution based on dynamic evaluation. The system is now ready +for production testing at the university. ## Assignment @@ -95,13 +101,14 @@ Charles University in Prague. However, the application should be designed in a modular fashion to be easily extended or even modified to make other ways of usage possible. -The system should be capable of dynamic analysis of programming code. It means, -that following four basic steps have to be supported: +The system should be capable of dynamic analysis of submitted source codes. This +consists of following basic steps: 1. compile the code and check for compilation errors 2. run compiled binary in a sandbox with predefined inputs 3. check constraints on used amount of memory and time 4. compare program outputs with predefined values +5. award the code with a numeric score The project has a great starting point -- there is an old grading system currently used at the university (CodEx), so its flaws and weaknesses can be @@ -111,14 +118,14 @@ they are willing to consult ideas or problems during development with us. ### Intended usage The whole system is intended to help both teachers (supervisors) and students. -To achieve this, it is crucial to keep in mind typical usage scenarios of the -system and try to make these tasks as simple as possible. +To achieve this, it is crucial to keep in mind the typical usage scenarios of +the system and to try to make these tasks as simple as possible. -The system has a database of users. Each user has assigned a role, which -corresponds to his/her privileges. There are user groups reflecting structure of -lectured courses. Groups can be hierarchically ordered to reflect additional -metadata such as the academic year. For example, a reasonable group hierarchy -can look like this: +The system has a database of users. Each user is assigned a role, which +corresponds to his/her privileges. There are user groups reflecting the +structure of lectured courses. Groups can be hierarchically ordered to reflect +additional metadata such as the academic year. For example, a reasonable group +hierarchy could look like this: ``` Summer term 2016 @@ -130,22 +137,22 @@ Summer term 2016 ... ``` -In this example, students are members of the leaf groups, the higher level -entities are just for keeping the related groups together. The hierarchy -structure can be modified and altered to fit specific needs of the university or -any other organization, even the flat structure (i.e., no hierarchy) is -possible. One user can be part of multiple groups and on the other hand one -group can have multiple users. Each user can have a specific role for every -group in which is a member, overriding his/her default role in this context. - -Database of exercises (algorithmic problems) is another part of the project. -Each exercise consists of a text in multiple language variants, an evaluation -configuration and a set of inputs and reference outputs. Exercises are created -by instructed privileged users. Assigning an exercise to a group means to -choose one of the available exercises and specifying additional properties. An -assignment has a deadline (optionally a second deadline), a maximum amount of -points, a configuration for calculating the final score, a maximum number of -submissions, and a list of supported runtime environments (e.g., programming +In this example, students are members of the leaf groups and the higher level +nodes are just for keeping related groups together. The structure can be +modified and altered to fit specific needs of the university or any other +organization, even a flat structure is possible. One user can be a member of +multiple groups and have a different role in each of them (a student can attend +labs for several courses while also teaching one). + +A database of exercises (algorithmic problems) is another part of the project. +Each exercise consists of a text describing the problem in multiple language +variants, an evaluation configuration (machine-readable instructions on how to +evaluate solutions to the exercise) and a set of inputs and reference outputs. +Exercises are created by instructed privileged users. Assigning an exercise to a +group means choosing one of the available exercises and specifying additional +properties: a deadline (optionally a second deadline), a maximum amount of +points, a configuration for calculating the score, a maximum number of +submissions, and a list of supported runtime environments (e.g. programming languages) including specific time and memory limits for each one. Typical use cases for supported user roles are illustrated on following UML @@ -161,32 +168,29 @@ is described in more detail below to give readers solid overview of what have to happen during evaluation process. First thing users have to do is to submit their solutions through some user -interface. Then, the system checks assignment invariants (deadlines, count -of submissions, ...) and stores submitted files. The runtime environment is -automatically detected based on input files and suitable exercise configuration -variant is chosen (one exercise can have multiple variants, for example C and -Java languages). Matching exercise configuration is then used for taking care of -evaluation process. - -There is a pool of worker computers dedicated to processing jobs. Some of them -may have different environment to allow testing programs in more conditions. -Incoming jobs are scheduled to particular worker depending on its capabilities -and job requirements. - -Job processing itself starts with obtaining source files and job configuration. -The configuration is parsed into small tasks with simple piece of work. -Evaluation itself goes in direction of tasks ordering. It is crucial to keep -executive computer secure and stable, so isolated sandboxed environment is used -when dealing with unknown source code. When the execution is finished, results -are saved. - -Results from worker contains only output data from processed tasks (this could -be return value, consumed time, ...). On top of that, one value is calculated to -express overall quality of the tested job. It is used as points for final -student grading. Calculation method of this value may be different for each -assignment. Data presented back to users include overview of job parts (which -succeeded and which failed, optionally with reason like "memory limit exceeded") -and achieved score (amount of awarded points). +interface. Then, the system checks assignment invariants (deadlines, count of +submissions, ...) and stores submitted files. The runtime environment is +automatically detected based on input files and a suitable evaluation +configuration variant is chosen (one exercise can have multiple variants, for +example C and Java languages). This exercise configuration is then used for +taking care of evaluation process. + +There is a pool of worker computers dedicated to evaluation jobs. Each one of +them can support different environments and programming languages to allow +testing programs for as many platforms as possible. Incoming jobs are scheduled +to a worker that is capable of running the job. + +The worker obtains the solution and its evaluation configuration, parses it and +starts executing the contained instructions. It is crucial to keep the worker +computer secure and stable, so a sandboxed environment is used for dealing with +unknown source code. When the execution is finished, results are saved and the +submitter is notified. + +The output of the worker contains data about the evaluation, such as time and +memory spent on running the program for each test input and whether its output +was correct. The system then calculates a numeric score from this data, which is +presented to the student. If the solution is wrong (incorrect output, uses too +much memory,..), error messages are also displayed to the submitter. ## Requirements From caa310060857b85661e926db889107d188e4728e Mon Sep 17 00:00:00 2001 From: Teyras Date: Wed, 11 Jan 2017 22:50:38 +0100 Subject: [PATCH 43/46] typos --- Rewritten-docs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 7277ece..6f2899c 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -62,7 +62,7 @@ fortunately, programming is one of them. University education system is one of the areas where this knowledge can be applied. In computer programming, there are several requirements a program -should satify, such as the code being syntactically correct, efficient and easy +should satisfy, such as the code being syntactically correct, efficient and easy to read, maintain and extend. Checking programs written by students takes time and requires a lot of @@ -79,7 +79,7 @@ save time for tasks such as examining bad design, bad coding habits and logical mistakes, which are difficult to perform automatically. There are two basic ways of automatically evaluating code -- statically -(checking the sourcecode without running it; safe, but not very precise) or +(checking the source code without running it; safe, but not very precise) or dynamically (running the code on test inputs and checking the correctness of outputs ones; provides good real world experience, but requires extensive security measures). From 3aeb0336c989186cf0db169c5cb88e6e9ee483dd Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Wed, 11 Jan 2017 23:23:46 +0100 Subject: [PATCH 44/46] WIP: restruct user doc --- Rewritten-docs.md | 124 ++++++++++++++++++++++++++++++---------------- 1 file changed, 82 insertions(+), 42 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 7277ece..95e0aa5 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -2049,28 +2049,23 @@ Loading results means fetching archive from fileserver, parsing the main YAML file generated by worker and saving data to the database. Also, points are assigned by score calculator. -User documentation -================== -Web Application ---------------- +# User documentation @todo: Describe different scenarios of the usage of the Web App -### Terminology +@todo: Describe the requirements of running the web application (modern web browser, enabled CSS, JavaScript, Cookies & Local storage) + +## Terminology @todo: Describe the terminology: Instance, User, Group, Student, Supervisor, Admin -### Web application requirements - -@todo: Describe the requirements of running the web application (modern -web browser, enabled CSS, JavaScript, Cookies & Local storage) +## General basics +@todo: actions which are available for all users -### Scenario \#1: Becoming a user of ReCodEx - -#### How to create a user account? +### How to create a user account? You can create an account if you click on the “*Create account*” menu item in the left sidebar. You can choose between two types of @@ -2091,7 +2086,7 @@ When crating your account both ways, you must select an instance your account will belong to by default. The instance you will select will be most likely your university or other organization you are a member of. -#### How to get into ReCodEx? +### How to get into ReCodEx? To log in, go to the homepage of ReCodEx and in the left sidebar choose the menu item “*Sign in*”. Then you must enter your credentials into one @@ -2102,7 +2097,7 @@ Authentication Service (CAS), you should put your student’s number and your CAS password into the second form called “Sign into ReCodEx using CAS UK”. -#### How do I sign out of ReCodEx? +### How do I sign out of ReCodEx? If you don’t use ReCodEx for a whole day, you will be logged out automatically. However, we recommend you sign out of the application @@ -2113,7 +2108,7 @@ have to expand the sidebar with a button next to the “*ReCodEx*” title @todo: Simon's image -#### What to do when you cannot remember your password? +### What to do when you cannot remember your password? If you can’t remember your password and you don’t use CAS UK authentication, then you can reset your password. You will find a link @@ -2132,7 +2127,7 @@ If you sign in through CAS UK, then please follow the instructions provided by the administrators of the service described on their website. -#### How to configure your account? +### How to configure your account? There are several options you have to edit your user account. @@ -2145,87 +2140,78 @@ You can access the settings page through the “*Settings*” button right under your name in the left sidebar. -### Scenario \#2: User is a student +## Student @todo: describe what it means to be a “student” and what are the student’s rights -#### How to join a group for my class? +### How to join a group for my class? @todo: How to join a specific group -#### Which assignments do I have to solve? +### Which assignments do I have to solve? @todo: Where the student can find the list of the assignment he is expected to solve, what is the first and second deadline. -#### Where can I see details of my classes’ group? +### Where can I see details of my classes’ group? @todo: Where can the user see groups description and details, what information is available. -#### How to submit a solution of an assignment? +### How to submit a solution of an assignment? @todo: How does a student submit his solution through the web app -#### Where are the results of my solutions? +### Where are the results of my solutions? @todo: When the results are ready and what the results mean and what to do about them, when the user is convinced, that his solution is correct although the results say different -#### How can I discuss my solution with my teacher/group’s supervisor directly through the web application? +### How can I discuss my solution with my teacher/group’s supervisor directly through the web application? @todo: Describe the comments thread behavior (public/private comments), who else can see the comments, how notifications work (*not implemented yet*!). -### Scenario \#3: User is supervisor of a group +## Group supervisor @todo: describe what it means to be a “supervisor” of a group and what are the supervisors rights -#### How do I become a supervisor of a group? +### How do I become a supervisor of a group? @todo: How does a user become a supervisor of a group? -#### How to add or remove a student to my group? +### How to add or remove a student to my group? @todo: How to add a specific student to a given group -#### How do I add another supervisor to my group? - -@todo: who can add another supervisor, what would be the rights of the -second supervisor - -#### How do I create a subgroup of my group? - -@todo: What it means to create a subgroup and how to do it. - -#### How do I assign an exercise to my students? +### How do I assign an exercise to my students? @todo: Describe how to access the database of the exercises and what are the possibilities of assignment setup – availability, deadlines, points, score configuration, limits -#### How do I configure the limits of an assignment and how to choose appropriate limits? +### How do I configure the limits of an assignment and how to choose appropriate limits? @todo: Describe the form and explain the concept of reference solutions. How to evaluate the reference solutions for the exercise right now (to get the up-to-date information). -#### How can I assign some exercises only to some students of the group? +### How can I assign some exercises only to some students of the group? @todo: Describe how to achieve this using subgroups -#### How can I see my students’ solutions? +### How can I see my students’ solutions? @todo Describe where all the students’ solutions for a given assignment can be found, where to look for all solutions of a given student, how to see results of a specific student’s solution’s evaluation result. -#### Can I assign points to my students’ solutions manually instead of depending on automatic scoring? +### Can I assign points to my students’ solutions manually instead of depending on automatic scoring? @todo If and how to change the score of a solution – assignment settings, setting points, bonus points, accepting a solution (*not @@ -2233,13 +2219,67 @@ implemented yet!*). Describe how the student and supervisor will still be able to see the percentage received from the automatic scoring, but the awarded points will be overridden. -#### How can I discuss student’s solution with him/her directly through the web application? +### How can I discuss student’s solution with him/her directly through the web application? @todo: Describe the comments thread behavior (public/private comments), who else can see the comments -- same as from the student perspective -### Writing job configuration +## Group administrator + +@todo: who is this? + +### How do I add another supervisor to my group? + +@todo: who can add another supervisor, what would be the rights of the +second supervisor + +### How do I create a subgroup of my group? + +@todo: What it means to create a subgroup and how to do it. + + +## Superadministrator + +Superadmin is user with the most priviledges and as such superadmin should be +quite unique role. Ideally there should be only one of this kind, used with +special caution and adequate security. With this stated it is obvious that +superadmin can perform any action the API is capable of. + +### Users management + +There are only few roles to which users can belong in ReCodEx. Basically there +are only three: _student_, _supervisor_, and _superadmin_. Base role is student +which is assigned to every registered user. Roles are stored in database +alongside other information about user. One user always has only one role at the +time. At first startup of ReCodEx administrator should create his account and +then change role in database by hand. After that manual intervention into +database should never be needed. + +There is a little catch in groups and instances management. Groups can have +admins and supervisors. This setting is valid only per one particular group and +has to be separated from basic role system. This implies that supervisor in one +group can be student in another and simultaneously have global supervisor role. +Changing role from student to supervisor and back is done automatically by +application and should not be managed by hand in database! Previously stated +information can be applied to instances as well, but instances can only have +admins. + +Roles description: + +- Student -- Default role which is used for newly created accounts. Student can + join or leave public groups and submit solutions of assigned exercises. +- Supervisor -- Inherits all permissions from student role. Can manage groups to + which he/she belongs to. Supervisor can also view and change groups details, + manage assigned exercises, view students in group and their solutions for + assigned exercises. On top of that supervisor can create/delete groups too, + but only as subgroup of groups he/she belongs to. +- Superadmin -- Inherits all permissions from supervisor role. Most powerful + user in ReCodEx who should be able to do everything which is provided by + application. + + +## Writing job configuration To run and evaluate an exercise the backend needs to know the steps how to do that. This is different for each environment (operation system, programming From 7c6863bd900dd8c20690d8f532ec7042eac1728e Mon Sep 17 00:00:00 2001 From: Teyras Date: Wed, 11 Jan 2017 23:26:54 +0100 Subject: [PATCH 45/46] job config requirement elaborated --- Rewritten-docs.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 7637b3a..11d8704 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -256,7 +256,11 @@ addons (mostly administrative features). workload for administrator (maintain separate installation for every course) and high hardware occupation - advanced low-level evaluation flow configuration with high-level abstraction - layer for ordinary configuration cases + layer for ordinary configuration cases; the configuration should be able to + express more complicated flows than just compiling a source code and running + the program against test inputs -- for example, some exercises need to build + the source code with a tool, run some tests, then run the program through + another tool and perform additional tests - use of modern technologies with state-of-the-art compilers ### Technical details From 87e4bcd39a4fca3eadbb4748e9a3b6ced2bd7150 Mon Sep 17 00:00:00 2001 From: Teyras Date: Thu, 12 Jan 2017 00:10:40 +0100 Subject: [PATCH 46/46] rewording bits and pieces of analysis --- Rewritten-docs.md | 94 +++++++++++++++++++++++------------------------ 1 file changed, 47 insertions(+), 47 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 11d8704..795d883 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -393,14 +393,14 @@ recruiters and also some universities. # Analysis -None of the existing projects we came across is capable of all the required -features for the new system. There is no grading system which is designed to -support a complicated evaluation pipeline, so this unexplored field has to be -designed with caution. Also, no project is modern and extensible enough so it -could be used as a base for the new system. After considering all these facts, -it is clear that a new system has to be written from scratch. This implies, that -only a subset of all the features will be implemented in the first version, the -others coming in the following releases. +None of the existing projects we came across fulfills all the requested features +for the new system. There is no grading system which supports arbitrary-length +evaluation pipeline, so we have to implement this feature ourselves, cautiously +treading through unexplored fields. Also, no existing solution is extensible +enough to be used as a base for the new system. After considering all these +facts, it is clear that a new system has to be written from scratch. This +implies that only a subset of all the features will be implemented in the first +version, the others coming in the following releases. Gathered features are categorized based on priorities for the whole system. The highest priority has main functionality similar to current CodEx. It is a base @@ -417,7 +417,7 @@ side) and command-line submit tool. Plagiarism detection is not likely to be part of any release in near future unless someone other makes the engine. The detection problem is too hard to be solved as part of this project. -We named the new project as **ReCodEx -- ReCodEx Code Examiner**. The name +We named the new project **ReCodEx -- ReCodEx Code Examiner**. The name should point to the old CodEx, but also reflect the new approach to solve issues. **Re** as part of the name means redesigned, rewritten, renewed, or restarted. @@ -446,28 +446,29 @@ of the system. ## Basic concepts -The system is designed as a web application. The requirements say, that user -interface must be accessible from students' computers without installation -additional software. This immediately implies that users are connected to the -internet, so it is used as communication medium. Today, there are two main ways -of designing graphical user interface -- native app or web page. Make nice and -multi-platform application with graphical interface is almost impossible because -of large number of different environments. Also, these applications often -requires installation or at least downloading its files (sources or binaries). -However, creating web application is easier, because every personal computer has -a internet browser installed. Also, browsers supports sort of unified and -standardized environment of HTML5 and JavaScript. CodEx is also web application -and everybody seems happy about it. There are other communicating channels every -user have available like email or git, but they are totally inappropriate for -designing user interfaces on top of them. - -The application interacts with users. From the project assignment is clear, that -the system has to keep personalized data about users and adapt presented content -according to this knowledge. User data cannot be publicly visible, so that -implies necessity of user authentication. The application also has to support -multiple ways of authentication (university authentication systems, a company -LDAP server, an OAuth server...) and permit adding more security measures in the -future, such as two-factor authentication. +The system is designed as a web application. The requirements say that the user +interface must be accessible from students' computers without the need to +install additional software. This immediately implies that users have to be +connected to the internet, so it is used as communication medium. Today, there +are two main ways of designing graphical user interface -- as a native +application or a web page. Creating a nice and multi-platform application with +graphical interface is almost impossible because of the large number of +different environments. Also, these applications often requires installation or +at least downloading its files (sources or binaries). On the other hand, +distributing a web application is easier, because every personal computer has an +internet browser installed. Also, browsers support an (mostly) unified and +standardized environment of HTML5 and JavaScript. CodEx is also a web +application and everybody seems satisfied with it. There are other communicating +channels most programmers have available, such as e-mail or git, but they are +inappropriate for designing user interfaces on top of them. + +The application interacts with users. From the project assignment it is clear, +that the system has to keep personalized data about users and adapt presented +content according to this knowledge. User data cannot be publicly visible, so +that implies necessity of user authentication. The application also has to +support multiple ways of authentication (university authentication systems, a +company LDAP server, an OAuth server...) and permit adding more security +measures in the future, such as two-factor authentication. User data also includes a privilege level. From the assignment it is required to have at least two roles, _student_ and _supervisor_. However, it is wise to add @@ -481,22 +482,21 @@ duties related with lecturing of labs are already associated with supervisors, so this role seems not so useful. In addition, no one requested more than three level privilege scheme. -School labs are lessons for some students lead by one (or sometimes two or -three) supervisors. Students have the same homework and supervisors are -evaluating its solutions. This organization has to be carried into the new -system. Counterpart to real labs are virtual groups. This concept was already -discussed in previous chapter including need for hierarchical structure of -groups. Right for attending labs has only a person, who is student of the -university and is recorded in university information system. To allow -restriction of group members in ReCodEx, there two type of groups -- _public_ -and _private_. Public groups are open for every registered users, but to become -a member of private group one of its supervisors have to add that user. This -could be done automatically at beginning of the term with data from information -system, but unfortunately there is no such API yet. However, creating this API -is now considered by university leadership. Another just as good solution for -restricting membership of a group is to allow anyone join the group with -supplementary confirmation of supervisors. It has no additional benefits, so -approach with public and private groups is implemented. +School labs are lessons for some students lead by supervisors. Students have the +same homework and supervisors are evaluating its solutions. This organization +has to be carried into the new system. Counterpart to real labs are virtual +groups. This concept was already discussed in previous chapter including need +for hierarchical structure of groups. Right for attending labs has only a +person, who is student of the university and is recorded in university +information system. To allow restriction of group members in ReCodEx, there two +type of groups -- _public_ and _private_. Public groups are open for every +registered users, but to become a member of private group one of its supervisors +have to add that user. This could be done automatically at beginning of the term +with data from information system, but unfortunately there is no such API yet. +However, creating this API is now considered by university leadership. Another +just as good solution for restricting membership of a group is to allow anyone +join the group with supplementary confirmation of supervisors. It has no +additional benefits, so approach with public and private groups is implemented. Supervisors using CodEx in their labs usually set minimum amount of points required to get a credit. These points can be get by solving assigned exercises.