diff --git a/Rewritten-docs.md b/Rewritten-docs.md index ef77ff6..4c4e2ed 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -115,7 +115,21 @@ currently used at the university (CodEx), so its flaws and weaknesses can be addressed. Furthermore, many teachers desire to use and test the new system and they are willing to consult ideas or problems during development with us. -### Intended usage +## Current system + +Currently used grading solution at the Faculty of Mathematics and Physics of +the Charles University in Prague which was implemented in 2006 by a group +of students. It is called [CodEx -- The Code Examiner](http://codex.ms.mff.cuni.cz/project/) +and it has been used with some improvements since then. The original plan was +to use the system only for basic programming courses, but there was a demand +for adapting it for many different subjects. + +CodEx is based on dynamic analysis. It features a web-based interface, where +supervisors can assign exercises to their students and the students have a time +window to submit their solutions. Each solution is compiled and run in sandbox +(MO-Eval). The metrics which are checked are: correctness of the output, time +and memory limits. It supports programs written in C, C++, C#, Java, Pascal, +Python and Haskell. The whole system is intended to help both teachers (supervisors) and students. To achieve this, it is crucial to keep in mind the typical usage scenarios of @@ -123,26 +137,7 @@ the system and to try to make these tasks as simple as possible. The system has a database of users. Each user is assigned a role, which corresponds to his/her privileges. There are user groups reflecting the -structure of lectured courses. Groups can be hierarchically ordered to reflect -additional metadata such as the academic year. For example, a reasonable group -hierarchy could look like this: - -``` -Summer term 2016 -|-- Language C# and .NET platform -|   |-- Labs Monday 10:30 -|   `-- Labs Thursday 9:00 -|-- Programming I -|   |-- Labs Monday 14:00 - ... -``` - -In this example, students are members of the leaf groups and the higher level -nodes are just for keeping related groups together. The structure can be -modified and altered to fit specific needs of the university or any other -organization, even a flat structure is possible. One user can be a member of -multiple groups and have a different role in each of them (a student can attend -labs for several courses while also teaching one). +structure of lectured courses. A database of exercises (algorithmic problems) is another part of the project. Each exercise consists of a text describing the problem in multiple language @@ -155,12 +150,50 @@ points, a configuration for calculating the score, a maximum number of submissions, and a list of supported runtime environments (e.g. programming languages) including specific time and memory limits for each one. -Typical use cases for supported user roles are illustrated on following UML -diagram: +Typical use cases for supported user roles are following: + +- **student** + - join a group + - get assignments in group + - submit solution to assignment + - view solution results +- **supervisor** + - create exercise + - assign exercise to group, modify assignment + - view all results in group + - alter automatic solution grading +- **administrator** + - create groups + - alter user privileges + - check system logs, upgrades and other management + +Current system is old, but robust. There were no major security incidents +during its production usage. However, from today's perspective there are +several drawbacks. The main ones are: -![System use case diagram](https://github.com/ReCodEx/wiki/raw/master/images/System_use_case.png) +- **web interface** -- The web interface is simple and fully functional. But + rapid development in web technologies opens new horizons of how web interface + can be made. +- **web API** -- CodEx offers a very limited XML API based on outdated + technologies that is not sufficient for users who would like to create custom + interfaces such as a command line tool or mobile application. +- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system + calls and blocking the bad ones. This can be easily done for single-threaded + applications, but proves difficult with multi-threaded ones. In present day, + parallelism is a very important area of computing, so there is requirement to + test multi-threaded applications too. +- **instances** -- Different ways of CodEx usage scenarios requires separate + instances (Programming I and II, Java, C#, etc.). This configuration is not + user friendly (students have to register in each instance separately) and + burdens administrators with unnecessary work. CodEx architecture does not + allow sharing hardware between instances, which results in an inefficient use + of hardware for evaluation. +- **task extensibility** -- There is a need to test and evaluate complicated + programs for classes such as Parallel programming or Compiler principles, + which have a more difficult evaluation chain than simple + compilation/execution/evaluation provided by CodEx. -#### Exercise evaluation chain +### Exercise evaluation chain The most important part of the system is evaluation of solutions submitted by students. Concepts of consecutive steps from source code to final results @@ -203,27 +236,23 @@ from administrators and supervisors. The ideas were gathered mostly from our personal experience with the system and from meetings with faculty staff involved with the current system. -For clear arrangement all the requirements and wishes are presented grouped by -categories. +In general, CodEx features should be preserved, so only differences are +presented here. For clear arrangement all the requirements and wishes are +presented grouped by categories. ### System features System features represents directly accessible functionality to users of the -system. They describe the evaluation system in general and also university +system. They describe the evaluation system in general and also university addons (mostly administrative features). #### End user requirements -- users have their own accounts in the system -- system users can be members of multiple groups (reflecting courses or labs) +- group hierarchy -- @todo: copy text from +[here](https://github.com/ReCodEx/wiki/wiki/Rewritten-docs/87e4bcd39a4fca3eadbb4748e9a3b6ced2bd7150#intended-usage) - there is a database of exercises; teachers can create exercises including textual description, sample inputs and correct reference outputs (for example "sum all numbers from given file and write the result to the standard output") -- there is a list of assigned exercises in each group and an interface to submit - a solution; teachers can assign an existing exercise to their class with some - specific properties set (deadlines, etc.) -- users can see a list of submitted solutions for each assignment with - corresponding results - teachers can specify way of computation grading points which will be awarded to the students depending on the quality of his/her solution for each assignment extra @@ -237,7 +266,7 @@ addons (mostly administrative features). - localization of all texts (UI and exercises) - Markdown support for creating exercise texts - tagging exercises in database and search by these tags -- comments, comments, comments (exercises, tests, solutions, ...) +- comments of exercises, tests and solutions - plagiarism detection #### Administrative requirements @@ -263,12 +292,12 @@ addons (mostly administrative features). another tool and perform additional tests - use of modern technologies with state-of-the-art compilers -### Technical details +### Nonfunctional requirements -Technical details are requirements of technical character with no direct mapping -to visible parts of system. In ideal word, users should not know about these if -they work properly, but would be at least annoyed if these requirements were not -met. Most notably they are these ones: +Nonfunctional requirements are requirements of technical character with no +direct mapping to visible parts of system. In ideal word, users should not know +about these if they work properly, but would be at least annoyed if these +requirements were not met. Most notably they are these ones: - user interface of the system accessible on users' computers without installation of any kind of additional software @@ -300,48 +329,6 @@ This is not a complete list of available evaluators, but only a few projects which are used these days and can be an inspiration for our project. Each project from the list has a brief description and some key features mentioned. -### CodEx - -Currently used grading solution at the Faculty of Mathematics and Physics of -the Charles University in Prague which was implemented in 2006 by a group -of students. It is called [CodEx -- The Code Examiner](http://codex.ms.mff.cuni.cz/project/) -and it has been used with some improvements since then. The original plan was -to use the system only for basic programming courses, but there was a demand -for adapting it for many different subjects. - -CodEx is based on dynamic analysis. It features a web-based interface, where -supervisors can assign exercises to their students and the students have a time -window to submit their solutions. Each solution is compiled and run in sandbox -(MO-Eval). The metrics which are checked are: correctness of the output, time -and memory limits. It supports programs written in C, C++, C#, Java, Pascal, -Python and Haskell. - -Current system is old, but robust. There were no major security incidents -during its production usage. However, from today's perspective there are -several drawbacks. The main ones are: - -- **web interface** -- The web interface is simple and fully functional. But - rapid development in web technologies opens new horizons of how web interface - can be made. -- **web API** -- CodEx offers a very limited XML API based on outdated - technologies that is not sufficient for users who would like to create custom - interfaces such as a command line tool or mobile application. -- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system - calls and blocking the bad ones. This can be easily done for single-threaded - applications, but proves difficult with multi-threaded ones. In present day, - parallelism is a very important area of computing, so there is requirement to - test multi-threaded applications too. -- **instances** -- Different ways of CodEx usage scenarios requires separate - instances (Programming I and II, Java, C#, etc.). This configuration is not - user friendly (students have to register in each instance separately) and - burdens administrators with unnecessary work. CodEx architecture does not - allow sharing hardware between instances, which results in an inefficient use - of hardware for evaluation. -- **task extensibility** -- There is a need to test and evaluate complicated - programs for classes such as Parallel programming or Compiler principles, - which have a more difficult evaluation chain than simple - compilation/execution/evaluation provided by CodEx. - ### Progtest [Progtest](https://progtest.fit.cvut.cz/) is private project of [FIT