diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 01fe9ce..bacd44f 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -78,18 +78,13 @@ the code without running it; safe, but not very precise) or dynamically (run the code on testing inputs with checking the outputs against reference ones; needs sandboxing, but provides good real world experience). - - This project focuses on the machine-controlled part of source code evaluation. -First, the problems of the software used at our university previously were -discussed and similar projects at other educational institutions were examined. -With acquired knowledge from such projects in production, we set up goals for -the new evaluation system, designed the architecture and implemented a fully -operational solution. The system is now ready for production testing -at our university. +First, the problems of the software used at Charles University in Prague +previously were discussed and similar projects at other educational institutions +were examined. With acquired knowledge from such projects in production, we set +up goals for the new evaluation system, designed the architecture and +implemented a fully operational solution. The system is now ready for production +testing at our university. ## Assignment @@ -102,80 +97,87 @@ of using it possible. The project has a great starting point -- there is an old grading system currently used at the university (CodEx), so its flaws and weaknesses can be addressed. Furthermore, many teachers are willing to use and test the new -system. Following requirements were collected both from our personal experience +system. Following requirements were collected both from our personal experience with CodEx and from teachers' requests. ## Requirements -@todo: sekce kde budou ty pozadavky (o neco podrobneji nez ted). Navrhoval bych -to rozdelit minimalne na funkcni a nefunkcni pozadavky, ale i ty funkcni je -mozna vhodne rozdelit na ciste uzivatelske (ulohy se maji bodovat, studenti -dostanou od cviciho komentar, ...) a administracne-technicke (napojeni na SIS a -LDAP, rizeni pristupu pomoci roli,...). - -### Basic grading system requirements: +There are bunch of different requirements for the system. Some of them are +features which are necessary for any system for evaluation of programming coding +assignments. Some of them are specific for university deployment and some are +wishes for new features collected for period of CodEx operation. + +CodEx satisfies all the basic requirements and a few more that originate from +the way courses are organized at university environment -- for example students +are divided into groups that correspond to lab groups. New wishes arose during +the ten year long lifetime of the old system. There are not many ways to improve +it from the perspective of a student, but a lot of feature requests came from +administrators and supervisors. The ideas were mostly gathered from meetings +with faculty staff involved with the current system. + +For clear arragement all the requirements and wishes are presented grouped by +categories. + +### System features + +System features are requirements directly accessible to users of the system. +They describe the evaluation system in general and also university addons +(mostly administrative features). + +#### Pure user requirements + +- system users can be members of multiple groups (reflecting courses or labs) +- there is a database of exercises; teachers can create exercises including + textual description, sample inputs and correct reference outputs (for example + "sum all numbers from given file and write the result to the standard + output") +- there is a list of assigned exercises in each group and interface to submit a + solution; teachers can assign an existing exercise to their class with some + specific properties set (deadlines, etc.) +- there is a list of submitted solutions for each assignment with corresponding + results +- teachers can specify scale of points which will be awarted to the students + depending on the correctness of his/her solution for each assignment extra + (expressed in percentage points) +- teachers can view detailed data about their students (users of a their groups) + including all submitted solutions; also, each of the solution can be manually + reviewed, commented and assigned additional points (positive or negative) +- one particular solution can be marked as accepted (used for grading this + assignment) +- teacher can edit student solution and privately resubmit it; optionaly saving + all results (including temporary ones) +- localization of all texts (UI and exercises) +- Markdown support for creating exercise texts +- tagging exercises in database and search by these tags +- comments, comments, comments (exercises, tests, solutions, ...) +- plagiarism detection -These are the features which are necessary for any system for evaluation of -programming coding assignments used in any university programming course: +#### Administrative requirements -- students can use an intuitive user interface for interaction with the system, +- users can use an intuitive user interface for interaction with the system, mainly for viewing assigned exercises, uploading their own solutions to the assignments, and viewing the results of the solutions after an automatic - evaluation is finished -- teachers can create exercises including textual description, sample inputs and - correct reference outputs (for example "sum all numbers from given file and - write the result to the standard output") -- teachers can assign an existing exercise to their class with some specific - properties set (deadlines, etc.) -- teachers can specify their scale of points which will be awarted to the - students depending on the correctness of his/her solution (expressed in - percentage points) -- teachers can view all of the solutions their students submitted and also the - results of the evaluations and they can override the automatically assigned - points to the solutions manually -- teachers can see the statistics of their classes and individual students of - these claseese -- administrators can depend on a safe environment in which the students' - solutions will be executed + evaluation is finished; wanted two interfaces are web and command-line based - administrators can manage users with support of roles (at least two -- _student_ and _supervisor_) - -CodEx satisfies all these requirements and a few more that originate from the -way courses are organized at our university -- for example, users have roles -(_student_, _supervisor_ and _administrator_) that determine their capabilities -in the system and students are divided into groups that correspond to lab -groups. - -However, further requirements arose during the ten year long lifetime of the old -system. There are not many ways to improve it from the perspective of a student, -but a lot of feature requests came from administrators and supervisors. The -ideas were mostly gathered from meetings with faculty staff involved with the -current system. - -### Requested features for the new system: - -First, there is a list of ideas influencing system capabilities: - - logging in through a university authentication system (e.g. LDAP) +- SIS (university information system) integration for fetching personal user + data +- administrators can depend on a safe environment in which the students' + solutions will be executed - support for multiple programming environments at once to avoid unacceptable workload for administrator (maintain separate installations for many courses) and high hardware occupation -- localization (both UI and exercises) -- Markdown support for exercise texts -- tagging exercises and search by tags -- comments, comments, comments (exercises, tests, solutions, ...) -- edit student solution and privately resubmit it -- resubmit solution with saving all results (including temporary ones) -- mark one student's solution as accepted (used for grading this assignment) -- web and command-line submission tool -- SIS (university information system) integration for fetching personal user data -- plagiarism detection - advanced low-level evaluation flow configuration with high-level abstraction layer for ordinary configuration cases - use of modern technologies with state-of-the-art compilers -Then, there are also some requirements of technical character with no direct -mapping to visible parts of system. Most notably they are these ones: +### Technical details + +Technical details are requirements of technical character with no direct mapping +to visible parts of system. In ideal word, users should not know about these if +they work properly, but would be at least annoyed if these requirements were not +met. Most notably they are these ones: - user interface of the system accessible on users' computers without installation of any kind of additional software @@ -188,16 +190,19 @@ melo byt lepe popsano, ze je potreba oddelit UI od aplikacni logiky, aby si nekdo mohl napsat jine UI, ze je potreba mit backend, ktery bude skalovat na vykon a dovoli pridavani dalsich platforem atd. +### Conclusion + The survey shows that the system is used in many different ways, but the core -functionality is the same for all of them. When the system is ready it is -likely that there will be new ideas of how to use the system and thus the system -must be designed to be easily extendable, so everyone can develop their own feature. -This also means that widely used programming languages and techniques should be used, -so users can quickly understand the code and make changes. +functionality is the same for all of them. When the system is ready it is likely +that there will be new ideas of how to use the system and thus the system must +be designed to be easily extendable, so everyone can develop their own feature. +This also means that widely used programming languages and techniques should be +used, so users can quickly understand the code and make changes. To find out the current state in the field of automatic grading systems we did a short survey at universities, programming contests, and other available tools. + ## Related work This is not a complete list of available evaluators, but only a few projects @@ -1663,9 +1668,9 @@ used. name: "isolate" limits: - hw-group-id: group1 - chdir: ${EVAL_DIR} + chdir: ${EVAL_DIR} ```