diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 9e48904..f9ecc13 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -43,10 +43,10 @@ used for programming classes at the Faculty of Mathematics and Physics, Charles University. However, the application should be designed in a modular fashion so that it can be easily extended to make other ways of using it possible. -The project has a great starting point -- there is an old grading system -currently used at our university, so its mistakes and weaknesses can be -adressed. Furthermore, many teachers are willing to use and test the new system. -Following requirements were collected both from our personal experience with +The project has a great starting point -- there is an old grading system +currently used at our university (CodEx), so its mistakes and weaknesses can be +adressed. Furthermore, many teachers are willing to use and test the new system. +Following requirements were collected both from our personal experience with CodEx and from teachers' requests. **Basic grading system requirements:** @@ -54,7 +54,10 @@ CodEx and from teachers' requests. These are features that are necessary for any system for evaluation of programming homework assignments used in a university programming course. + - creating exercises including textual description, sample inputs and correct reference outputs (for example "sum all numbers from given file and write the @@ -78,33 +81,24 @@ way courses are organized at our university -- for example, users have roles in the system and students are divided into groups that correspond to lab groups. -However, further requirements arose during the ten year long lifetime of the old -system. There are not many ways to improve it from the perspective of a student, -but a lot of feature requests came from administrators and supervisors. -Collected ideas were mostly gathered from meetings with faculty staff involved +However, further requirements arose during the ten year long lifetime of the old +system. There are not many ways to improve it from the perspective of a student, +but a lot of feature requests came from administrators and supervisors. +Collected ideas were mostly gathered from meetings with faculty staff involved with the current system. -**Drawbacks of CodEx:** - -@todo merge this with the CodEx entry in "Related work" - -- there has to be a separate installation of the system for some courses (e.g. - Java programming and .NET programming) that do not use the same sandbox as the - majority of courses. - **Requested features for the new system:** - logging in through a university authentication system (e.g. LDAP) - support for multiple programming environments at once to avoid unacceptable - workload for administrator and hardware occupation -- elimination of the need to maintain separate installations for Java and C# - programming courses + workload for administrator (maintain separate installations for many courses) + and high hardware occupation - localization (both UI and exercises) - Markdown support for exercise texts - tagging exercises and search by tags - comments, comments, comments (exercises, tests, solutions, ...) - edit student solution and privately resubmit it -- resubmit solution with saving all (including temporary) results +- resubmit solution with saving all results (including temporary ones) - mark one student solution as accepted (used for grading this assignment) - web and command-line submit tool - SIS (university information system) integration for fetching personal user @@ -115,11 +109,11 @@ with the current system. - use of modern technologies with state-of-the-art compilers The survey shows that the system is used in many different ways, but the core -functionality is the same for all. When the system is ready, it is likely that -new ideas are developed. Thus the system must be designed to be easily -extendable, so everyone can develop his dream feature. This also means, that -widely used programming languages and techniques should be used, so users can -quickly understand the code and make changes. +functionality is the same for all of them. When the system is ready, it is +likely that new ideas are figured out, thus the system must be designed to be +easily extendable, so everyone can develop his dream feature. This also means, +that widely used programming languages and techniques should be used, so users +can quickly understand the code and make changes. To find out current state in the field of automatic grading systems, let's do a short survey at universities, programming contests or online tools. @@ -239,9 +233,12 @@ to CodEx, previous evaluation solution, but also reflect new approach to solve issues. **Re** as part of the name means redesigned, rewritten, renewed or restarted. -From the previous research, we set up several goals, which a new system should -have. They mostly reflect drawbacks of current version of CodEx and reasonable -wishes of university users. Most notable features are following: +At this point there is a clear idea how the new system will be used and what are +major enhancements for future releases. With this in mind, the overall +architecture can be sketched. From the previous research, we set up several +goals, which a new system should have. They mostly reflect drawbacks of current +version of CodEx and reasonable wishes of university users. Most notable +features are following: - modern HTML5 web frontend written in Javascript using a suitable framework - REST API implemented in PHP, communicating with database, backend and file @@ -253,9 +250,88 @@ wishes of university users. Most notable features are following: - evaluation procedure configured in YAML file, compound of small tasks connected into arbitrary oriented acyclic graph -#### Usage design +#### Intended usage + +Whole system is intended to help both supervisors and students. To achieve this, +it is crucial to keep in mind typical usage scenarios of the system and try to +make these typical tasks as simple as possible. To synchronize visions of +readers, basic concepts are recapitulated. + +First of all, the system has database of users. Each user has assigned a role, +which correspond to his/her privileges. User can be logged in via local +authentication service or university system. There are groups of users, which +corresponds to lectured courses. Groups can be hierarchically ordered to reflect +additional metadata like academic year. For example, reasonable group hierarchy +is like this: + +``` +Summer term 2016 +├── Language C# and .NET platform +│   ├── Labs Monday 10:30 +│   └── Labs Thursday 9:00 +├── Programming I +│   ├── Labs Monday 14:00 + ... + +``` -@todo: describe detailed usage ... grading and this kind of stuff +In this example, student users are part of the leaf groups, higher groups are +just for keeping related groups together. The hierarchy tree can be modified and +altered to fit specific needs for each organization, even the flat structure is +possible. + +One user can be part of multiple groups and also one group can have multiple +users. Each user in a group has a role which defines its capabilities. +Priviledged user can assign a new exercise in his/her group, change assignment +details, view results of other users and manually change them. Normal user can +join a group, get list of assigned exercises, view assignment detail, submit +his/her solution and of course view the results. + +Database of exercises (algorithmic problems) is another part of the project. +Each exercise consists of text in multiple language variants, evaluation +configuration and set of inputs and reference outputs. Exercises are created by +instructed priviledged users. Assigning exercise to a group means choose one of +the exercises in the list and specify additional data. Assignment has a +deadline, maximum amount of points and configuration for calculating the final +amount, number of tries and supported runtimes (programming languages) including +specific time and memory limits for sandboxed tasks. + +##### Exercise evaluation chain + +The most important part of the application is evaluating exercises for solutions +submitted by users. For imaginary system architecture _UI_, _API_, _Broker_ and +_Worker_ this goes as follows. + +First thing users have to do is to submit their solutions to _UI_ which provides +interface to upload files and then submit them. _UI_ sends a request to _API_ +that user wants to evaluate assignment with provided files. + +_API_ checks the assignment invariants (deadlines, count of submissions, ...) +and stores submitted files. The runtime environment is automatically detected +based on input files and suitable exercise configuration variant is chosen (one +exercise can have multiple variants, for example C and Java languages). Matching +exercise configuration is then send to _Broker_ alongside solution source files. + +_Broker_ has to find suitable _Worker_ for execution of this particular +submission. This decission is made based on capabilities of each _Worker_ and +job requirements. When a match is found, the job is held until the _Worker_ is +jobless and can receive an evaluation request. + +_Worker_ gets evaluation request with source files and job configuration. The +configuration is parsed into small tasks with simple piece of work. Evaluation +itself goes in direction of tasks ordering. It is crucial to keep _Worker_ +machine secure and stable, so isolated sandboxed environment is used when +dealing with unknown source code. When the execution is finished, results are +uploaded back. + +_API_ is notified about finished job from _Broker_. The results are parsed and +results of important tasks (comparing actual and expected results) saved into +database. Also, points are calculated depending on solution correctness and +assignment configuration. + +_UI_ then only displays results summary fetched from the _API_. Presented data +includes overview which part succeeded and which failed (optionally with reason +like "memory limit exceeded") and amount of awarded points. ### Solution concepts analysis @@ -1159,4 +1235,7 @@ used. - hw-group-id: group1 chdir: ${EVAL_DIR} ``` +