diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 3362d46..0c1d836 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -86,16 +86,15 @@ corresponds to his/her privileges. There are user groups reflecting the structure of lectured courses. A database of exercises (algorithmic problems) is another part of the project. -Each exercise consists of a text describing the problem (optionally in two -language variants -- Czech and English), an evaluation configuration -(machine-readable instructions on how to evaluate solutions to the exercise) and -a set of inputs and reference outputs. Exercises are created by instructed -privileged users. Assigning an exercise to a group means choosing one of the -available exercises and specifying additional properties: a deadline (optionally -a second deadline), a maximum amount of points, a configuration for calculating -the score, a maximum number of submissions, and a list of supported runtime -environments (e.g. programming languages) including specific time and memory -limits for each one. +Each exercise consists of a text describing the problem, an evaluation +configuration (machine-readable instructions on how to evaluate solutions to the +exercise), time and memory limits for all supported runtimes (e.g. programming +languages), a configuration for calculating the final score and a set of inputs +and reference outputs. Exercises are created by instructed privileged users. +Assigning an exercise to a group means choosing one of the available exercises +and specifying additional properties: a deadline (optionally a second deadline), +a maximum amount of points, a maximum number of submissions and a list of +supported runtime environments. Typical use cases for supported user roles are following: @@ -107,7 +106,7 @@ Typical use cases for supported user roles are following: evaluation process - view solution results -- which parts succeeded and failed, total number of acquired points, bonus points -- **supervisor** +- **supervisor** (similar to CodEx **operator**) - create exercise -- create description text and evaluation configuration (for each programming environment), upload testing inputs and outputs - assign exercise to group -- choose exercise and set deadlines, number of @@ -142,10 +141,12 @@ Incoming jobs are kept in a queue until a free worker picks them. Workers are capable of sequential evaluation of jobs, one at a time. The worker obtains the solution and its evaluation configuration, parses it and -starts executing the contained instructions. It is crucial to keep the worker -computer secure and stable, so a sandboxed environment is used for dealing with -unknown source code. When the execution is finished, results are saved and the -submitter is notified. +starts executing the contained instructions. Each job should have more testing +cases, which examine wrong inputs, corner values and data of different sizes to +guess the program complexity. It is crucial to keep the worker computer secure +and stable, so a sandboxed environment is used for dealing with unknown source +code. When the execution is finished, results are saved and the submitter is +notified. The output of the worker contains data about the evaluation, such as time and memory spent on running the program for each test input and whether its output @@ -172,10 +173,10 @@ several drawbacks. The main ones are: test multi-threaded applications as well. - **instances** -- Different ways of CodEx usage scenarios requires separate installations (Programming I and II, Java, C#, etc.). This configuration is - not user friendly (students have to register in each instance separately) and - burdens administrators with unnecessary work. CodEx architecture does not - allow sharing hardware between instances, which results in an inefficient use - of hardware for evaluation. + not user friendly (students have to register in each installation separately) + and burdens administrators with unnecessary work. CodEx architecture does not + allow sharing workers between installations, which results in an inefficient + use of hardware for evaluation. - **task extensibility** -- There is a need to test and evaluate complicated programs for classes such as Parallel programming or Compiler principles, which have a more difficult evaluation chain than simple @@ -196,17 +197,12 @@ In general, CodEx features should be preserved, so only differences are presented here. For clear arrangement all the requirements and wishes are presented grouped by categories. -### System Features - -System features represents directly accessible functionality to users of the -system. They describe the evaluation system in general and also university -addons (mostly administrative features). - -#### Requirements of The Users +### Requirements of The Users - _group hierarchy_ -- creating an arbitrarily nested tree structure should be supported to allow keeping related groups together, such as in the example - below. A group hierarchy also allows archiving data from past courses. + below. CodEx supported only a flat group structure. A group hierarchy also + allows archiving data from past courses. ``` Summer term 2016 @@ -218,17 +214,14 @@ addons (mostly administrative features). ... ``` -- _a database of exercises_ -- teachers should be able to create exercises - including textual description, sample inputs and correct reference outputs - (for example "sum all numbers from given file and write the result to the - standard output") and to browse this database +- _a database of exercises_ -- teachers should be able to filter viewed + exercises according to several criteria, for example supported runtime + environment or author +- _advanced exercises_ -- the system should support more advanced evaluation + pipeline than basic compilation/execution/evaluation which is in CodEx - _customizable grading system_ -- teachers need to specify the way of computation of the final score, which will be awarded to the submissions of the student depending on their quality -- _viewing student details_ -- teachers should be able to view the details of - their students (members of their groups), including all submitted solutions -- _awarding additional points_ -- adding (or subtracting) points from the final - score of a submission by a supervisor must be supported - _marking a solution as accepted_ -- the system should allow marking one particular solution as accepted (used for grading the assignment) by the supervisor @@ -239,13 +232,11 @@ addons (mostly administrative features). - _localization_ -- all texts (UI and exercises) should be translatable - _formatted exercise texts_ -- Markdown or another lightweight markup language should be supported for formatting exercise texts -- _exercise tags_ -- the system should support tagging exercises searching by - these tags - _comments_ -- adding both private and public comments to exercises, tests and solutions should be supported - _plagiarism detection_ -#### Administrative Requirements +### Administrative Requirements - _pluggable user interface_ -- the system should allow using an alternative user interface, such as a command line client; implementation of such clients @@ -258,10 +249,10 @@ addons (mostly administrative features). OAuth, should be supported - _querying SIS_ -- loading user data from the university information system should be supported -- _sandboxing_ -- there should be a safe environment in which the solutions of - the students are executed to prevent system failures due to malicious code - being submitted; the sandboxed environment should have the least possible - impact on measurement results (most importantly on measured times) +- _sandboxing_ -- there should be more advanced sandboxing which supports + execution of parallel programs and easy integration of different programming + environments and tools; the sandboxed environment should have the least + possible impact on measurement results (most importantly on measured times) - _heterogeneous worker pool_ -- there must be support for submission evaluation in multiple programming environments in a single installation to avoid unacceptable workload for the administrator (maintaining a separate @@ -276,14 +267,10 @@ addons (mostly administrative features). ### Non-functional Requirements -Non-functional requirements are requirements of technical character with no -direct mapping to visible parts of the system. In an ideal world, users should -not know about these features if they work properly, but would be at least -annoyed if they did not. - - _no installation_ -- the primary user interface of the system must be accessible on the computers of the users without the need to install any - additional software + additional software except for a web browser (which is mostly already + installed) - _performance_ -- the system must be ready for at least hundreds of students and tens of supervisors using it at once - _automated deployment_ -- all of the components of the system must be easy to