Intro update

master
Petr Stefan 8 years ago
parent 2f432ab3bc
commit 04515afbbc

@ -86,16 +86,15 @@ corresponds to his/her privileges. There are user groups reflecting the
structure of lectured courses. structure of lectured courses.
A database of exercises (algorithmic problems) is another part of the project. A database of exercises (algorithmic problems) is another part of the project.
Each exercise consists of a text describing the problem (optionally in two Each exercise consists of a text describing the problem, an evaluation
language variants -- Czech and English), an evaluation configuration configuration (machine-readable instructions on how to evaluate solutions to the
(machine-readable instructions on how to evaluate solutions to the exercise) and exercise), time and memory limits for all supported runtimes (e.g. programming
a set of inputs and reference outputs. Exercises are created by instructed languages), a configuration for calculating the final score and a set of inputs
privileged users. Assigning an exercise to a group means choosing one of the and reference outputs. Exercises are created by instructed privileged users.
available exercises and specifying additional properties: a deadline (optionally Assigning an exercise to a group means choosing one of the available exercises
a second deadline), a maximum amount of points, a configuration for calculating and specifying additional properties: a deadline (optionally a second deadline),
the score, a maximum number of submissions, and a list of supported runtime a maximum amount of points, a maximum number of submissions and a list of
environments (e.g. programming languages) including specific time and memory supported runtime environments.
limits for each one.
Typical use cases for supported user roles are following: Typical use cases for supported user roles are following:
@ -107,7 +106,7 @@ Typical use cases for supported user roles are following:
evaluation process evaluation process
- view solution results -- which parts succeeded and failed, total number of - view solution results -- which parts succeeded and failed, total number of
acquired points, bonus points acquired points, bonus points
- **supervisor** - **supervisor** (similar to CodEx **operator**)
- create exercise -- create description text and evaluation configuration - create exercise -- create description text and evaluation configuration
(for each programming environment), upload testing inputs and outputs (for each programming environment), upload testing inputs and outputs
- assign exercise to group -- choose exercise and set deadlines, number of - assign exercise to group -- choose exercise and set deadlines, number of
@ -142,10 +141,12 @@ Incoming jobs are kept in a queue until a free worker picks them. Workers are
capable of sequential evaluation of jobs, one at a time. capable of sequential evaluation of jobs, one at a time.
The worker obtains the solution and its evaluation configuration, parses it and The worker obtains the solution and its evaluation configuration, parses it and
starts executing the contained instructions. It is crucial to keep the worker starts executing the contained instructions. Each job should have more testing
computer secure and stable, so a sandboxed environment is used for dealing with cases, which examine wrong inputs, corner values and data of different sizes to
unknown source code. When the execution is finished, results are saved and the guess the program complexity. It is crucial to keep the worker computer secure
submitter is notified. and stable, so a sandboxed environment is used for dealing with unknown source
code. When the execution is finished, results are saved and the submitter is
notified.
The output of the worker contains data about the evaluation, such as time and The output of the worker contains data about the evaluation, such as time and
memory spent on running the program for each test input and whether its output memory spent on running the program for each test input and whether its output
@ -172,10 +173,10 @@ several drawbacks. The main ones are:
test multi-threaded applications as well. test multi-threaded applications as well.
- **instances** -- Different ways of CodEx usage scenarios requires separate - **instances** -- Different ways of CodEx usage scenarios requires separate
installations (Programming I and II, Java, C#, etc.). This configuration is installations (Programming I and II, Java, C#, etc.). This configuration is
not user friendly (students have to register in each instance separately) and not user friendly (students have to register in each installation separately)
burdens administrators with unnecessary work. CodEx architecture does not and burdens administrators with unnecessary work. CodEx architecture does not
allow sharing hardware between instances, which results in an inefficient use allow sharing workers between installations, which results in an inefficient
of hardware for evaluation. use of hardware for evaluation.
- **task extensibility** -- There is a need to test and evaluate complicated - **task extensibility** -- There is a need to test and evaluate complicated
programs for classes such as Parallel programming or Compiler principles, programs for classes such as Parallel programming or Compiler principles,
which have a more difficult evaluation chain than simple which have a more difficult evaluation chain than simple
@ -196,17 +197,12 @@ In general, CodEx features should be preserved, so only differences are
presented here. For clear arrangement all the requirements and wishes are presented here. For clear arrangement all the requirements and wishes are
presented grouped by categories. presented grouped by categories.
### System Features ### Requirements of The Users
System features represents directly accessible functionality to users of the
system. They describe the evaluation system in general and also university
addons (mostly administrative features).
#### Requirements of The Users
- _group hierarchy_ -- creating an arbitrarily nested tree structure should be - _group hierarchy_ -- creating an arbitrarily nested tree structure should be
supported to allow keeping related groups together, such as in the example supported to allow keeping related groups together, such as in the example
below. A group hierarchy also allows archiving data from past courses. below. CodEx supported only a flat group structure. A group hierarchy also
allows archiving data from past courses.
``` ```
Summer term 2016 Summer term 2016
@ -218,17 +214,14 @@ addons (mostly administrative features).
... ...
``` ```
- _a database of exercises_ -- teachers should be able to create exercises - _a database of exercises_ -- teachers should be able to filter viewed
including textual description, sample inputs and correct reference outputs exercises according to several criteria, for example supported runtime
(for example "sum all numbers from given file and write the result to the environment or author
standard output") and to browse this database - _advanced exercises_ -- the system should support more advanced evaluation
pipeline than basic compilation/execution/evaluation which is in CodEx
- _customizable grading system_ -- teachers need to specify the way of - _customizable grading system_ -- teachers need to specify the way of
computation of the final score, which will be awarded to the submissions of computation of the final score, which will be awarded to the submissions of
the student depending on their quality the student depending on their quality
- _viewing student details_ -- teachers should be able to view the details of
their students (members of their groups), including all submitted solutions
- _awarding additional points_ -- adding (or subtracting) points from the final
score of a submission by a supervisor must be supported
- _marking a solution as accepted_ -- the system should allow marking one - _marking a solution as accepted_ -- the system should allow marking one
particular solution as accepted (used for grading the assignment) by the particular solution as accepted (used for grading the assignment) by the
supervisor supervisor
@ -239,13 +232,11 @@ addons (mostly administrative features).
- _localization_ -- all texts (UI and exercises) should be translatable - _localization_ -- all texts (UI and exercises) should be translatable
- _formatted exercise texts_ -- Markdown or another lightweight markup language - _formatted exercise texts_ -- Markdown or another lightweight markup language
should be supported for formatting exercise texts should be supported for formatting exercise texts
- _exercise tags_ -- the system should support tagging exercises searching by
these tags
- _comments_ -- adding both private and public comments to exercises, tests and - _comments_ -- adding both private and public comments to exercises, tests and
solutions should be supported solutions should be supported
- _plagiarism detection_ - _plagiarism detection_
#### Administrative Requirements ### Administrative Requirements
- _pluggable user interface_ -- the system should allow using an alternative - _pluggable user interface_ -- the system should allow using an alternative
user interface, such as a command line client; implementation of such clients user interface, such as a command line client; implementation of such clients
@ -258,10 +249,10 @@ addons (mostly administrative features).
OAuth, should be supported OAuth, should be supported
- _querying SIS_ -- loading user data from the university information system - _querying SIS_ -- loading user data from the university information system
should be supported should be supported
- _sandboxing_ -- there should be a safe environment in which the solutions of - _sandboxing_ -- there should be more advanced sandboxing which supports
the students are executed to prevent system failures due to malicious code execution of parallel programs and easy integration of different programming
being submitted; the sandboxed environment should have the least possible environments and tools; the sandboxed environment should have the least
impact on measurement results (most importantly on measured times) possible impact on measurement results (most importantly on measured times)
- _heterogeneous worker pool_ -- there must be support for submission evaluation - _heterogeneous worker pool_ -- there must be support for submission evaluation
in multiple programming environments in a single installation to avoid in multiple programming environments in a single installation to avoid
unacceptable workload for the administrator (maintaining a separate unacceptable workload for the administrator (maintaining a separate
@ -276,14 +267,10 @@ addons (mostly administrative features).
### Non-functional Requirements ### Non-functional Requirements
Non-functional requirements are requirements of technical character with no
direct mapping to visible parts of the system. In an ideal world, users should
not know about these features if they work properly, but would be at least
annoyed if they did not.
- _no installation_ -- the primary user interface of the system must be - _no installation_ -- the primary user interface of the system must be
accessible on the computers of the users without the need to install any accessible on the computers of the users without the need to install any
additional software additional software except for a web browser (which is mostly already
installed)
- _performance_ -- the system must be ready for at least hundreds of students - _performance_ -- the system must be ready for at least hundreds of students
and tens of supervisors using it at once and tens of supervisors using it at once
- _automated deployment_ -- all of the components of the system must be easy to - _automated deployment_ -- all of the components of the system must be easy to

Loading…
Cancel
Save