Intro update

master
Petr Stefan 8 years ago
parent 2f432ab3bc
commit 04515afbbc

@ -86,16 +86,15 @@ corresponds to his/her privileges. There are user groups reflecting the
structure of lectured courses.
A database of exercises (algorithmic problems) is another part of the project.
Each exercise consists of a text describing the problem (optionally in two
language variants -- Czech and English), an evaluation configuration
(machine-readable instructions on how to evaluate solutions to the exercise) and
a set of inputs and reference outputs. Exercises are created by instructed
privileged users. Assigning an exercise to a group means choosing one of the
available exercises and specifying additional properties: a deadline (optionally
a second deadline), a maximum amount of points, a configuration for calculating
the score, a maximum number of submissions, and a list of supported runtime
environments (e.g. programming languages) including specific time and memory
limits for each one.
Each exercise consists of a text describing the problem, an evaluation
configuration (machine-readable instructions on how to evaluate solutions to the
exercise), time and memory limits for all supported runtimes (e.g. programming
languages), a configuration for calculating the final score and a set of inputs
and reference outputs. Exercises are created by instructed privileged users.
Assigning an exercise to a group means choosing one of the available exercises
and specifying additional properties: a deadline (optionally a second deadline),
a maximum amount of points, a maximum number of submissions and a list of
supported runtime environments.
Typical use cases for supported user roles are following:
@ -107,7 +106,7 @@ Typical use cases for supported user roles are following:
evaluation process
- view solution results -- which parts succeeded and failed, total number of
acquired points, bonus points
- **supervisor**
- **supervisor** (similar to CodEx **operator**)
- create exercise -- create description text and evaluation configuration
(for each programming environment), upload testing inputs and outputs
- assign exercise to group -- choose exercise and set deadlines, number of
@ -142,10 +141,12 @@ Incoming jobs are kept in a queue until a free worker picks them. Workers are
capable of sequential evaluation of jobs, one at a time.
The worker obtains the solution and its evaluation configuration, parses it and
starts executing the contained instructions. It is crucial to keep the worker
computer secure and stable, so a sandboxed environment is used for dealing with
unknown source code. When the execution is finished, results are saved and the
submitter is notified.
starts executing the contained instructions. Each job should have more testing
cases, which examine wrong inputs, corner values and data of different sizes to
guess the program complexity. It is crucial to keep the worker computer secure
and stable, so a sandboxed environment is used for dealing with unknown source
code. When the execution is finished, results are saved and the submitter is
notified.
The output of the worker contains data about the evaluation, such as time and
memory spent on running the program for each test input and whether its output
@ -172,10 +173,10 @@ several drawbacks. The main ones are:
test multi-threaded applications as well.
- **instances** -- Different ways of CodEx usage scenarios requires separate
installations (Programming I and II, Java, C#, etc.). This configuration is
not user friendly (students have to register in each instance separately) and
burdens administrators with unnecessary work. CodEx architecture does not
allow sharing hardware between instances, which results in an inefficient use
of hardware for evaluation.
not user friendly (students have to register in each installation separately)
and burdens administrators with unnecessary work. CodEx architecture does not
allow sharing workers between installations, which results in an inefficient
use of hardware for evaluation.
- **task extensibility** -- There is a need to test and evaluate complicated
programs for classes such as Parallel programming or Compiler principles,
which have a more difficult evaluation chain than simple
@ -196,17 +197,12 @@ In general, CodEx features should be preserved, so only differences are
presented here. For clear arrangement all the requirements and wishes are
presented grouped by categories.
### System Features
System features represents directly accessible functionality to users of the
system. They describe the evaluation system in general and also university
addons (mostly administrative features).
#### Requirements of The Users
### Requirements of The Users
- _group hierarchy_ -- creating an arbitrarily nested tree structure should be
supported to allow keeping related groups together, such as in the example
below. A group hierarchy also allows archiving data from past courses.
below. CodEx supported only a flat group structure. A group hierarchy also
allows archiving data from past courses.
```
Summer term 2016
@ -218,17 +214,14 @@ addons (mostly administrative features).
...
```
- _a database of exercises_ -- teachers should be able to create exercises
including textual description, sample inputs and correct reference outputs
(for example "sum all numbers from given file and write the result to the
standard output") and to browse this database
- _a database of exercises_ -- teachers should be able to filter viewed
exercises according to several criteria, for example supported runtime
environment or author
- _advanced exercises_ -- the system should support more advanced evaluation
pipeline than basic compilation/execution/evaluation which is in CodEx
- _customizable grading system_ -- teachers need to specify the way of
computation of the final score, which will be awarded to the submissions of
the student depending on their quality
- _viewing student details_ -- teachers should be able to view the details of
their students (members of their groups), including all submitted solutions
- _awarding additional points_ -- adding (or subtracting) points from the final
score of a submission by a supervisor must be supported
- _marking a solution as accepted_ -- the system should allow marking one
particular solution as accepted (used for grading the assignment) by the
supervisor
@ -239,13 +232,11 @@ addons (mostly administrative features).
- _localization_ -- all texts (UI and exercises) should be translatable
- _formatted exercise texts_ -- Markdown or another lightweight markup language
should be supported for formatting exercise texts
- _exercise tags_ -- the system should support tagging exercises searching by
these tags
- _comments_ -- adding both private and public comments to exercises, tests and
solutions should be supported
- _plagiarism detection_
#### Administrative Requirements
### Administrative Requirements
- _pluggable user interface_ -- the system should allow using an alternative
user interface, such as a command line client; implementation of such clients
@ -258,10 +249,10 @@ addons (mostly administrative features).
OAuth, should be supported
- _querying SIS_ -- loading user data from the university information system
should be supported
- _sandboxing_ -- there should be a safe environment in which the solutions of
the students are executed to prevent system failures due to malicious code
being submitted; the sandboxed environment should have the least possible
impact on measurement results (most importantly on measured times)
- _sandboxing_ -- there should be more advanced sandboxing which supports
execution of parallel programs and easy integration of different programming
environments and tools; the sandboxed environment should have the least
possible impact on measurement results (most importantly on measured times)
- _heterogeneous worker pool_ -- there must be support for submission evaluation
in multiple programming environments in a single installation to avoid
unacceptable workload for the administrator (maintaining a separate
@ -276,14 +267,10 @@ addons (mostly administrative features).
### Non-functional Requirements
Non-functional requirements are requirements of technical character with no
direct mapping to visible parts of the system. In an ideal world, users should
not know about these features if they work properly, but would be at least
annoyed if they did not.
- _no installation_ -- the primary user interface of the system must be
accessible on the computers of the users without the need to install any
additional software
additional software except for a web browser (which is mostly already
installed)
- _performance_ -- the system must be ready for at least hundreds of students
and tens of supervisors using it at once
- _automated deployment_ -- all of the components of the system must be easy to

Loading…
Cancel
Save