|
|
|
@ -86,16 +86,15 @@ corresponds to his/her privileges. There are user groups reflecting the
|
|
|
|
|
structure of lectured courses.
|
|
|
|
|
|
|
|
|
|
A database of exercises (algorithmic problems) is another part of the project.
|
|
|
|
|
Each exercise consists of a text describing the problem (optionally in two
|
|
|
|
|
language variants -- Czech and English), an evaluation configuration
|
|
|
|
|
(machine-readable instructions on how to evaluate solutions to the exercise) and
|
|
|
|
|
a set of inputs and reference outputs. Exercises are created by instructed
|
|
|
|
|
privileged users. Assigning an exercise to a group means choosing one of the
|
|
|
|
|
available exercises and specifying additional properties: a deadline (optionally
|
|
|
|
|
a second deadline), a maximum amount of points, a configuration for calculating
|
|
|
|
|
the score, a maximum number of submissions, and a list of supported runtime
|
|
|
|
|
environments (e.g. programming languages) including specific time and memory
|
|
|
|
|
limits for each one.
|
|
|
|
|
Each exercise consists of a text describing the problem, an evaluation
|
|
|
|
|
configuration (machine-readable instructions on how to evaluate solutions to the
|
|
|
|
|
exercise), time and memory limits for all supported runtimes (e.g. programming
|
|
|
|
|
languages), a configuration for calculating the final score and a set of inputs
|
|
|
|
|
and reference outputs. Exercises are created by instructed privileged users.
|
|
|
|
|
Assigning an exercise to a group means choosing one of the available exercises
|
|
|
|
|
and specifying additional properties: a deadline (optionally a second deadline),
|
|
|
|
|
a maximum amount of points, a maximum number of submissions and a list of
|
|
|
|
|
supported runtime environments.
|
|
|
|
|
|
|
|
|
|
Typical use cases for supported user roles are following:
|
|
|
|
|
|
|
|
|
@ -107,7 +106,7 @@ Typical use cases for supported user roles are following:
|
|
|
|
|
evaluation process
|
|
|
|
|
- view solution results -- which parts succeeded and failed, total number of
|
|
|
|
|
acquired points, bonus points
|
|
|
|
|
- **supervisor**
|
|
|
|
|
- **supervisor** (similar to CodEx **operator**)
|
|
|
|
|
- create exercise -- create description text and evaluation configuration
|
|
|
|
|
(for each programming environment), upload testing inputs and outputs
|
|
|
|
|
- assign exercise to group -- choose exercise and set deadlines, number of
|
|
|
|
@ -142,10 +141,12 @@ Incoming jobs are kept in a queue until a free worker picks them. Workers are
|
|
|
|
|
capable of sequential evaluation of jobs, one at a time.
|
|
|
|
|
|
|
|
|
|
The worker obtains the solution and its evaluation configuration, parses it and
|
|
|
|
|
starts executing the contained instructions. It is crucial to keep the worker
|
|
|
|
|
computer secure and stable, so a sandboxed environment is used for dealing with
|
|
|
|
|
unknown source code. When the execution is finished, results are saved and the
|
|
|
|
|
submitter is notified.
|
|
|
|
|
starts executing the contained instructions. Each job should have more testing
|
|
|
|
|
cases, which examine wrong inputs, corner values and data of different sizes to
|
|
|
|
|
guess the program complexity. It is crucial to keep the worker computer secure
|
|
|
|
|
and stable, so a sandboxed environment is used for dealing with unknown source
|
|
|
|
|
code. When the execution is finished, results are saved and the submitter is
|
|
|
|
|
notified.
|
|
|
|
|
|
|
|
|
|
The output of the worker contains data about the evaluation, such as time and
|
|
|
|
|
memory spent on running the program for each test input and whether its output
|
|
|
|
@ -172,10 +173,10 @@ several drawbacks. The main ones are:
|
|
|
|
|
test multi-threaded applications as well.
|
|
|
|
|
- **instances** -- Different ways of CodEx usage scenarios requires separate
|
|
|
|
|
installations (Programming I and II, Java, C#, etc.). This configuration is
|
|
|
|
|
not user friendly (students have to register in each instance separately) and
|
|
|
|
|
burdens administrators with unnecessary work. CodEx architecture does not
|
|
|
|
|
allow sharing hardware between instances, which results in an inefficient use
|
|
|
|
|
of hardware for evaluation.
|
|
|
|
|
not user friendly (students have to register in each installation separately)
|
|
|
|
|
and burdens administrators with unnecessary work. CodEx architecture does not
|
|
|
|
|
allow sharing workers between installations, which results in an inefficient
|
|
|
|
|
use of hardware for evaluation.
|
|
|
|
|
- **task extensibility** -- There is a need to test and evaluate complicated
|
|
|
|
|
programs for classes such as Parallel programming or Compiler principles,
|
|
|
|
|
which have a more difficult evaluation chain than simple
|
|
|
|
@ -196,17 +197,12 @@ In general, CodEx features should be preserved, so only differences are
|
|
|
|
|
presented here. For clear arrangement all the requirements and wishes are
|
|
|
|
|
presented grouped by categories.
|
|
|
|
|
|
|
|
|
|
### System Features
|
|
|
|
|
|
|
|
|
|
System features represents directly accessible functionality to users of the
|
|
|
|
|
system. They describe the evaluation system in general and also university
|
|
|
|
|
addons (mostly administrative features).
|
|
|
|
|
|
|
|
|
|
#### Requirements of The Users
|
|
|
|
|
### Requirements of The Users
|
|
|
|
|
|
|
|
|
|
- _group hierarchy_ -- creating an arbitrarily nested tree structure should be
|
|
|
|
|
supported to allow keeping related groups together, such as in the example
|
|
|
|
|
below. A group hierarchy also allows archiving data from past courses.
|
|
|
|
|
below. CodEx supported only a flat group structure. A group hierarchy also
|
|
|
|
|
allows archiving data from past courses.
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
Summer term 2016
|
|
|
|
@ -218,17 +214,14 @@ addons (mostly administrative features).
|
|
|
|
|
...
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- _a database of exercises_ -- teachers should be able to create exercises
|
|
|
|
|
including textual description, sample inputs and correct reference outputs
|
|
|
|
|
(for example "sum all numbers from given file and write the result to the
|
|
|
|
|
standard output") and to browse this database
|
|
|
|
|
- _a database of exercises_ -- teachers should be able to filter viewed
|
|
|
|
|
exercises according to several criteria, for example supported runtime
|
|
|
|
|
environment or author
|
|
|
|
|
- _advanced exercises_ -- the system should support more advanced evaluation
|
|
|
|
|
pipeline than basic compilation/execution/evaluation which is in CodEx
|
|
|
|
|
- _customizable grading system_ -- teachers need to specify the way of
|
|
|
|
|
computation of the final score, which will be awarded to the submissions of
|
|
|
|
|
the student depending on their quality
|
|
|
|
|
- _viewing student details_ -- teachers should be able to view the details of
|
|
|
|
|
their students (members of their groups), including all submitted solutions
|
|
|
|
|
- _awarding additional points_ -- adding (or subtracting) points from the final
|
|
|
|
|
score of a submission by a supervisor must be supported
|
|
|
|
|
- _marking a solution as accepted_ -- the system should allow marking one
|
|
|
|
|
particular solution as accepted (used for grading the assignment) by the
|
|
|
|
|
supervisor
|
|
|
|
@ -239,13 +232,11 @@ addons (mostly administrative features).
|
|
|
|
|
- _localization_ -- all texts (UI and exercises) should be translatable
|
|
|
|
|
- _formatted exercise texts_ -- Markdown or another lightweight markup language
|
|
|
|
|
should be supported for formatting exercise texts
|
|
|
|
|
- _exercise tags_ -- the system should support tagging exercises searching by
|
|
|
|
|
these tags
|
|
|
|
|
- _comments_ -- adding both private and public comments to exercises, tests and
|
|
|
|
|
solutions should be supported
|
|
|
|
|
- _plagiarism detection_
|
|
|
|
|
|
|
|
|
|
#### Administrative Requirements
|
|
|
|
|
### Administrative Requirements
|
|
|
|
|
|
|
|
|
|
- _pluggable user interface_ -- the system should allow using an alternative
|
|
|
|
|
user interface, such as a command line client; implementation of such clients
|
|
|
|
@ -258,10 +249,10 @@ addons (mostly administrative features).
|
|
|
|
|
OAuth, should be supported
|
|
|
|
|
- _querying SIS_ -- loading user data from the university information system
|
|
|
|
|
should be supported
|
|
|
|
|
- _sandboxing_ -- there should be a safe environment in which the solutions of
|
|
|
|
|
the students are executed to prevent system failures due to malicious code
|
|
|
|
|
being submitted; the sandboxed environment should have the least possible
|
|
|
|
|
impact on measurement results (most importantly on measured times)
|
|
|
|
|
- _sandboxing_ -- there should be more advanced sandboxing which supports
|
|
|
|
|
execution of parallel programs and easy integration of different programming
|
|
|
|
|
environments and tools; the sandboxed environment should have the least
|
|
|
|
|
possible impact on measurement results (most importantly on measured times)
|
|
|
|
|
- _heterogeneous worker pool_ -- there must be support for submission evaluation
|
|
|
|
|
in multiple programming environments in a single installation to avoid
|
|
|
|
|
unacceptable workload for the administrator (maintaining a separate
|
|
|
|
@ -276,14 +267,10 @@ addons (mostly administrative features).
|
|
|
|
|
|
|
|
|
|
### Non-functional Requirements
|
|
|
|
|
|
|
|
|
|
Non-functional requirements are requirements of technical character with no
|
|
|
|
|
direct mapping to visible parts of the system. In an ideal world, users should
|
|
|
|
|
not know about these features if they work properly, but would be at least
|
|
|
|
|
annoyed if they did not.
|
|
|
|
|
|
|
|
|
|
- _no installation_ -- the primary user interface of the system must be
|
|
|
|
|
accessible on the computers of the users without the need to install any
|
|
|
|
|
additional software
|
|
|
|
|
additional software except for a web browser (which is mostly already
|
|
|
|
|
installed)
|
|
|
|
|
- _performance_ -- the system must be ready for at least hundreds of students
|
|
|
|
|
and tens of supervisors using it at once
|
|
|
|
|
- _automated deployment_ -- all of the components of the system must be easy to
|
|
|
|
|