|
|
|
|
@ -115,7 +115,21 @@ currently used at the university (CodEx), so its flaws and weaknesses can be
|
|
|
|
|
addressed. Furthermore, many teachers desire to use and test the new system and
|
|
|
|
|
they are willing to consult ideas or problems during development with us.
|
|
|
|
|
|
|
|
|
|
### Intended usage
|
|
|
|
|
## Current system
|
|
|
|
|
|
|
|
|
|
Currently used grading solution at the Faculty of Mathematics and Physics of
|
|
|
|
|
the Charles University in Prague which was implemented in 2006 by a group
|
|
|
|
|
of students. It is called [CodEx -- The Code Examiner](http://codex.ms.mff.cuni.cz/project/)
|
|
|
|
|
and it has been used with some improvements since then. The original plan was
|
|
|
|
|
to use the system only for basic programming courses, but there was a demand
|
|
|
|
|
for adapting it for many different subjects.
|
|
|
|
|
|
|
|
|
|
CodEx is based on dynamic analysis. It features a web-based interface, where
|
|
|
|
|
supervisors can assign exercises to their students and the students have a time
|
|
|
|
|
window to submit their solutions. Each solution is compiled and run in sandbox
|
|
|
|
|
(MO-Eval). The metrics which are checked are: correctness of the output, time
|
|
|
|
|
and memory limits. It supports programs written in C, C++, C#, Java, Pascal,
|
|
|
|
|
Python and Haskell.
|
|
|
|
|
|
|
|
|
|
The whole system is intended to help both teachers (supervisors) and students.
|
|
|
|
|
To achieve this, it is crucial to keep in mind the typical usage scenarios of
|
|
|
|
|
@ -123,26 +137,7 @@ the system and to try to make these tasks as simple as possible.
|
|
|
|
|
|
|
|
|
|
The system has a database of users. Each user is assigned a role, which
|
|
|
|
|
corresponds to his/her privileges. There are user groups reflecting the
|
|
|
|
|
structure of lectured courses. Groups can be hierarchically ordered to reflect
|
|
|
|
|
additional metadata such as the academic year. For example, a reasonable group
|
|
|
|
|
hierarchy could look like this:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
Summer term 2016
|
|
|
|
|
|-- Language C# and .NET platform
|
|
|
|
|
| |-- Labs Monday 10:30
|
|
|
|
|
| `-- Labs Thursday 9:00
|
|
|
|
|
|-- Programming I
|
|
|
|
|
| |-- Labs Monday 14:00
|
|
|
|
|
...
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
In this example, students are members of the leaf groups and the higher level
|
|
|
|
|
nodes are just for keeping related groups together. The structure can be
|
|
|
|
|
modified and altered to fit specific needs of the university or any other
|
|
|
|
|
organization, even a flat structure is possible. One user can be a member of
|
|
|
|
|
multiple groups and have a different role in each of them (a student can attend
|
|
|
|
|
labs for several courses while also teaching one).
|
|
|
|
|
structure of lectured courses.
|
|
|
|
|
|
|
|
|
|
A database of exercises (algorithmic problems) is another part of the project.
|
|
|
|
|
Each exercise consists of a text describing the problem in multiple language
|
|
|
|
|
@ -155,12 +150,50 @@ points, a configuration for calculating the score, a maximum number of
|
|
|
|
|
submissions, and a list of supported runtime environments (e.g. programming
|
|
|
|
|
languages) including specific time and memory limits for each one.
|
|
|
|
|
|
|
|
|
|
Typical use cases for supported user roles are illustrated on following UML
|
|
|
|
|
diagram:
|
|
|
|
|
Typical use cases for supported user roles are following:
|
|
|
|
|
|
|
|
|
|
- **student**
|
|
|
|
|
- join a group
|
|
|
|
|
- get assignments in group
|
|
|
|
|
- submit solution to assignment
|
|
|
|
|
- view solution results
|
|
|
|
|
- **supervisor**
|
|
|
|
|
- create exercise
|
|
|
|
|
- assign exercise to group, modify assignment
|
|
|
|
|
- view all results in group
|
|
|
|
|
- alter automatic solution grading
|
|
|
|
|
- **administrator**
|
|
|
|
|
- create groups
|
|
|
|
|
- alter user privileges
|
|
|
|
|
- check system logs, upgrades and other management
|
|
|
|
|
|
|
|
|
|
Current system is old, but robust. There were no major security incidents
|
|
|
|
|
during its production usage. However, from today's perspective there are
|
|
|
|
|
several drawbacks. The main ones are:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- **web interface** -- The web interface is simple and fully functional. But
|
|
|
|
|
rapid development in web technologies opens new horizons of how web interface
|
|
|
|
|
can be made.
|
|
|
|
|
- **web API** -- CodEx offers a very limited XML API based on outdated
|
|
|
|
|
technologies that is not sufficient for users who would like to create custom
|
|
|
|
|
interfaces such as a command line tool or mobile application.
|
|
|
|
|
- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system
|
|
|
|
|
calls and blocking the bad ones. This can be easily done for single-threaded
|
|
|
|
|
applications, but proves difficult with multi-threaded ones. In present day,
|
|
|
|
|
parallelism is a very important area of computing, so there is requirement to
|
|
|
|
|
test multi-threaded applications too.
|
|
|
|
|
- **instances** -- Different ways of CodEx usage scenarios requires separate
|
|
|
|
|
instances (Programming I and II, Java, C#, etc.). This configuration is not
|
|
|
|
|
user friendly (students have to register in each instance separately) and
|
|
|
|
|
burdens administrators with unnecessary work. CodEx architecture does not
|
|
|
|
|
allow sharing hardware between instances, which results in an inefficient use
|
|
|
|
|
of hardware for evaluation.
|
|
|
|
|
- **task extensibility** -- There is a need to test and evaluate complicated
|
|
|
|
|
programs for classes such as Parallel programming or Compiler principles,
|
|
|
|
|
which have a more difficult evaluation chain than simple
|
|
|
|
|
compilation/execution/evaluation provided by CodEx.
|
|
|
|
|
|
|
|
|
|
#### Exercise evaluation chain
|
|
|
|
|
### Exercise evaluation chain
|
|
|
|
|
|
|
|
|
|
The most important part of the system is evaluation of solutions submitted by
|
|
|
|
|
students. Concepts of consecutive steps from source code to final results
|
|
|
|
|
@ -203,8 +236,9 @@ from administrators and supervisors. The ideas were gathered mostly from our
|
|
|
|
|
personal experience with the system and from meetings with faculty staff
|
|
|
|
|
involved with the current system.
|
|
|
|
|
|
|
|
|
|
For clear arrangement all the requirements and wishes are presented grouped by
|
|
|
|
|
categories.
|
|
|
|
|
In general, CodEx features should be preserved, so only differences are
|
|
|
|
|
presented here. For clear arrangement all the requirements and wishes are
|
|
|
|
|
presented grouped by categories.
|
|
|
|
|
|
|
|
|
|
### System features
|
|
|
|
|
|
|
|
|
|
@ -214,16 +248,11 @@ addons (mostly administrative features).
|
|
|
|
|
|
|
|
|
|
#### End user requirements
|
|
|
|
|
|
|
|
|
|
- users have their own accounts in the system
|
|
|
|
|
- system users can be members of multiple groups (reflecting courses or labs)
|
|
|
|
|
- group hierarchy -- @todo: copy text from
|
|
|
|
|
[here](https://github.com/ReCodEx/wiki/wiki/Rewritten-docs/87e4bcd39a4fca3eadbb4748e9a3b6ced2bd7150#intended-usage)
|
|
|
|
|
- there is a database of exercises; teachers can create exercises including
|
|
|
|
|
textual description, sample inputs and correct reference outputs (for example
|
|
|
|
|
"sum all numbers from given file and write the result to the standard output")
|
|
|
|
|
- there is a list of assigned exercises in each group and an interface to submit
|
|
|
|
|
a solution; teachers can assign an existing exercise to their class with some
|
|
|
|
|
specific properties set (deadlines, etc.)
|
|
|
|
|
- users can see a list of submitted solutions for each assignment with
|
|
|
|
|
corresponding results
|
|
|
|
|
- teachers can specify way of computation grading points which will be awarded
|
|
|
|
|
to the students depending on the quality of his/her solution for each
|
|
|
|
|
assignment extra
|
|
|
|
|
@ -237,7 +266,7 @@ addons (mostly administrative features).
|
|
|
|
|
- localization of all texts (UI and exercises)
|
|
|
|
|
- Markdown support for creating exercise texts
|
|
|
|
|
- tagging exercises in database and search by these tags
|
|
|
|
|
- comments, comments, comments (exercises, tests, solutions, ...)
|
|
|
|
|
- comments of exercises, tests and solutions
|
|
|
|
|
- plagiarism detection
|
|
|
|
|
|
|
|
|
|
#### Administrative requirements
|
|
|
|
|
@ -263,12 +292,12 @@ addons (mostly administrative features).
|
|
|
|
|
another tool and perform additional tests
|
|
|
|
|
- use of modern technologies with state-of-the-art compilers
|
|
|
|
|
|
|
|
|
|
### Technical details
|
|
|
|
|
### Nonfunctional requirements
|
|
|
|
|
|
|
|
|
|
Technical details are requirements of technical character with no direct mapping
|
|
|
|
|
to visible parts of system. In ideal word, users should not know about these if
|
|
|
|
|
they work properly, but would be at least annoyed if these requirements were not
|
|
|
|
|
met. Most notably they are these ones:
|
|
|
|
|
Nonfunctional requirements are requirements of technical character with no
|
|
|
|
|
direct mapping to visible parts of system. In ideal word, users should not know
|
|
|
|
|
about these if they work properly, but would be at least annoyed if these
|
|
|
|
|
requirements were not met. Most notably they are these ones:
|
|
|
|
|
|
|
|
|
|
- user interface of the system accessible on users' computers without
|
|
|
|
|
installation of any kind of additional software
|
|
|
|
|
@ -300,48 +329,6 @@ This is not a complete list of available evaluators, but only a few projects
|
|
|
|
|
which are used these days and can be an inspiration for our project. Each
|
|
|
|
|
project from the list has a brief description and some key features mentioned.
|
|
|
|
|
|
|
|
|
|
### CodEx
|
|
|
|
|
|
|
|
|
|
Currently used grading solution at the Faculty of Mathematics and Physics of
|
|
|
|
|
the Charles University in Prague which was implemented in 2006 by a group
|
|
|
|
|
of students. It is called [CodEx -- The Code Examiner](http://codex.ms.mff.cuni.cz/project/)
|
|
|
|
|
and it has been used with some improvements since then. The original plan was
|
|
|
|
|
to use the system only for basic programming courses, but there was a demand
|
|
|
|
|
for adapting it for many different subjects.
|
|
|
|
|
|
|
|
|
|
CodEx is based on dynamic analysis. It features a web-based interface, where
|
|
|
|
|
supervisors can assign exercises to their students and the students have a time
|
|
|
|
|
window to submit their solutions. Each solution is compiled and run in sandbox
|
|
|
|
|
(MO-Eval). The metrics which are checked are: correctness of the output, time
|
|
|
|
|
and memory limits. It supports programs written in C, C++, C#, Java, Pascal,
|
|
|
|
|
Python and Haskell.
|
|
|
|
|
|
|
|
|
|
Current system is old, but robust. There were no major security incidents
|
|
|
|
|
during its production usage. However, from today's perspective there are
|
|
|
|
|
several drawbacks. The main ones are:
|
|
|
|
|
|
|
|
|
|
- **web interface** -- The web interface is simple and fully functional. But
|
|
|
|
|
rapid development in web technologies opens new horizons of how web interface
|
|
|
|
|
can be made.
|
|
|
|
|
- **web API** -- CodEx offers a very limited XML API based on outdated
|
|
|
|
|
technologies that is not sufficient for users who would like to create custom
|
|
|
|
|
interfaces such as a command line tool or mobile application.
|
|
|
|
|
- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system
|
|
|
|
|
calls and blocking the bad ones. This can be easily done for single-threaded
|
|
|
|
|
applications, but proves difficult with multi-threaded ones. In present day,
|
|
|
|
|
parallelism is a very important area of computing, so there is requirement to
|
|
|
|
|
test multi-threaded applications too.
|
|
|
|
|
- **instances** -- Different ways of CodEx usage scenarios requires separate
|
|
|
|
|
instances (Programming I and II, Java, C#, etc.). This configuration is not
|
|
|
|
|
user friendly (students have to register in each instance separately) and
|
|
|
|
|
burdens administrators with unnecessary work. CodEx architecture does not
|
|
|
|
|
allow sharing hardware between instances, which results in an inefficient use
|
|
|
|
|
of hardware for evaluation.
|
|
|
|
|
- **task extensibility** -- There is a need to test and evaluate complicated
|
|
|
|
|
programs for classes such as Parallel programming or Compiler principles,
|
|
|
|
|
which have a more difficult evaluation chain than simple
|
|
|
|
|
compilation/execution/evaluation provided by CodEx.
|
|
|
|
|
|
|
|
|
|
### Progtest
|
|
|
|
|
|
|
|
|
|
[Progtest](https://progtest.fit.cvut.cz/) is private project of [FIT
|
|
|
|
|
|
Petr Stefan
8 years ago