|
|
|
@ -246,29 +246,30 @@ presented in groups by the user categories.
|
|
|
|
|
|
|
|
|
|
### Administrative Requirements
|
|
|
|
|
|
|
|
|
|
- _pluggable user interface_ -- the system should allow using an alternative
|
|
|
|
|
- _independent user interface_ -- the system should allow the use of an alternative
|
|
|
|
|
user interface, such as a command line client; implementation of such clients
|
|
|
|
|
should be as straightforward as possible
|
|
|
|
|
- _privilege separation_ -- there should be at least two roles -- _student_ and
|
|
|
|
|
_supervisor_. Cases when a student of a course is also a teacher of another
|
|
|
|
|
lab must be handled correctly
|
|
|
|
|
- _alternate authentication methods_ -- logging in through a university
|
|
|
|
|
_supervisor_. The cases when a student of a course is also a teacher of another
|
|
|
|
|
course must be handled correctly.
|
|
|
|
|
- _alternative authentication methods_ -- logging in through a university
|
|
|
|
|
authentication system (e.g. LDAP) and potentially other services, such as
|
|
|
|
|
OAuth, should be supported
|
|
|
|
|
- _querying SIS_ -- loading user data from the university information system
|
|
|
|
|
Github or some other OAuth service, should be supported
|
|
|
|
|
- _querying SIS_ -- loading user data from the university information system (SIS)
|
|
|
|
|
should be supported
|
|
|
|
|
- _sandboxing_ -- there should be more advanced sandboxing which supports
|
|
|
|
|
execution of parallel programs and easy integration of different programming
|
|
|
|
|
environments and tools; the sandboxed environment should have the least
|
|
|
|
|
possible impact on measurement results (most importantly on measured times)
|
|
|
|
|
- _heterogeneous worker pool_ -- there must be support for submission evaluation
|
|
|
|
|
- _sandboxing_ -- there should be a more advanced sandbox which supports
|
|
|
|
|
execution of parallel programs and an easy integration of different programming
|
|
|
|
|
environments and tools; the sandboxed environment should have the minimum
|
|
|
|
|
possible impact on the measurement of results (most importantly on the measured
|
|
|
|
|
duration of execution)
|
|
|
|
|
- _heterogeneous worker pool_ -- there must be a support for submission evaluation
|
|
|
|
|
in multiple programming environments in a single installation to avoid
|
|
|
|
|
unacceptable workload for the administrator (maintaining a separate
|
|
|
|
|
installation for every course) and high hardware occupation
|
|
|
|
|
unacceptable workload of the administrator (i.e., maintaining a separate
|
|
|
|
|
installation for every course) and high hardware requirements
|
|
|
|
|
- advanced low-level evaluation flow configuration with high-level abstraction
|
|
|
|
|
layer for ordinary configuration cases; the configuration should be able to
|
|
|
|
|
express more complicated flows than just compiling a source code and running
|
|
|
|
|
the program against test inputs -- for example, some exercises need to build
|
|
|
|
|
the program against the test inputs -- for example, some exercises need to build
|
|
|
|
|
the source code with a tool, run some tests, then run the program through
|
|
|
|
|
another tool and perform additional tests
|
|
|
|
|
- use of modern technologies with state-of-the-art compilers
|
|
|
|
@ -277,14 +278,14 @@ presented in groups by the user categories.
|
|
|
|
|
|
|
|
|
|
- _no installation_ -- the primary user interface of the system must be
|
|
|
|
|
accessible on the computers of the users without the need to install any
|
|
|
|
|
additional software except for a web browser (which is installed on a vast
|
|
|
|
|
majority of personal computers)
|
|
|
|
|
additional software except for a web browser which is installed on almost
|
|
|
|
|
every personal computer
|
|
|
|
|
- _performance_ -- the system must be ready for at least hundreds of students
|
|
|
|
|
and tens of supervisors using it at once
|
|
|
|
|
and tens of supervisors who are using it at the same time
|
|
|
|
|
- _automated deployment_ -- all of the components of the system must be easy to
|
|
|
|
|
deploy in an automated fashion
|
|
|
|
|
- _open source licensing_ -- the source code should be released under a
|
|
|
|
|
permissive licence allowing further development; this also applies to used
|
|
|
|
|
permissive licence allowing further development; this also applies to the used
|
|
|
|
|
libraries and frameworks
|
|
|
|
|
- _multi-platform worker_ -- worker machines running Linux, Windows and
|
|
|
|
|
potentially other operating systems must be supported
|
|
|
|
@ -294,40 +295,40 @@ presented in groups by the user categories.
|
|
|
|
|
The survey shows that there are a lot of different requirements and wishes for
|
|
|
|
|
the new system. When the system is ready, it is likely that there will be new
|
|
|
|
|
ideas on how to use the system and thus the system must be designed to be easily
|
|
|
|
|
extendable, so that these new ideas can be easily implemented, either by us or
|
|
|
|
|
extendable so that these new ideas can be easily implemented, either by us or
|
|
|
|
|
community members. This also means that widely used programming languages and
|
|
|
|
|
techniques should be used, so that users can quickly understand the code and
|
|
|
|
|
make changes.
|
|
|
|
|
techniques should be used so the programmers can quickly understand the code and
|
|
|
|
|
make changes easily.
|
|
|
|
|
|
|
|
|
|
## Related work
|
|
|
|
|
|
|
|
|
|
To find out the current state in the field of automatic grading systems, we did
|
|
|
|
|
a short market survey on the field of automatic grading systems at universities,
|
|
|
|
|
programming contests, and possibly other places where similar tools are
|
|
|
|
|
available.
|
|
|
|
|
To find out the current state in the field of automatic grading systems, we conducted
|
|
|
|
|
a short market survey of the field of automatic grading systems at universities,
|
|
|
|
|
programming contests, and other places where similar tools are used.
|
|
|
|
|
|
|
|
|
|
This is not a complete list of available evaluators, but only a few projects
|
|
|
|
|
which are used these days and can be an inspiration for our project. Each
|
|
|
|
|
project from the list has a brief description and some key features mentioned.
|
|
|
|
|
This is not a complete list of available evaluators, but only of a few projects
|
|
|
|
|
which are used these days and can be of an inspiration for our project. Each
|
|
|
|
|
project on the list is provided with a brief description and some of its key features.
|
|
|
|
|
|
|
|
|
|
### Progtest
|
|
|
|
|
|
|
|
|
|
[Progtest](https://progtest.fit.cvut.cz/) is private project of [FIT
|
|
|
|
|
[Progtest](https://progtest.fit.cvut.cz/) is a private project of [FIT
|
|
|
|
|
ČVUT](https://fit.cvut.cz) in Prague. As far as we know it is used for C/C++,
|
|
|
|
|
Bash programming and knowledge-based quizzes. There are several bonus points
|
|
|
|
|
and penalties and also a few hints what is failing in the submitted solution. It
|
|
|
|
|
is very strict on source code quality, for example `-pedantic` option of GCC,
|
|
|
|
|
Valgrind for memory leaks or array boundaries checks via `mudflap` library.
|
|
|
|
|
Bash programming, and knowledge-based quizzes. Each submitted solution can receive
|
|
|
|
|
several bonus points or penalties and also a few hints can be attached of
|
|
|
|
|
what is incorrect in the solution. It is very strict with the source code quality,
|
|
|
|
|
for example the `-pedantic` option of GCC is used; Valgrind is used for detection of
|
|
|
|
|
memory leaks; array boundaries are checked via the `mudflap` library.
|
|
|
|
|
|
|
|
|
|
### Codility
|
|
|
|
|
|
|
|
|
|
[Codility](https://codility.com/) is a web based solution primary targeted to
|
|
|
|
|
company recruiters. It is a commercial product available as a SaaS and it
|
|
|
|
|
[Codility](https://codility.com/) is a web based solution primarily targeted at
|
|
|
|
|
company recruiters. It is a commercial product available as SaaS and it
|
|
|
|
|
supports 16 programming languages. The
|
|
|
|
|
[UI](http://1.bp.blogspot.com/-_isqWtuEvvY/U8_SbkUMP-I/AAAAAAAAAL0/Hup_amNYU2s/s1600/cui.png)
|
|
|
|
|
of Codility is [opensource](https://github.com/Codility/cui), the rest of source
|
|
|
|
|
code is not available. One interesting feature is 'task timeline' -- captured
|
|
|
|
|
progress of writing code for each user.
|
|
|
|
|
code is not available. One interesting feature is the 'task timeline' -- the captured
|
|
|
|
|
progress of writing the code for each user.
|
|
|
|
|
|
|
|
|
|
### CMS
|
|
|
|
|
|
|
|
|
@ -337,17 +338,17 @@ contains several modules. CMS supports C/C++, Pascal, Python, PHP, and Java
|
|
|
|
|
programming languages. PostgreSQL is a single point of failure, all modules
|
|
|
|
|
heavily depend on the database connection. Task evaluation can be only a three
|
|
|
|
|
step pipeline -- compilation, execution, evaluation. Execution is performed in
|
|
|
|
|
[Isolate](https://github.com/ioi/isolate), sandbox written by the consultant
|
|
|
|
|
[Isolate](https://github.com/ioi/isolate), a sandbox written by the consultant
|
|
|
|
|
of our project, Mgr. Martin Mareš, Ph.D.
|
|
|
|
|
|
|
|
|
|
### MOE
|
|
|
|
|
|
|
|
|
|
[MOE](http://www.ucw.cz/moe/) is a grading system written in Shell scripts, C
|
|
|
|
|
and Python. It does not provide a default GUI interface, all actions have to be
|
|
|
|
|
performed from command line. The system does not evaluate submissions in real
|
|
|
|
|
time, results are computed in batch mode after exercise deadline, using Isolate
|
|
|
|
|
performed from the command line. The system does not evaluate submissions in real
|
|
|
|
|
time, results are computed in a batch mode after the exercise deadline, using Isolate
|
|
|
|
|
for sandboxing. Parts of MOE are used in other systems like CodEx or CMS, but
|
|
|
|
|
the system is generally obsolete.
|
|
|
|
|
the system is obsolete in general.
|
|
|
|
|
|
|
|
|
|
### Kattis
|
|
|
|
|
|
|
|
|
@ -358,7 +359,6 @@ format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for
|
|
|
|
|
exercises. Kattis is primarily used by programming contest organizers, company
|
|
|
|
|
recruiters and also some universities.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# Analysis
|
|
|
|
|
|
|
|
|
|
None of the existing projects we came across fulfills all the requested features
|
|
|
|
|