requirements fixed

master
Simon Rozsival 8 years ago
parent 009928f304
commit af0c09586d

@ -246,29 +246,30 @@ presented in groups by the user categories.
### Administrative Requirements
- _pluggable user interface_ -- the system should allow using an alternative
- _independent user interface_ -- the system should allow the use of an alternative
user interface, such as a command line client; implementation of such clients
should be as straightforward as possible
- _privilege separation_ -- there should be at least two roles -- _student_ and
_supervisor_. Cases when a student of a course is also a teacher of another
lab must be handled correctly
- _alternate authentication methods_ -- logging in through a university
_supervisor_. The cases when a student of a course is also a teacher of another
course must be handled correctly.
- _alternative authentication methods_ -- logging in through a university
authentication system (e.g. LDAP) and potentially other services, such as
OAuth, should be supported
- _querying SIS_ -- loading user data from the university information system
Github or some other OAuth service, should be supported
- _querying SIS_ -- loading user data from the university information system (SIS)
should be supported
- _sandboxing_ -- there should be more advanced sandboxing which supports
execution of parallel programs and easy integration of different programming
environments and tools; the sandboxed environment should have the least
possible impact on measurement results (most importantly on measured times)
- _heterogeneous worker pool_ -- there must be support for submission evaluation
- _sandboxing_ -- there should be a more advanced sandbox which supports
execution of parallel programs and an easy integration of different programming
environments and tools; the sandboxed environment should have the minimum
possible impact on the measurement of results (most importantly on the measured
duration of execution)
- _heterogeneous worker pool_ -- there must be a support for submission evaluation
in multiple programming environments in a single installation to avoid
unacceptable workload for the administrator (maintaining a separate
installation for every course) and high hardware occupation
unacceptable workload of the administrator (i.e., maintaining a separate
installation for every course) and high hardware requirements
- advanced low-level evaluation flow configuration with high-level abstraction
layer for ordinary configuration cases; the configuration should be able to
express more complicated flows than just compiling a source code and running
the program against test inputs -- for example, some exercises need to build
the program against the test inputs -- for example, some exercises need to build
the source code with a tool, run some tests, then run the program through
another tool and perform additional tests
- use of modern technologies with state-of-the-art compilers
@ -277,14 +278,14 @@ presented in groups by the user categories.
- _no installation_ -- the primary user interface of the system must be
accessible on the computers of the users without the need to install any
additional software except for a web browser (which is installed on a vast
majority of personal computers)
additional software except for a web browser which is installed on almost
every personal computer
- _performance_ -- the system must be ready for at least hundreds of students
and tens of supervisors using it at once
and tens of supervisors who are using it at the same time
- _automated deployment_ -- all of the components of the system must be easy to
deploy in an automated fashion
- _open source licensing_ -- the source code should be released under a
permissive licence allowing further development; this also applies to used
permissive licence allowing further development; this also applies to the used
libraries and frameworks
- _multi-platform worker_ -- worker machines running Linux, Windows and
potentially other operating systems must be supported
@ -294,40 +295,40 @@ presented in groups by the user categories.
The survey shows that there are a lot of different requirements and wishes for
the new system. When the system is ready, it is likely that there will be new
ideas on how to use the system and thus the system must be designed to be easily
extendable, so that these new ideas can be easily implemented, either by us or
extendable so that these new ideas can be easily implemented, either by us or
community members. This also means that widely used programming languages and
techniques should be used, so that users can quickly understand the code and
make changes.
techniques should be used so the programmers can quickly understand the code and
make changes easily.
## Related work
To find out the current state in the field of automatic grading systems, we did
a short market survey on the field of automatic grading systems at universities,
programming contests, and possibly other places where similar tools are
available.
To find out the current state in the field of automatic grading systems, we conducted
a short market survey of the field of automatic grading systems at universities,
programming contests, and other places where similar tools are used.
This is not a complete list of available evaluators, but only a few projects
which are used these days and can be an inspiration for our project. Each
project from the list has a brief description and some key features mentioned.
This is not a complete list of available evaluators, but only of a few projects
which are used these days and can be of an inspiration for our project. Each
project on the list is provided with a brief description and some of its key features.
### Progtest
[Progtest](https://progtest.fit.cvut.cz/) is private project of [FIT
[Progtest](https://progtest.fit.cvut.cz/) is a private project of [FIT
ČVUT](https://fit.cvut.cz) in Prague. As far as we know it is used for C/C++,
Bash programming and knowledge-based quizzes. There are several bonus points
and penalties and also a few hints what is failing in the submitted solution. It
is very strict on source code quality, for example `-pedantic` option of GCC,
Valgrind for memory leaks or array boundaries checks via `mudflap` library.
Bash programming, and knowledge-based quizzes. Each submitted solution can receive
several bonus points or penalties and also a few hints can be attached of
what is incorrect in the solution. It is very strict with the source code quality,
for example the `-pedantic` option of GCC is used; Valgrind is used for detection of
memory leaks; array boundaries are checked via the `mudflap` library.
### Codility
[Codility](https://codility.com/) is a web based solution primary targeted to
company recruiters. It is a commercial product available as a SaaS and it
[Codility](https://codility.com/) is a web based solution primarily targeted at
company recruiters. It is a commercial product available as SaaS and it
supports 16 programming languages. The
[UI](http://1.bp.blogspot.com/-_isqWtuEvvY/U8_SbkUMP-I/AAAAAAAAAL0/Hup_amNYU2s/s1600/cui.png)
of Codility is [opensource](https://github.com/Codility/cui), the rest of source
code is not available. One interesting feature is 'task timeline' -- captured
progress of writing code for each user.
code is not available. One interesting feature is the 'task timeline' -- the captured
progress of writing the code for each user.
### CMS
@ -337,17 +338,17 @@ contains several modules. CMS supports C/C++, Pascal, Python, PHP, and Java
programming languages. PostgreSQL is a single point of failure, all modules
heavily depend on the database connection. Task evaluation can be only a three
step pipeline -- compilation, execution, evaluation. Execution is performed in
[Isolate](https://github.com/ioi/isolate), sandbox written by the consultant
[Isolate](https://github.com/ioi/isolate), a sandbox written by the consultant
of our project, Mgr. Martin Mareš, Ph.D.
### MOE
[MOE](http://www.ucw.cz/moe/) is a grading system written in Shell scripts, C
and Python. It does not provide a default GUI interface, all actions have to be
performed from command line. The system does not evaluate submissions in real
time, results are computed in batch mode after exercise deadline, using Isolate
performed from the command line. The system does not evaluate submissions in real
time, results are computed in a batch mode after the exercise deadline, using Isolate
for sandboxing. Parts of MOE are used in other systems like CodEx or CMS, but
the system is generally obsolete.
the system is obsolete in general.
### Kattis
@ -358,7 +359,6 @@ format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for
exercises. Kattis is primarily used by programming contest organizers, company
recruiters and also some universities.
# Analysis
None of the existing projects we came across fulfills all the requested features

Loading…
Cancel
Save