|
|
@ -178,9 +178,9 @@ Typical use cases for supported user roles are following:
|
|
|
|
### Exercise evaluation chain
|
|
|
|
### Exercise evaluation chain
|
|
|
|
|
|
|
|
|
|
|
|
The most important part of the system is evaluation of solutions submitted by
|
|
|
|
The most important part of the system is evaluation of solutions submitted by
|
|
|
|
students. Concepts of consecutive steps from source code to final results
|
|
|
|
students. The process leading from source code to final results (score) is
|
|
|
|
is described in more detail below to give readers solid overview of what have to
|
|
|
|
described in more detail below to give readers a solid overview of what happens
|
|
|
|
happen during evaluation process.
|
|
|
|
during the evaluation process.
|
|
|
|
|
|
|
|
|
|
|
|
First thing students have to do is to submit their solutions through web user
|
|
|
|
First thing students have to do is to submit their solutions through web user
|
|
|
|
interface. The system checks assignment invariants (deadlines, count of
|
|
|
|
interface. The system checks assignment invariants (deadlines, count of
|
|
|
@ -190,9 +190,9 @@ configuration variant is chosen (one exercise can have multiple variants, for
|
|
|
|
example C and Java languages). This exercise configuration is then used for
|
|
|
|
example C and Java languages). This exercise configuration is then used for
|
|
|
|
taking care of evaluation process.
|
|
|
|
taking care of evaluation process.
|
|
|
|
|
|
|
|
|
|
|
|
There is a pool of uniform worker engines dedicated to evaluation jobs. Incoming
|
|
|
|
There is a pool of uniform worker machines dedicated to evaluation jobs.
|
|
|
|
jobs are kept in a queue until a free worker picks them. Worker is capable of
|
|
|
|
Incoming jobs are kept in a queue until a free worker picks them. Workers are
|
|
|
|
sequential evaluation of jobs, one at a time.
|
|
|
|
capable of sequential evaluation of jobs, one at a time.
|
|
|
|
|
|
|
|
|
|
|
|
The worker obtains the solution and its evaluation configuration, parses it and
|
|
|
|
The worker obtains the solution and its evaluation configuration, parses it and
|
|
|
|
starts executing the contained instructions. It is crucial to keep the worker
|
|
|
|
starts executing the contained instructions. It is crucial to keep the worker
|
|
|
@ -212,9 +212,9 @@ Current system is old, but robust. There were no major security incidents
|
|
|
|
during its production usage. However, from today's perspective there are
|
|
|
|
during its production usage. However, from today's perspective there are
|
|
|
|
several drawbacks. The main ones are:
|
|
|
|
several drawbacks. The main ones are:
|
|
|
|
|
|
|
|
|
|
|
|
- **web interface** -- The web interface is simple and fully functional. But
|
|
|
|
- **web interface** -- The web interface is simple and fully functional.
|
|
|
|
rapid development in web technologies opens new horizons of how web interface
|
|
|
|
However, recent rapid development in web technologies opens new horizons of
|
|
|
|
can be made.
|
|
|
|
how web interfaces can be made.
|
|
|
|
- **web API** -- CodEx offers a very limited XML API based on outdated
|
|
|
|
- **web API** -- CodEx offers a very limited XML API based on outdated
|
|
|
|
technologies that is not sufficient for users who would like to create custom
|
|
|
|
technologies that is not sufficient for users who would like to create custom
|
|
|
|
interfaces such as a command line tool or mobile application.
|
|
|
|
interfaces such as a command line tool or mobile application.
|
|
|
@ -224,8 +224,8 @@ several drawbacks. The main ones are:
|
|
|
|
parallelism is a very important area of computing, so there is requirement to
|
|
|
|
parallelism is a very important area of computing, so there is requirement to
|
|
|
|
test multi-threaded applications too.
|
|
|
|
test multi-threaded applications too.
|
|
|
|
- **instances** -- Different ways of CodEx usage scenarios requires separate
|
|
|
|
- **instances** -- Different ways of CodEx usage scenarios requires separate
|
|
|
|
instances (Programming I and II, Java, C#, etc.). This configuration is not
|
|
|
|
installations (Programming I and II, Java, C#, etc.). This configuration is
|
|
|
|
user friendly (students have to register in each instance separately) and
|
|
|
|
not user friendly (students have to register in each instance separately) and
|
|
|
|
burdens administrators with unnecessary work. CodEx architecture does not
|
|
|
|
burdens administrators with unnecessary work. CodEx architecture does not
|
|
|
|
allow sharing hardware between instances, which results in an inefficient use
|
|
|
|
allow sharing hardware between instances, which results in an inefficient use
|
|
|
|
of hardware for evaluation.
|
|
|
|
of hardware for evaluation.
|
|
|
@ -234,7 +234,6 @@ several drawbacks. The main ones are:
|
|
|
|
which have a more difficult evaluation chain than simple
|
|
|
|
which have a more difficult evaluation chain than simple
|
|
|
|
compilation/execution/evaluation provided by CodEx.
|
|
|
|
compilation/execution/evaluation provided by CodEx.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Requirements
|
|
|
|
## Requirements
|
|
|
|
|
|
|
|
|
|
|
|
There are many different formal requirements for the system. Some of them
|
|
|
|
There are many different formal requirements for the system. Some of them
|
|
|
@ -352,7 +351,7 @@ annoyed if they did not.
|
|
|
|
|
|
|
|
|
|
|
|
The survey shows that there are a lot of different requirements and wishes for
|
|
|
|
The survey shows that there are a lot of different requirements and wishes for
|
|
|
|
the new system. When the system is ready, it is likely that there will be new
|
|
|
|
the new system. When the system is ready, it is likely that there will be new
|
|
|
|
ideas of how to use the system and thus the system must be designed to be easily
|
|
|
|
ideas on how to use the system and thus the system must be designed to be easily
|
|
|
|
extendable, so that these new ideas can be easily implemented, either by us or
|
|
|
|
extendable, so that these new ideas can be easily implemented, either by us or
|
|
|
|
community members. This also means that widely used programming languages and
|
|
|
|
community members. This also means that widely used programming languages and
|
|
|
|
techniques should be used, so that users can quickly understand the code and
|
|
|
|
techniques should be used, so that users can quickly understand the code and
|
|
|
@ -430,7 +429,7 @@ implies that only a subset of all the features will be implemented in the first
|
|
|
|
version, and more of them will come in the following releases.
|
|
|
|
version, and more of them will come in the following releases.
|
|
|
|
|
|
|
|
|
|
|
|
Gathered features are categorized based on priorities for the whole system. The
|
|
|
|
Gathered features are categorized based on priorities for the whole system. The
|
|
|
|
highest priority is the functionality presente in current CodEx. It is a base
|
|
|
|
highest priority is the functionality present in current CodEx. It is a base
|
|
|
|
line for being useful in production environment. The design of the new solution
|
|
|
|
line for being useful in production environment. The design of the new solution
|
|
|
|
shall allow extending the system easily. Ideas from faculty staff have secondary
|
|
|
|
shall allow extending the system easily. Ideas from faculty staff have secondary
|
|
|
|
priority, but most of them will be implemented as part of the project. The most
|
|
|
|
priority, but most of them will be implemented as part of the project. The most
|
|
|
@ -458,17 +457,17 @@ reasonable wishes of university users and our major design choices.
|
|
|
|
|
|
|
|
|
|
|
|
- modern HTML5 web frontend written in JavaScript using a suitable framework
|
|
|
|
- modern HTML5 web frontend written in JavaScript using a suitable framework
|
|
|
|
- REST API communicating with database, evaluation backend and a file server
|
|
|
|
- REST API communicating with database, evaluation backend and a file server
|
|
|
|
- evaluation backend implemented as a distributed system on top of a message
|
|
|
|
- evaluation backend implemented as a distributed system on top of a messaging
|
|
|
|
queue framework with master-worker architecture
|
|
|
|
framework with master-worker architecture
|
|
|
|
- multi-platform worker supporting Linux and Windows environment (latter
|
|
|
|
- multi-platform worker supporting Linux and Windows environment (latter without
|
|
|
|
without sandbox, no general purpose suitable tool available yet)
|
|
|
|
sandbox, no general purpose suitable tool available yet)
|
|
|
|
- evaluation procedure configured in a human readable text file, compound of
|
|
|
|
- evaluation procedure configured in a human readable text file, compound of
|
|
|
|
small tasks connected into an arbitrary oriented acyclic graph
|
|
|
|
small tasks connected into an arbitrary oriented acyclic graph
|
|
|
|
|
|
|
|
|
|
|
|
The reasons supporting these decisions are explained in the rest of analysis
|
|
|
|
The reasons supporting these decisions are explained in the rest of analysis
|
|
|
|
chapter. Also a lot of smaller design choices are mentioned including possible
|
|
|
|
chapter. Also a lot of smaller design choices are mentioned including possible
|
|
|
|
options, what is picked to implement and why. But first, discuss basic concepts
|
|
|
|
options, what is picked to implement and why. But first, we discuss basic
|
|
|
|
of the system.
|
|
|
|
concepts of the system.
|
|
|
|
|
|
|
|
|
|
|
|
## Basic concepts
|
|
|
|
## Basic concepts
|
|
|
|
|
|
|
|
|
|
|
|