@ -105,6 +105,14 @@ addressed. Furthermore, many teachers are willing to use and test the new
system. Following requirements were collected both from our personal experience
with CodEx and from teachers' requests.
## Requirements
@todo: sekce kde budou ty pozadavky (o neco podrobneji nez ted). Navrhoval bych
to rozdelit minimalne na funkcni a nefunkcni pozadavky, ale i ty funkcni je
mozna vhodne rozdelit na ciste uzivatelske (ulohy se maji bodovat, studenti
dostanou od cviciho komentar, ...) a administracne-technicke (napojeni na SIS a
LDAP, rizeni pristupu pomoci roli,...).
###Basic grading system requirements:
These are the features which are necessary for any system for evaluation of
@ -117,7 +125,7 @@ programming coding assignments used in any university programming course:
- teachers can create exercises including textual description, sample inputs and
correct reference outputs (for example "sum all numbers from given file and
write the result to the standard output")
- teachers can assigning an existing exercise to their class with some specific
- teachers can assign an existing exercise to their class with some specific
properties set (deadlines, etc.)
- teachers can specify their scale of points which will be awarted to the
students depending on the correctness of his/her solution (expressed in
@ -146,6 +154,8 @@ current system.
###Requested features for the new system:
First, there is a list of ideas influencing system capabilities:
- logging in through a university authentication system (e.g. LDAP)
- support for multiple programming environments at once to avoid unacceptable
workload for administrator (maintain separate installations for many courses)
@ -164,6 +174,20 @@ current system.
layer for ordinary configuration cases
- use of modern technologies with state-of-the-art compilers
Then, there are also some requirements of technical character with no direct
mapping to visible parts of system. Most notably they are these ones:
- user interface of the system accessible on users' computers without
installation of any kind of additional software
- easy implementation of different user interfaces
- be ready for workload hundreads of students and tens of supervisors
- automated installation of all components
@todo: fill some nonfunctional requirements;
melo byt lepe popsano, ze je potreba oddelit UI od aplikacni logiky, aby si
nekdo mohl napsat jine UI, ze je potreba mit backend, ktery bude skalovat na
vykon a dovoli pridavani dalsich platforem atd.
The survey shows that the system is used in many different ways, but the core
functionality is the same for all of them. When the system is ready it is
likely that there will be new ideas of how to use the system and thus the system
@ -271,53 +295,6 @@ exercises. Kattis is primarily used by programming contest organizators, company
recruiters and also some universities.
## ReCodEx goals
None of the existing systems we came across is capable of all the required
features of the new system. There is no grading system which is designed to
support a complicated evaluation pipeline, so this part is an unexplored field
and has to be designed with caution. Also, no project is modern and extensible
so it could be used as a base for ReCodEx. After considering all these facts,
it was clear that a new system has to be written from scratch. This implies,
that only a subset of all the features will be implemented in the first version,
the other in the following ones.
Gathered features are categorized based on priorities for the whole system. The
highest priority has main functionality similar to current CodEx. It is a base
line to be useful in production environment, but a new design allows to easily
develop further. On top of that, most of ideas from faculty staff belongs to
second priority bucket, which will be implemented as part of the project. Most
advanced tasks from this category are advanced low-level evaluation
configuration format, using modern tools, connecting to a university systems and
merging separate system instances into single one. Other tasks are scheduled for
next releases after successful project defense. Namely, these are high-level
exercise evaluation configuration with user-friendly interface for common
exercise types, SIS integration (when some API will be available from their
side) and command-line submit tool. Plagiarism detection is not likely to be
part of any release in near future unless someone other makes the engine. The
detection problem is too hard to be solved as part of this project.
We named the project as **ReCodEx -- ReCodEx Code Examiner**. The name should
point to the old CodEx, but also reflect the new approach to solve issues.
**Re** as part of the name means redesigned, rewritten, renewed, or restarted.
At this point there is a clear idea how the new system will be used and what are
the major enhancements for future releases. With this in mind, the overall
architecture can be sketched. From the previous research, we set up several
goals, which the new system should have. They mostly reflect drawbacks of the
current version of CodEx and some reasonable wishes of university users. Most
notable features are following:
- modern HTML5 web frontend written in JavaScript using a suitable framework
- REST API implemented in PHP, communicating with database, evaluation backend
and a file server
- evaluation backend implemented as a distributed system on top of a message
queue framework (ZeroMQ) with master-worker architecture <!-- @todo: WTF is
worker??? The concept has not been introduced yet! -->
- worker with basic support of the Windows environment (without sandbox, no
general purpose suitable tool available yet)
- evaluation procedure configured in a YAML file, compound of small tasks
connected into an arbitrary oriented acyclic graph
### Intended usage
@ -404,6 +381,61 @@ overview which part succeeded and which failed (optionally with reason like
# Analysis
## ReCodEx goals
@todo: improve and extend this chapter - analysis of user requrements and way we
solve them; exercise is a template for assignment, users are in groups, what is
group, how points are assigned for solutions, ...
@todo: merge with next chapter (Solution concept analysis)
None of the existing systems we came across is capable of all the required
features of the new system. There is no grading system which is designed to
support a complicated evaluation pipeline, so this part is an unexplored field
and has to be designed with caution. Also, no project is modern and extensible
so it could be used as a base for ReCodEx. After considering all these facts,
it was clear that a new system has to be written from scratch. This implies,
that only a subset of all the features will be implemented in the first version,
the other in the following ones.
Gathered features are categorized based on priorities for the whole system. The
highest priority has main functionality similar to current CodEx. It is a base
line to be useful in production environment, but a new design allows to easily
develop further. On top of that, most of ideas from faculty staff belongs to
second priority bucket, which will be implemented as part of the project. Most
advanced tasks from this category are advanced low-level evaluation
configuration format, using modern tools, connecting to a university systems and
merging separate system instances into single one. Other tasks are scheduled for
next releases after successful project defense. Namely, these are high-level
exercise evaluation configuration with user-friendly interface for common
exercise types, SIS integration (when some API will be available from their
side) and command-line submit tool. Plagiarism detection is not likely to be
part of any release in near future unless someone other makes the engine. The
detection problem is too hard to be solved as part of this project.
We named the project as **ReCodEx -- ReCodEx Code Examiner**. The name should
point to the old CodEx, but also reflect the new approach to solve issues.
**Re** as part of the name means redesigned, rewritten, renewed, or restarted.
At this point there is a clear idea how the new system will be used and what are
the major enhancements for future releases. With this in mind, the overall
architecture can be sketched. From the previous research, we set up several
goals, which the new system should have. They mostly reflect drawbacks of the
current version of CodEx and some reasonable wishes of university users. Most
notable features are following:
- modern HTML5 web frontend written in JavaScript using a suitable framework
- REST API implemented in PHP, communicating with database, evaluation backend
and a file server
- evaluation backend implemented as a distributed system on top of a message
queue framework (ZeroMQ) with master-worker architecture <!-- @todo: WTF is
worker??? The concept has not been introduced yet! -->
- worker with basic support of the Windows environment (without sandbox, no
general purpose suitable tool available yet)
- evaluation procedure configured in a YAML file, compound of small tasks
connected into an arbitrary oriented acyclic graph
## Solution concepts analysis
@todo: what problems were solved on abstract and high levels, how they can be solved and what was the final solution
@ -421,7 +453,8 @@ overview which part succeeded and which failed (optionally with reason like
- discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why
- and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks
### Structure of the project
## Structure of the project
There are numerous ways how to divide some sort of system into separated
services, from one single component to many and many single-purpose components.
@ -608,16 +641,6 @@ interesting observations will be discussed in this chapter.
@todo: what type of communication within backend could be used, mention some frameworks, queue managers, protocols, which was considered
### Fileserver
@todo: fileserver and why is separated
@todo: mention hashing on fileserver and why this approach was chosen
@todo: what can be stored on fileserver
@todo: how can jobs be stored on fileserver, mainly mention that it is nonsence to store inputs and outputs within job archive
### Broker
@todo: assigning of jobs to workers, which are possible algorithms, queues, which one was chosen
@ -709,6 +732,16 @@ Previous description implies that there is gap between detection of last access
@todo: sandboxing, what possibilites are out there (Linux, Windows), what are general and really needed features, mention isolate, what are isolate features
### Fileserver
@todo: fileserver and why is separated
@todo: mention hashing on fileserver and why this approach was chosen
@todo: what can be stored on fileserver
@todo: how can jobs be stored on fileserver, mainly mention that it is nonsence to store inputs and outputs within job archive
### Monitor
Users want to view real time evaluation progress of their solution. It can be
@ -765,69 +798,85 @@ of connection with no message loss.
### Frontend communication
The first thing we need to address is the communication protocol of this client-server
architecture. There are several options:
- *TCP sockets* -- TCP sockets give a reliable means of a full-duplex communication. All major
operating systems support this protocol and there are libraries which simplify the implementation.
On the other side, it is not possible to initiate a TCP socket from a web browser.
- *WebSockets* -- The WebSocket standard is built on top of TCP. It enables a web browser to connect
to a server over a TCP socket. WebSockets are implemented in recent versions of all modern web browsers
and there are libraries for several programming languages like Python or JavaScript (running in Node.js).
Encryption of the communication over a WebSocket is supported as a standard.
- *HTTP protocol* -- The HTTP protocol is a state-less protocol implemented on top of the TCP protocol.
The communication between the client and server consists of a requests sent by the client and reponses
to these requests sent back by the sever. The client can send as many requests as needed and it may ignore
the responses from the server, but the server must respond only to the requests of the client and
it cannot initiate communication on its own. End-to-end encryption can be achieved easily using SSL (HTTPS).
We chose the HTTP(S) protocol because of the simple implementation in all sorts of operating systems
and runtime environments on both the client and the server side.
The API of the server should expose basic CRUD (Create, Read, Update, Delete) operations. There are some
options on what kind of messages to send over the HTTP:
The first thing we need to address is the communication protocol of this
client-server architecture. There are several options:
- *TCP sockets* -- TCP sockets give a reliable means of a full-duplex
communication. All major operating systems support this protocol and there are
libraries which simplify the implementation. On the other side, it is not
possible to initiate a TCP socket from a web browser.
- *WebSockets* -- The WebSocket standard is built on top of TCP. It enables a
web browser to connect to a server over a TCP socket. WebSockets are
implemented in recent versions of all modern web browsers and there are
libraries for several programming languages like Python or JavaScript (running
in Node.js). Encryption of the communication over a WebSocket is supported as
a standard.
- *HTTP protocol* -- The HTTP protocol is a state-less protocol implemented on
top of the TCP protocol. The communication between the client and server
consists of a requests sent by the client and reponses to these requests sent
back by the sever. The client can send as many requests as needed and it may
ignore the responses from the server, but the server must respond only to the
requests of the client and it cannot initiate communication on its own.
End-to-end encryption can be achieved easily using SSL (HTTPS).
We chose the HTTP(S) protocol because of the simple implementation in all sorts
of operating systems and runtime environments on both the client and the server
side.
The API of the server should expose basic CRUD (Create, Read, Update, Delete)
operations. There are some options on what kind of messages to send over the
HTTP:
- SOAP -- a protocol for exchanging XML messages. It is very robust and complex.
- REST -- is a stateless architecture style, not a protocol or a technology. It relies on HTTP (but not
necessarily) and its method verbs (e.g., GET, POST, PUT, DELETE). It can fully implement the CRUD operations.
- REST -- is a stateless architecture style, not a protocol or a technology. It
relies on HTTP (but not necessarily) and its method verbs (e.g., GET, POST,
PUT, DELETE). It can fully implement the CRUD operations.
Even though there are some other technologies we chose the REST style over the HTTP protocolo. It is widely
used, there are many tools available for development and testing, and it is understood by programmers so
it should be easy for a new developer with some experience in client-side applications to get to know with
the ReCodEx API and develop a client application.
Even though there are some other technologies we chose the REST style over the
HTTP protocolo. It is widely used, there are many tools available for
development and testing, and it is understood by programmers so it should be
easy for a new developer with some experience in client-side applications to get
to know with the ReCodEx API and develop a client application.
### API server
The API server must handle HTTP requests and manage the state of the application in some kind of a database.
It must also be able to communicate with the backend over ZeroMQ.
The API server must handle HTTP requests and manage the state of the application
in some kind of a database. It must also be able to communicate with the
backend over ZeroMQ.
We considered several technologies which could be used:
- PHP + Apache -- one of the most widely used technologies for creating web servers. It is a suitable
technology for this kind of a project. It has all the features we need when some additional extensions
are installed (to support LDAP or ZeroMQ).
- ASP.NET (C#), JSP (Java) -- these technologies are very robust and are used to create server technologies in many big
enterpises. Both can run on Windows and Linux servers (ASP.NET using the .NET Core).
- JavaScript (Node.js) -- it is a quite new technology and it is being used to create REST APIs lately.
Applications running on Node.js are quite performant and the number of open-source libraries avialble
on the Internet is very huge.
We chose PHP and Apache mainly because we were familiar with these technologies and we were able to develop
all the features we needed without learning to use a new technology. Since the number of features was quite
high and needed to meet a strict deadline. This does not mean that we would find all the other technologies
superior to PHP in all other aspects - PHP 7 is a mature language with a huge comunity and a wide range of
tools, libraries, and frameworks.
We decided to use an ORM framework to manage the database, namely the widely used PHP ORM Doctrine 2.
This framework has a robust abstraction layer DBAL so the database engine is not very important and it can be changed
without any need for changing the code. We chose an open-source database MariaDB.
To speed up the development process of the PHP server application we decided to use an MVC framework.
After evaluating and trying several frameworks, such as Lumen, Laravel, and Symfony, we ended up using
the framework Nette. This framework is very common in the Czech Republic -- its main developer is a
well-known Czech programmer David Grudl -- and we were already familiar with the patterns used in this
framework (e.g., dependency injection, authentication, routing). There is a good extension for the
Nette framework which makes usage of Doctrine 2 very straighforward.
- PHP + Apache -- one of the most widely used technologies for creating web
servers. It is a suitable technology for this kind of a project. It has all
the features we need when some additional extensions are installed (to support
LDAP or ZeroMQ).
- ASP.NET (C#), JSP (Java) -- these technologies are very robust and are used to
create server technologies in many big enterpises. Both can run on Windows and
Linux servers (ASP.NET using the .NET Core).
- JavaScript (Node.js) -- it is a quite new technology and it is being used to
create REST APIs lately. Applications running on Node.js are quite performant
and the number of open-source libraries avialble on the Internet is very huge.
We chose PHP and Apache mainly because we were familiar with these technologies
and we were able to develop all the features we needed without learning to use a
new technology. Since the number of features was quite high and needed to meet a
strict deadline. This does not mean that we would find all the other
technologies superior to PHP in all other aspects - PHP 7 is a mature language
with a huge comunity and a wide range of tools, libraries, and frameworks.
We decided to use an ORM framework to manage the database, namely the widely
used PHP ORM Doctrine 2. This framework has a robust abstraction layer DBAL so
the database engine is not very important and it can be changed without any need
for changing the code. We chose an open-source database MariaDB.
To speed up the development process of the PHP server application we decided to
use an MVC framework. After evaluating and trying several frameworks, such as
Lumen, Laravel, and Symfony, we ended up using the framework Nette. This
framework is very common in the Czech Republic -- its main developer is a
well-known Czech programmer David Grudl -- and we were already familiar with the
patterns used in this framework (e.g., dependency injection, authentication,
routing). There is a good extension for the Nette framework which makes usage of
Doctrine 2 very straighforward.
@todo: what database can be used, how it is mapped and used within code
@ -1594,9 +1643,9 @@ our case) is necessary.
- "${SOURCE_DIR}/reference.txt"
```
Comparison of results is quite straightforward. It is important to set the task
type to _evaluation_, so that the return code is set to 0 if the program is
correct and 1 otherwise. We do not set our own limits, so the default limits are
Comparison of results is quite straightforward. It is important to set the task
type to _evaluation_, so that the return code is set to 0 if the program is
correct and 1 otherwise. We do not set our own limits, so the default limits are
used.
```{.yml}
@ -1617,6 +1666,6 @@ used.
chdir: ${EVAL_DIR}
```
<!---
// vim: set textwidth=80:
// vim: set formatoptions=ant textwidth=80 colorcolumn=+1:
Petr Stefan
8 years ago