@ -105,6 +105,14 @@ addressed. Furthermore, many teachers are willing to use and test the new
system. Following requirements were collected both from our personal experience
system. Following requirements were collected both from our personal experience
with CodEx and from teachers' requests.
with CodEx and from teachers' requests.
## Requirements
@todo: sekce kde budou ty pozadavky (o neco podrobneji nez ted). Navrhoval bych
to rozdelit minimalne na funkcni a nefunkcni pozadavky, ale i ty funkcni je
mozna vhodne rozdelit na ciste uzivatelske (ulohy se maji bodovat, studenti
dostanou od cviciho komentar, ...) a administracne-technicke (napojeni na SIS a
LDAP, rizeni pristupu pomoci roli,...).
###Basic grading system requirements:
###Basic grading system requirements:
These are the features which are necessary for any system for evaluation of
These are the features which are necessary for any system for evaluation of
@ -117,7 +125,7 @@ programming coding assignments used in any university programming course:
- teachers can create exercises including textual description, sample inputs and
- teachers can create exercises including textual description, sample inputs and
correct reference outputs (for example "sum all numbers from given file and
correct reference outputs (for example "sum all numbers from given file and
write the result to the standard output")
write the result to the standard output")
- teachers can assigning an existing exercise to their class with some specific
- teachers can assign an existing exercise to their class with some specific
properties set (deadlines, etc.)
properties set (deadlines, etc.)
- teachers can specify their scale of points which will be awarted to the
- teachers can specify their scale of points which will be awarted to the
students depending on the correctness of his/her solution (expressed in
students depending on the correctness of his/her solution (expressed in
@ -146,6 +154,8 @@ current system.
###Requested features for the new system:
###Requested features for the new system:
First, there is a list of ideas influencing system capabilities:
- logging in through a university authentication system (e.g. LDAP)
- logging in through a university authentication system (e.g. LDAP)
- support for multiple programming environments at once to avoid unacceptable
- support for multiple programming environments at once to avoid unacceptable
workload for administrator (maintain separate installations for many courses)
workload for administrator (maintain separate installations for many courses)
@ -164,6 +174,20 @@ current system.
layer for ordinary configuration cases
layer for ordinary configuration cases
- use of modern technologies with state-of-the-art compilers
- use of modern technologies with state-of-the-art compilers
Then, there are also some requirements of technical character with no direct
mapping to visible parts of system. Most notably they are these ones:
- user interface of the system accessible on users' computers without
installation of any kind of additional software
- easy implementation of different user interfaces
- be ready for workload hundreads of students and tens of supervisors
- automated installation of all components
@todo: fill some nonfunctional requirements;
melo byt lepe popsano, ze je potreba oddelit UI od aplikacni logiky, aby si
nekdo mohl napsat jine UI, ze je potreba mit backend, ktery bude skalovat na
vykon a dovoli pridavani dalsich platforem atd.
The survey shows that the system is used in many different ways, but the core
The survey shows that the system is used in many different ways, but the core
functionality is the same for all of them. When the system is ready it is
functionality is the same for all of them. When the system is ready it is
likely that there will be new ideas of how to use the system and thus the system
likely that there will be new ideas of how to use the system and thus the system
@ -271,53 +295,6 @@ exercises. Kattis is primarily used by programming contest organizators, company
recruiters and also some universities.
recruiters and also some universities.
## ReCodEx goals
None of the existing systems we came across is capable of all the required
features of the new system. There is no grading system which is designed to
support a complicated evaluation pipeline, so this part is an unexplored field
and has to be designed with caution. Also, no project is modern and extensible
so it could be used as a base for ReCodEx. After considering all these facts,
it was clear that a new system has to be written from scratch. This implies,
that only a subset of all the features will be implemented in the first version,
the other in the following ones.
Gathered features are categorized based on priorities for the whole system. The
highest priority has main functionality similar to current CodEx. It is a base
line to be useful in production environment, but a new design allows to easily
develop further. On top of that, most of ideas from faculty staff belongs to
second priority bucket, which will be implemented as part of the project. Most
advanced tasks from this category are advanced low-level evaluation
configuration format, using modern tools, connecting to a university systems and
merging separate system instances into single one. Other tasks are scheduled for
next releases after successful project defense. Namely, these are high-level
exercise evaluation configuration with user-friendly interface for common
exercise types, SIS integration (when some API will be available from their
side) and command-line submit tool. Plagiarism detection is not likely to be
part of any release in near future unless someone other makes the engine. The
detection problem is too hard to be solved as part of this project.
We named the project as **ReCodEx -- ReCodEx Code Examiner**. The name should
point to the old CodEx, but also reflect the new approach to solve issues.
**Re** as part of the name means redesigned, rewritten, renewed, or restarted.
At this point there is a clear idea how the new system will be used and what are
the major enhancements for future releases. With this in mind, the overall
architecture can be sketched. From the previous research, we set up several
goals, which the new system should have. They mostly reflect drawbacks of the
current version of CodEx and some reasonable wishes of university users. Most
notable features are following:
- modern HTML5 web frontend written in JavaScript using a suitable framework
- REST API implemented in PHP, communicating with database, evaluation backend
and a file server
- evaluation backend implemented as a distributed system on top of a message
queue framework (ZeroMQ) with master-worker architecture <!-- @todo: WTF is
worker??? The concept has not been introduced yet! -->
- worker with basic support of the Windows environment (without sandbox, no
general purpose suitable tool available yet)
- evaluation procedure configured in a YAML file, compound of small tasks
connected into an arbitrary oriented acyclic graph
### Intended usage
### Intended usage
@ -404,6 +381,61 @@ overview which part succeeded and which failed (optionally with reason like
# Analysis
# Analysis
## ReCodEx goals
@todo: improve and extend this chapter - analysis of user requrements and way we
solve them; exercise is a template for assignment, users are in groups, what is
group, how points are assigned for solutions, ...
@todo: merge with next chapter (Solution concept analysis)
None of the existing systems we came across is capable of all the required
features of the new system. There is no grading system which is designed to
support a complicated evaluation pipeline, so this part is an unexplored field
and has to be designed with caution. Also, no project is modern and extensible
so it could be used as a base for ReCodEx. After considering all these facts,
it was clear that a new system has to be written from scratch. This implies,
that only a subset of all the features will be implemented in the first version,
the other in the following ones.
Gathered features are categorized based on priorities for the whole system. The
highest priority has main functionality similar to current CodEx. It is a base
line to be useful in production environment, but a new design allows to easily
develop further. On top of that, most of ideas from faculty staff belongs to
second priority bucket, which will be implemented as part of the project. Most
advanced tasks from this category are advanced low-level evaluation
configuration format, using modern tools, connecting to a university systems and
merging separate system instances into single one. Other tasks are scheduled for
next releases after successful project defense. Namely, these are high-level
exercise evaluation configuration with user-friendly interface for common
exercise types, SIS integration (when some API will be available from their
side) and command-line submit tool. Plagiarism detection is not likely to be
part of any release in near future unless someone other makes the engine. The
detection problem is too hard to be solved as part of this project.
We named the project as **ReCodEx -- ReCodEx Code Examiner**. The name should
point to the old CodEx, but also reflect the new approach to solve issues.
**Re** as part of the name means redesigned, rewritten, renewed, or restarted.
At this point there is a clear idea how the new system will be used and what are
the major enhancements for future releases. With this in mind, the overall
architecture can be sketched. From the previous research, we set up several
goals, which the new system should have. They mostly reflect drawbacks of the
current version of CodEx and some reasonable wishes of university users. Most
notable features are following:
- modern HTML5 web frontend written in JavaScript using a suitable framework
- REST API implemented in PHP, communicating with database, evaluation backend
and a file server
- evaluation backend implemented as a distributed system on top of a message
queue framework (ZeroMQ) with master-worker architecture <!-- @todo: WTF is
worker??? The concept has not been introduced yet! -->
- worker with basic support of the Windows environment (without sandbox, no
general purpose suitable tool available yet)
- evaluation procedure configured in a YAML file, compound of small tasks
connected into an arbitrary oriented acyclic graph
## Solution concepts analysis
## Solution concepts analysis
@todo: what problems were solved on abstract and high levels, how they can be solved and what was the final solution
@todo: what problems were solved on abstract and high levels, how they can be solved and what was the final solution
@ -421,7 +453,8 @@ overview which part succeeded and which failed (optionally with reason like
- discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why
- discuss several ways how points can be assigned to solution, propose basic systems but also general systems which can use outputs from judges or other executed programs, there is need for variables or other concept, explain why
- and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks
- and many many more general concepts which can be discussed and solved... please append more of them if something comes to your mind... thanks
### Structure of the project
## Structure of the project
There are numerous ways how to divide some sort of system into separated
There are numerous ways how to divide some sort of system into separated
services, from one single component to many and many single-purpose components.
services, from one single component to many and many single-purpose components.
@ -608,16 +641,6 @@ interesting observations will be discussed in this chapter.
@todo: what type of communication within backend could be used, mention some frameworks, queue managers, protocols, which was considered
@todo: what type of communication within backend could be used, mention some frameworks, queue managers, protocols, which was considered
### Fileserver
@todo: fileserver and why is separated
@todo: mention hashing on fileserver and why this approach was chosen
@todo: what can be stored on fileserver
@todo: how can jobs be stored on fileserver, mainly mention that it is nonsence to store inputs and outputs within job archive
### Broker
### Broker
@todo: assigning of jobs to workers, which are possible algorithms, queues, which one was chosen
@todo: assigning of jobs to workers, which are possible algorithms, queues, which one was chosen
@ -709,6 +732,16 @@ Previous description implies that there is gap between detection of last access
@todo: sandboxing, what possibilites are out there (Linux, Windows), what are general and really needed features, mention isolate, what are isolate features
@todo: sandboxing, what possibilites are out there (Linux, Windows), what are general and really needed features, mention isolate, what are isolate features
### Fileserver
@todo: fileserver and why is separated
@todo: mention hashing on fileserver and why this approach was chosen
@todo: what can be stored on fileserver
@todo: how can jobs be stored on fileserver, mainly mention that it is nonsence to store inputs and outputs within job archive
### Monitor
### Monitor
Users want to view real time evaluation progress of their solution. It can be
Users want to view real time evaluation progress of their solution. It can be
@ -765,69 +798,85 @@ of connection with no message loss.
### Frontend communication
### Frontend communication
The first thing we need to address is the communication protocol of this client-server
The first thing we need to address is the communication protocol of this
architecture. There are several options:
client-server architecture. There are several options:
- *TCP sockets* -- TCP sockets give a reliable means of a full-duplex communication. All major
- *TCP sockets* -- TCP sockets give a reliable means of a full-duplex
operating systems support this protocol and there are libraries which simplify the implementation.
communication. All major operating systems support this protocol and there are
On the other side, it is not possible to initiate a TCP socket from a web browser.
libraries which simplify the implementation. On the other side, it is not
- *WebSockets* -- The WebSocket standard is built on top of TCP. It enables a web browser to connect
possible to initiate a TCP socket from a web browser.
to a server over a TCP socket. WebSockets are implemented in recent versions of all modern web browsers
- *WebSockets* -- The WebSocket standard is built on top of TCP. It enables a
and there are libraries for several programming languages like Python or JavaScript (running in Node.js).
web browser to connect to a server over a TCP socket. WebSockets are
Encryption of the communication over a WebSocket is supported as a standard.
implemented in recent versions of all modern web browsers and there are
- *HTTP protocol* -- The HTTP protocol is a state-less protocol implemented on top of the TCP protocol.
libraries for several programming languages like Python or JavaScript (running
The communication between the client and server consists of a requests sent by the client and reponses
in Node.js). Encryption of the communication over a WebSocket is supported as
to these requests sent back by the sever. The client can send as many requests as needed and it may ignore
a standard.
the responses from the server, but the server must respond only to the requests of the client and
- *HTTP protocol* -- The HTTP protocol is a state-less protocol implemented on
it cannot initiate communication on its own. End-to-end encryption can be achieved easily using SSL (HTTPS).
top of the TCP protocol. The communication between the client and server
consists of a requests sent by the client and reponses to these requests sent
We chose the HTTP(S) protocol because of the simple implementation in all sorts of operating systems
back by the sever. The client can send as many requests as needed and it may
and runtime environments on both the client and the server side.
ignore the responses from the server, but the server must respond only to the
requests of the client and it cannot initiate communication on its own.
The API of the server should expose basic CRUD (Create, Read, Update, Delete) operations. There are some
End-to-end encryption can be achieved easily using SSL (HTTPS).
options on what kind of messages to send over the HTTP:
We chose the HTTP(S) protocol because of the simple implementation in all sorts
of operating systems and runtime environments on both the client and the server
side.
The API of the server should expose basic CRUD (Create, Read, Update, Delete)
operations. There are some options on what kind of messages to send over the
HTTP:
- SOAP -- a protocol for exchanging XML messages. It is very robust and complex.
- SOAP -- a protocol for exchanging XML messages. It is very robust and complex.
- REST -- is a stateless architecture style, not a protocol or a technology. It relies on HTTP (but not
- REST -- is a stateless architecture style, not a protocol or a technology. It
necessarily) and its method verbs (e.g., GET, POST, PUT, DELETE). It can fully implement the CRUD operations.
relies on HTTP (but not necessarily) and its method verbs (e.g., GET, POST,
PUT, DELETE). It can fully implement the CRUD operations.
Even though there are some other technologies we chose the REST style over the HTTP protocolo. It is widely
Even though there are some other technologies we chose the REST style over the
used, there are many tools available for development and testing, and it is understood by programmers so
HTTP protocolo. It is widely used, there are many tools available for
it should be easy for a new developer with some experience in client-side applications to get to know with
development and testing, and it is understood by programmers so it should be
the ReCodEx API and develop a client application.
easy for a new developer with some experience in client-side applications to get
to know with the ReCodEx API and develop a client application.
### API server
### API server
The API server must handle HTTP requests and manage the state of the application in some kind of a database.
The API server must handle HTTP requests and manage the state of the application
It must also be able to communicate with the backend over ZeroMQ.
in some kind of a database. It must also be able to communicate with the
backend over ZeroMQ.
We considered several technologies which could be used:
We considered several technologies which could be used:
- PHP + Apache -- one of the most widely used technologies for creating web servers. It is a suitable
- PHP + Apache -- one of the most widely used technologies for creating web
technology for this kind of a project. It has all the features we need when some additional extensions
servers. It is a suitable technology for this kind of a project. It has all
are installed (to support LDAP or ZeroMQ).
the features we need when some additional extensions are installed (to support
- ASP.NET (C#), JSP (Java) -- these technologies are very robust and are used to create server technologies in many big
LDAP or ZeroMQ).
enterpises. Both can run on Windows and Linux servers (ASP.NET using the .NET Core).
- ASP.NET (C#), JSP (Java) -- these technologies are very robust and are used to
- JavaScript (Node.js) -- it is a quite new technology and it is being used to create REST APIs lately.
create server technologies in many big enterpises. Both can run on Windows and
Applications running on Node.js are quite performant and the number of open-source libraries avialble
Linux servers (ASP.NET using the .NET Core).
on the Internet is very huge.
- JavaScript (Node.js) -- it is a quite new technology and it is being used to
create REST APIs lately. Applications running on Node.js are quite performant
We chose PHP and Apache mainly because we were familiar with these technologies and we were able to develop
and the number of open-source libraries avialble on the Internet is very huge.
all the features we needed without learning to use a new technology. Since the number of features was quite
high and needed to meet a strict deadline. This does not mean that we would find all the other technologies
We chose PHP and Apache mainly because we were familiar with these technologies
superior to PHP in all other aspects - PHP 7 is a mature language with a huge comunity and a wide range of
and we were able to develop all the features we needed without learning to use a
tools, libraries, and frameworks.
new technology. Since the number of features was quite high and needed to meet a
strict deadline. This does not mean that we would find all the other
We decided to use an ORM framework to manage the database, namely the widely used PHP ORM Doctrine 2.
technologies superior to PHP in all other aspects - PHP 7 is a mature language
This framework has a robust abstraction layer DBAL so the database engine is not very important and it can be changed
with a huge comunity and a wide range of tools, libraries, and frameworks.
without any need for changing the code. We chose an open-source database MariaDB.
We decided to use an ORM framework to manage the database, namely the widely
To speed up the development process of the PHP server application we decided to use an MVC framework.
used PHP ORM Doctrine 2. This framework has a robust abstraction layer DBAL so
After evaluating and trying several frameworks, such as Lumen, Laravel, and Symfony, we ended up using
the database engine is not very important and it can be changed without any need
the framework Nette. This framework is very common in the Czech Republic -- its main developer is a
for changing the code. We chose an open-source database MariaDB.
well-known Czech programmer David Grudl -- and we were already familiar with the patterns used in this
framework (e.g., dependency injection, authentication, routing). There is a good extension for the
To speed up the development process of the PHP server application we decided to
Nette framework which makes usage of Doctrine 2 very straighforward.
use an MVC framework. After evaluating and trying several frameworks, such as
Lumen, Laravel, and Symfony, we ended up using the framework Nette. This
framework is very common in the Czech Republic -- its main developer is a
well-known Czech programmer David Grudl -- and we were already familiar with the
patterns used in this framework (e.g., dependency injection, authentication,
routing). There is a good extension for the Nette framework which makes usage of
Doctrine 2 very straighforward.
@todo: what database can be used, how it is mapped and used within code
@todo: what database can be used, how it is mapped and used within code
@ -1617,6 +1666,6 @@ used.
chdir: ${EVAL_DIR}
chdir: ${EVAL_DIR}
```
```
<!---
<!---
// vim: set textwidth=80:
// vim: set formatoptions=ant textwidth=80 colorcolumn=+1:
Petr Stefan
8 years ago