LEdoian
/
recodex-wiki
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Divide rewritten documentation into separate chapters

Browse Source
master
Martin Polanka 8 years ago
parent abec266d97
commit 6c5736f7ea
8 changed files with 4078 additions and 335 deletions
Show Stats Download Patch File Download Diff File

1445
Analysis.md
View File

File diff suppressed because it is too large Load Diff

88
Conclusion.md
Unescape Escape View File

@ -0,0 +1,88 @@
# Conclusion
The project of ReCodEx was a great experience with developing a bigger
application in team of people. We mostly enjoyed the time spent on the project,
however the deadline was too tight to implement all nice features we thought of.
Our implementation meets our predefined goals and is ready to be deployed
instead of current tool CodEx.
We made several design choices during planning and implementing the project.
From perspective of today we are mostly happy about them and we believe we have
done the best we could. One of the biggest questions was the broker/worker
programming language. The winner was C++, which is really great language
especially in revisions starting at C++11. But we are not sure if Python would
not have been better option. From the list of good choices we would like to
mention early unit testing with continuous integration, using ZeroMQ messaging
framework and the design with splitting logic into multiple components.
To sum up, we created an extensible base environment for years of further
development by numerous student projects. The faculty would benefit from higher
hardware utilization, less administration effort and a possibility to offer
ReCodEx as SaaS to partner grammar schools. Students would also appreciate
state-of-art development tools.
During the development we were using whole bunch of development tools. We would
like to thank authors, maintainers and contributors of these tools to help us
complete this project. Most notably we would like to mention explicitly the top
projects: Git and GitHub, Visual Studio Code, IDEs from JetBrains, ZeroMQ
framework, C++ spdlog library, Travis and AppVeyor CI, Nette framework and many
many others. Thank you.
Finally, we would like to thank our supervisor, Martin Kruliš, for his support
and great effort correcting text of this documentation.
## Further Improvements
A lot of work has been done, but it opened a whole bunch of new possibilities
for subsequent student projects and other kinds of contribution. We would be
happy to see people contributing to this project to make it even more awesome.
We are presenting a brief list of features we think might be worth of
completing. Surely, the list is not complete and may change in time.
- Finish web frontend. In time of the project submission it does not contain all
features for the best user experience in all roles. This task is suitable for
people with basic web programming experience, React and Redux knowledge is an
advantage.
- Create web editor of job configuration. ReCodEx job evaluation follows quite
complex task flow. To create valid configuration, author of exercise has to
write a pretty long YAML file by hand. This of course can be partially or
fully automated but was not implemented at time. This task therefore consist
of creating web based editor of this configuration with prepared task
templates allowing user to change their properties and connect them. After
submit, YAML job configuration should be properly created.
- Write alternative commandline frontend. A lot of users want to submit their
solutions directly from command line. That is newly possible in ReCodEx
because of REST API, but no suitable frontend exists yet. There is unfinished
attempt to create one in NodeJS, accessible in ReCodEx organization on
[GitHub](https://github.com/ReCodEx/cli). The goal is to finish this project
or create alternative tool. We would like to see one written in Python.
- Create mobile frontend. It would be really nice to have mobile application
when you can read exercise assignments and examine your progress and
statistics. All is possible due to ReCodEx REST API. The basic Android
application is in development by ReCodEx team as project for programming
mobile devices class. The code will be published as part of ReCodEx GitHub
organization.
- Design and implement backend monitoring. For administrator would be great to
examine status of backend (broker, workers, monitor, fileserver) directly in
browser. Key information are which machines are online, offline or failed,
which worker is now running a job (maybe with current progress) and view
configuration of worker. Also some statistics with graphs should be made, for
example workload of workers. More advanced feature is the ability to restart
workers on the change of their configurations.
- Finish .NET sandbox for Windows. We developed initial sandbox for Windows
environment, [WrapSharp](https://github.com/ReCodEx/wrapsharp). WrapSharp is
only for .NET platform assemblies and most notably for C# programs and cannot
be used generally for sandboxing on Windows. The goal is to finish
implementation and do a really detailed security audit of the project. Also,
integration to the worker is not fully done yet.
- SIS integration. A very nice feature is (semi)automatic creation of groups and
assigning students to them depending on their timetable in student information
system (SIS). However there is no standardized API for such communication yet,
but we hope this will be possible in the future. Implementing this feature
means extending ReCodEx API by a SIS module in PHP.
<!---
// vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
-->

1323
Implementation.md
View File

File diff suppressed because it is too large Load Diff

365
Introduction.md
Unescape Escape View File

@ -0,0 +1,365 @@
# Introduction
In general there are many different ways and opinions on how to teach people
something new. However, most people agree that a hands-on experience is one of
the best ways to make the human brain remember a new skill. Learning must be
entertaining and interactive, with fast and frequent feedback. Some areas
are more suitable for this practical way of learning than others, and
fortunately, programming is one of them.
University education system is one of the areas where this knowledge can be
applied. In computer programming, there are several requirements a program
should satisfy, such as the code being syntactically correct, efficient and easy
to read, maintain and extend.
Checking programs written by students by hand takes time and requires a lot of
repetitive work -- reviewing source codes, compiling them and
running them through test scenarios. It is therefore desirable to automate as
much of this process as possible.
The first idea of an automatic evaluation system
comes from Stanford University professors in 1965. They implemented a system
which evaluated code in Algol submitted on punch cards. In following years, many
similar products were written.
Nowadays properties like correctness and efficiency can be tested
to a large extent automatically. This fact should be exploited to help teachers
save time for tasks such as examining bad design, bad coding habits, or logical
mistakes, which are difficult to perform automatically.
There are two basic ways of automatically evaluating code:
- **statically** -- by checking the source code without running it.
This is safe, but not very practical.
- **dynamically** -- by running the code on test inputs and checking the correctness of
outputs ones. This provides good real world experience, but requires extensive
security measures).
This project focuses on the machine-controlled part of source code evaluation.
First we observed the general concepts of grading systems and discussed the problems of the
software previously used at the Charles University in Prague.
Then the new requirements were specified and we examined projects with similar functionality.
With the acquired knowledge from these projects, we set up
goals for the new evaluation system, designed the architecture and implemented a
fully operational solution based on dynamic evaluation. The system is now ready
for production testing at the university.
## Assignment
The major goal of this project is to create a grading application which will be
used for programming classes at the Faculty of Mathematics and Physics of the
Charles University in Prague. However, the application should be designed in a
modular fashion to be easily extended or even modified to make other ways of
usage possible.
The system should be capable of doing a dynamic analysis of the submitted source
codes. This consists of the following basic steps:
1. compile the code and check for compilation errors
2. run the compiled program in a sandbox with predefined inputs
3. check the constraints of the amount of used memory and time
4. compare the outputs of the program with the defined expected outputs
5. award the solution with a numeric score
The whole system is intended to help both the teachers (supervisors) and the students.
To achieve this, it is crucial for us to keep in mind the typical usage scenarios of
the system and to try to make these tasks as simple as possible. To fulfill this
task, the project has a great starting point -- there is an old grading system
currently used at the university (CodEx), so its flaws and weaknesses can be
addressed. Furthermore, many teachers desire to use and test the new system and
they are willing to consult our ideas or problems during the development with us.
## Current System
The grading solution currently used at the Faculty of Mathematics and Physics of
the Charles University in Prague was implemented in 2006 by a group of students.
It is called [CodEx -- The Code Examiner](http://codex.ms.mff.cuni.cz/project/)
and it has been used with some improvements since then. The original plan was to
use the system only for the basic programming courses, but there was a demand for
adapting it for several different courses.
CodEx is based on dynamic analysis. It features a web-based interface, where
supervisors can assign exercises to their students and the students have a time
window to submit their solutions. Each solution is compiled and run in sandbox
(MO-Eval). The metrics which are checked are: correctness of the output, time
and memory limits. It supports programs written in C, C++, C#, Java, Pascal,
Python and Haskell.
The system has a database of users. Each user is assigned a role, which
corresponds to his/her privileges. There are user groups reflecting the
structure of lectured courses.
A database of exercises (algorithmic problems) is another part of the project.
Each exercise consists of a text describing the problem, a configuration of the
evaluation (machine-readable instructions on how to evaluate solutions to the
exercise), time and memory limits for all supported runtimes (e.g. programming
languages), a configuration for calculating the final score and a set of inputs
and reference outputs. Exercises are created by instructed privileged users.
Assigning an exercise to a group means choosing one of the available exercises
and specifying additional properties: a deadline (optionally a second deadline),
a maximum amount of points, a maximum number of submissions and a list of
supported runtime environments.
The typical use cases for the user roles are the following:
- **student**
- create a new user account via a registration form
- join groups (e.g., the courses he attends)
- get assignments in the groups
- submit a solution to an assignment -- upload one source file and start the
evaluation process
- view the results of the solution -- which parts succeeded and failed, the total
number of the acquired points, bonus points
- **supervisor** (similar to CodEx *operator*)
- create a new exercise -- create description text and evaluation configuration
(for each programming environment), upload testing inputs and outputs
- assign an exercise to a group -- choose an exercise and set the deadlines,
the number of allowed submissions, the weights of all test cases and the amount
of points for the correct solutions
- modify an assignment
- view all of the results of the students in a group
- review the automatic solution evaluation -- view the submitted source files
and optionally set bonus points (including negative points)
- **administrator**
- create groups
- alter user privileges -- make supervisor accounts
- check system logs
### Exercise Evaluation Chain
The most important part of the system is the evaluation of solutions submitted by
the students. The process from the source code to final results (score) is
described in more detail below to give readers a solid overview of what is happening
during the evaluation process.
The first thing students have to do is to submit their solutions through the web user
interface. The system checks assignment invariants (e.g., deadlines, number of
submissions) and stores the submitted code. The runtime environment is
automatically detected based on the extension of the input file, and a suitable evaluation
configuration type is chosen (one exercise can have multiple variants, for
example C and Java is allowed). This exercise configuration is then used for
the evaluation process.
There is a pool of uniform worker machines dedicated to evaluation jobs.
Incoming jobs are kept in a queue until a free worker picks them. Workers are
capable of a sequential evaluation of jobs, one at a time.
The worker obtains the solution and its evaluation configuration, parses it and
starts executing the instructions contained. Each job should have more test
cases which examine invalid inputs, corner cases and data of different sizes to
estimate the program complexity. It is crucial to keep the computer running the worker
secure and stable, so a sandboxed environment is used for dealing with an
unknown source code. When the execution is finished, results are saved, and the
student is notified.
The output of the worker contains data about the evaluation, such as time and
memory spent on running the program for each test input and whether its output
is correct. The system then calculates a numeric score from the data which is
presented to the student. If the solution is incorrect (e.g., incorrect output,
exceeds memory or time limits), error messages are also displayed to the student.
### Possible Improvements
The current system is old, but robust. There were no major security incidents
in the course of its usage. However, from the present day perspective there are
several major drawbacks:
- **web interface** -- The web interface is simple and fully functional.
However, the recent rapid development in web technologies provides us with new
possibilities of making web interfaces.
- **public API** -- CodEx offers a very limited public XML API based on outdated
technologies that are not sufficient for users who would like to create their
custom interfaces such as a command line tool or a mobile application.
- **sandboxing** -- the MO-Eval sandbox is based on the principle of monitoring
system calls and blocking the forbidden ones. This can be sufficient with
single-threaded programs, but proves to be difficult with multi-threaded ones.
Nowadays, parallelism is a very important area of computing, it is required that
multi-threaded programs can be securely tested as well.
- **instances** -- Different ways of CodEx use require separate
installations (e.g., Programming I and II, Java, C#). This configuration is
not user friendly as students have to register in each installation separately
and burdens administrators with unnecessary work. The CodEx architecture does not
allow sharing workers between installations which results in an inefficient
use of hardware for evaluation.
- **task extensibility** -- There is a need to test and evaluate complicated
programs for courses such as *Parallel programming* or *Compiler principles*,
which have a more difficult evaluation chain than simple
*compilation/execution/evaluation* provided by CodEx.
## Requirements
There are many different formal requirements for the system. Some of them
are necessary for any system for source code evaluation, some of them are
specific for university deployment and some of them arose during the ten year
long lifetime of the old system. There are not many ways of improving CodEx
experience from the perspective of a student, but a lot of feature requests come
from administrators and supervisors. The ideas were gathered mostly from our
personal experience with the system and from meetings with the faculty staff
who use the current system.
In general, CodEx features should be preserved, so only the differences are
presented here. For clear arrangement, all the requirements and wishes are
presented in groups by the user categories.
### Requirements of The Users
- _group hierarchy_ -- creating an arbitrarily nested tree structure should be
supported to keep related groups together, such as in the example
below. CodEx supported only a flat group structure. A group hierarchy also
allows to archive data from the past courses.
```
Summer term 2016
|-- Language C# and .NET platform
| |-- Labs Monday 10:30
| `-- Labs Thursday 9:00
|-- Programming I
| |-- Labs Monday 14:00
...
```
- _a database of exercises_ -- teachers should be able to filter the displayed
exercises according to several criteria, for example by the supported runtime
environments or by the author. It should also be possible to link exercises to a group
so that group supervisors do not have to browse hundreds of exercises when
their group only uses a few of them
- _advanced exercises_ -- the system should support a more advanced evaluation
pipeline than basic *compilation/execution/evaluation* which is in CodEx
- _customizable grading system_ -- teachers need to specify the way of
calculating the final score which will be allocated to the submissions
depending on their correctness and quality
- _marking a solution as accepted_ -- a supervisor should be able to choose
one of the submitted solutions of a student as accepted. The score of this
particular solution will be used as the score which the student receives
for the given assignment instead of the one with the highest score.
- _solution resubmission_ -- teachers should be able to edit the solutions of the
student and privately resubmit them, optionally saving all results (including
temporary ones); this feature can be used to quickly fix obvious errors in the
solution and see if it is otherwise correct
- _localization_ -- all texts (the UI and the assignments of the exercises) should
be translatable into several languages
- _formatted texts of assignments_ -- Markdown or another lightweight markup language
should be supported for the formatting of the texts of the exercises
- _comments_ -- adding both private and public comments to exercises, tests and
solutions should be supported
- _plagiarism detection_
### Administrative Requirements
- _independent user interface_ -- the system should allow the use of an alternative
user interface, such as a command line client; implementation of such clients
should be as straightforward as possible
- _privilege separation_ -- there should be at least two roles -- _student_ and
_supervisor_. The cases when a student of a course is also a teacher of another
course must be handled correctly.
- _alternative authentication methods_ -- logging in through a university
authentication system (e.g. LDAP) and potentially other services, such as
Github or some other OAuth service, should be supported
- _querying SIS_ -- loading user data from the university information system (SIS)
should be supported
- _sandboxing_ -- there should be a more advanced sandbox which supports
execution of parallel programs and an easy integration of different programming
environments and tools; the sandboxed environment should have the minimum
possible impact on the measurement of results (most importantly on the measured
duration of execution)
- _heterogeneous worker pool_ -- there must be a support for submission evaluation
in multiple programming environments in a single installation to avoid
unacceptable workload of the administrator (i.e., maintaining a separate
installation for every course) and high hardware requirements
- advanced low-level evaluation flow configuration with high-level abstraction
layer for ordinary configuration cases; the configuration should be able to
express more complicated flows than just compiling a source code and running
the program against the test inputs -- for example, some exercises need to build
the source code with a tool, run some tests, then run the program through
another tool and perform additional tests
- use of modern technologies with state-of-the-art compilers
### Non-functional Requirements
- _no installation_ -- the primary user interface of the system must be
accessible on the computers of the users without the need to install any
additional software except for a web browser which is installed on almost
every personal computer
- _performance_ -- the system must be ready for at least hundreds of students
and tens of supervisors who are using it at the same time
- _automated deployment_ -- all of the components of the system must be easy to
deploy in an automated fashion
- _open source licensing_ -- the source code should be released under a
permissive licence allowing further development; this also applies to the used
libraries and frameworks
- _multi-platform worker_ -- worker machines running Linux, Windows and
potentially other operating systems must be supported
### Conclusion
The survey shows that there are a lot of different requirements and wishes for
the new system. When the system is ready, it is likely that there will be new
ideas on how to use the system and thus the system must be designed to be easily
extendable so that these new ideas can be easily implemented, either by us or
community members. This also means that widely used programming languages and
techniques should be used so the programmers can quickly understand the code and
make changes easily.
## Related work
To find out the current state in the field of automatic grading systems, we conducted
a short market survey of the field of automatic grading systems at universities,
programming contests, and other places where similar tools are used.
This is not a complete list of available evaluators, but only of a few projects
which are used these days and can be of an inspiration for our project. Each
project on the list is provided with a brief description and some of its key features.
### Progtest
[Progtest](https://progtest.fit.cvut.cz/) is a private project of [FIT
ČVUT](https://fit.cvut.cz) in Prague. As far as we know it is used for C/C++,
Bash programming, and knowledge-based quizzes. Each submitted solution can receive
several bonus points or penalties and also a few hints can be attached of
what is incorrect in the solution. It is very strict with the source code quality,
for example the `-pedantic` option of GCC is used; Valgrind is used for detection of
memory leaks; array boundaries are checked via the `mudflap` library.
### Codility
[Codility](https://codility.com/) is a web based solution primarily targeted at
company recruiters. It is a commercial product available as SaaS and it
supports 16 programming languages. The
[UI](http://1.bp.blogspot.com/-_isqWtuEvvY/U8_SbkUMP-I/AAAAAAAAAL0/Hup_amNYU2s/s1600/cui.png)
of Codility is [opensource](https://github.com/Codility/cui), the rest of source
code is not available. One interesting feature is the 'task timeline' -- the captured
progress of writing the code for each user.
### CMS
[CMS](http://cms-dev.github.io/index.html) is an opensource distributed system
for running and organizing programming contests. It is written in Python and
contains several modules. CMS supports C/C++, Pascal, Python, PHP, and Java
programming languages. PostgreSQL is a single point of failure, all modules
heavily depend on the database connection. Task evaluation can be only a three
step pipeline -- compilation, execution, evaluation. Execution is performed in
[Isolate](https://github.com/ioi/isolate), a sandbox written by the consultant
of our project, Mgr. Martin Mareš, Ph.D.
### MOE
[MOE](http://www.ucw.cz/moe/) is a grading system written in Shell scripts, C
and Python. It does not provide a default GUI interface, all actions have to be
performed from the command line. The system does not evaluate submissions in real
time, results are computed in a batch mode after the exercise deadline, using Isolate
for sandboxing. Parts of MOE are used in other systems like CodEx or CMS, but
the system is obsolete in general.
### Kattis
[Kattis](http://www.kattis.com/) is another SaaS solution. It provides a clean
and functional web UI, but the rest of the application is too simple. A nice
feature is the usage of a [standardized
format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for
exercises. Kattis is primarily used by programming contest organizers, company
recruiters and also some universities.
<!---
// vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
-->

90
Submission-flow.md
Unescape Escape View File

@ -1,90 +0,0 @@
# Submission flow
This article will describe in execution flow of submission from the point of submission into web application to the point of evaluation of results from execution. Note that following text is simplified and should only show concepts which are used during execution.
## Web Application
First thing users have to do is to submit their solutions to web application which provides interface to upload multiple files and then submit them. More detailed description follows:
1. user upload file by file into prepared submit form
2. after uploading all files connected to assignment user clicks on submit button
3. Web Application send request to Web API that user wants to evaluate assignment with provided files
## Web API
After user submits solution then web application has to hand over all needed information about submission to broker. Submit endpoint is extended with support of "cross-submitting" which means that user with high privileges (supervisor, administrator) can submit solution for user with lower privileges (student). More detailed description follows:
1. deadlines and count of submissions are checked
2. programming language of uploaded files is automatically detected
3. based on programming language proper job configuration file is obtained
4. submitted files from user and job configuration are uploaded to fileserver as archive
5. send message to broker that there is new submission
6. several information about new submission including websocket channel are sent back to user
## Broker
Broker gets information about new submission from API server. At this point broker has to find suitable worker for execution of this particular submission. When worker is found and is jobless, then broker send detailed submission to worker to evaluation. More detailed description follows:
1. broker gets message from API about new submission
2. based on information from API, broker has to choose suitable worker which matches all submission criteria
1. if suitable worker is not found then broker notices Web API about this information and all execution stops
2. suitable worker is found
1. worker has no jobs at the moment, broker immediately sends job evaluation request to it
2. otherwise evaluation request is queued to this particular worker and waits until all previous jobs are done
3. broker sends prepared evaluation request to worker with all information which was obtained from API
## Worker
Worker gets request from broker to evaluate particular submission. Next step is to evaluate given submission and upload results to fileserver. After this worker only send broker that submission was evaluated. More detailed description follows:
1. worker gets evaluation request from broker
2. worker now has to do some initialization of directories and some internal structures for new incoming evaluation
3. users submission archive from given address from evaluation request is downloaded and decompressed
4. job configuration file is located and parsed into internal job structure
5. tasks which are present in configuration are loaded into tree like structure with dependencies and are divided into internal or external ones
1. internal tasks are tasks which has no defined limits and contains some internal action
2. external tasks have defined limits in configuration and are executed in sandbox
6. last step of initializing of job is to topologically sort tasks and prepare queue with right execution order
7. after that topologically sorted execution queue is processed
1. if execution of `inner` task fails worker will immediately stop execution and send back to broker 'internal error during execution' message
2. execution of `execution` or `evaluation` task fails then worker stops and send back to broker 'execution failed' message
3. during execution worker can optionally sends current job status back to web application, this communication is described in more detailed [[communication|Overall architecture#communication]] article.
8. when execution successfully ends results are collected from all executed tasks and written into yaml file
9. results yaml file alongside with content of job result folder is sent back to fileserver in compressed form
10. of course there has to be some cleaning after whole evaluation which will mostly delete content of temporarily created folders
11. last step of worker is to send back to broker message that execution successfully ended
## Broker
Broker gets done message from worker and basically only mark submission as done in its internal structures. After that broker has to tell Web API that execution of particular job ended. More detailed description follows:
1. broker gets message from worker that execution of job ended
2. job execution on worker was successful
1. Web API is notified that job ended successfully
2. if there are some waiting jobs for worker than the first one is sent for execution
3. job execution on worker ended with internal error
1. if job ends with internal error than it is possible to reassign it to another worker
2. broker keeps track of job reassignments and if number of them reach some predefined constant, job is declared as failed
3. suitable worker different that the original one is picked and evaluation request is sent to it
4. job execution on worker failed
1. Web API is notified that execution of this particular job failed
2. again if there is waiting job for execution on worker than it is sent for execution
## Web API
Web API is notified about job status from broker. After that API is deciding if evaluated submission will be evaluated immediately or on demand. But for the sake of simplicity we will consider only immediately evaluated results. More detailed description follows:
1. job arrived with successful evaluation status
1. results of evaluation are downloaded from fileserver where it was uploaded by worker and then unzipped
2. result yaml file is loaded and actual time and memory consumption are retrieved
3. overall results are computed and stored into database
2. job arrived as failed
1. job failure is saved into database alongside other reported errors
2 email is sent to ReCodEx administrator that job failed and cannot be properly executed
## Web Application
Web application has only a simple work to do. It has to obtain results from web API. More detailed description follows:
1. web application sends request for results of particular submission
2. if results are available then they are sent back to web application and displayed to user

857
User-documentation.md
Unescape Escape View File

@ -0,0 +1,857 @@
# User Documentation
Users interact with the ReCodEx through the web application. It is required to
use a modern web browser with good HTML5 and CSS3 support. Among others, cookies
and local storage are used. Also a decent JavaScript runtime must be provided by
the browser.
Supported and tested browsers are: Firefox 50+, Chrome 55+, Opera 42+ and Edge
13+. Mobile devices often have problems with internationalization and possibly
lack support for some common features of desktop browsers. In this stage of
development is not possible for us to fine tune the interface for major mobile
browsers on all mobile platforms. However, it is confirmed to work with latest
Google Chrome and Gello browser on Android 7.1+. Issues have been reported with
Firefox that will be fixed in the future. Also, it is confirmed to work with
Safari browser on iOS 10.
Usage of the web application is divided into sections concerning particular user
roles. Under these sections all possible use cases can be found. These sections
are inclusive, so more privileged users need to read instructions for all less
privileged users. Described roles are:
- Student
- Group supervisor
- Group administrator
- Instance administrator
- Superadministrator
## Terminology
**Instance** -- Represents a university, company or some other organization
unit. Multiple instances can exist in a single ReCodEx installation.
**Group** -- A group of students to which exercises are assigned by a
supervisor. It should typically correspond with a real world lab group.
**User** -- A person that interacts with the system using the web interface (or
an alternative client).
**Student** -- A user with least privileges who is subscribed to some groups and
submits solutions to exercise assignments.
**Supervisor** -- A person responsible for assigning exercises to a group and
reviewing submissions.
**Admin** -- A person responsible for the maintenance of the system and fixing
problems supervisors cannot solve.
**Exercise** -- An algorithmic problem that can be assigned to a group. They
can be shared by the teachers using an exercise database in ReCodEx.
**Assignment** -- An exercise assigned to a group, possibly with modifications.
**Runtime environment** -- Runtime environment is unique combination of platform
(OS) and programming language runtime/compiler in specific version. Runtime
environments are managed by the administrators to reflect abilities of whole
system.
**Hardware group** -- Hardware group is a set of workers with similar hardware.
Its purpose is to group workers that are likely to run a program using the same
amount of resources. Hardware groups are managed byt the system administrators
who have to keep them up-to-date.
## General Basics
Description of general basics which are the same for all users of ReCodEx web
application follows.
### First Steps in ReCodEx
You can create an account by clicking the "Create account" menu item in the left
sidebar. You can choose between two types of registration methods -- by creating
a local account with a specific password, or pairing your new account with an
existing CAS UK account.
If you decide to create a new local account using the "Create ReCodEx account”
form, you will have to provide your details and choose a password for your
account. Although ReCodEx allows using quite weak passwords, it is wise to use a
bit stronger ones The actual strength is shown in progress bar near the password
field during registration. You will later sign in using your email address as
your username and the password you select.
If you decide to use the CAS UK service, then ReCodEx will verify your CAS
credentials and create a new account based on information stored there (name and
email address). You can change your personal information later on the
"Settings" page.
Regardless of the desired account type, an instance it will belong to must be
selected. The instance will be most likely your university or other organization
you are a member of.
To log in, go to the homepage of ReCodEx and in the left sidebar choose the menu
item "Sign in". Then you must enter your credentials into one of the two forms
-- if you selected a password during registration, then you should sign with
your email and password in the first form called "Sign into ReCodEx". If you
registered using the Charles University Authentication Service (CAS), you should
put your student’s number and your CAS password into the second form called
"Sign into ReCodEx using CAS UK".
There are several options you can edit in your user account:
- changing your personal information (i.e., name)
- changing your credentials (email and password)
- updating your preferences (source code viewer/editor settings, default
language)
You can access the settings page through the "Settings" button right under
your name in the left sidebar.
If you are not active in ReCodEx for a whole day, you will be logged out
automatically. However, we recommend you sign out of the application after you
finish your interaction with it. The logout button is placed in the top section
of the left sidebar right under your name. You may need to expand the sidebar
with a button next to the "ReCodEx” title (informally known as _hamburger
button_), depending on your screen size.
### Forgotten Password
If you cannot remember your password and you do not use CAS UK authentication,
then you can reset your password. You will find a link saying "Cannot remember
what your password was? Reset your password." under the sign in form. After you
click this link, you will be asked to submit your registration email address. A
message with a link containing a special token will be sent to you by e-mail --
we make sure that the person who requested password resetting is really you.
When you visit the link, you will be able to enter a new password for your
account. The token is valid only for a couple of minutes, so do not forget to
reset the password as soon as possible, or you will have to request a new link
with a valid token.
If you sign in through CAS UK, then please follow the instructions
provided by the administrators of the service described on their
website.
### Dashboard
When you log into the system you should be redirected to your "Dashboard". On
this page you can see some brief information about the groups you are member of.
The information presented there varies with your role in the system -- further
description of dashboard will be provided later on with according roles.
## Student
Student is a default role for every newly registered user. This role has quite
limited capabilities in ReCodEx. Generally, a student can only submit solutions
of exercises in some particular groups. These groups should correspond to
courses he/she attends.
On the "Dashboard" page there is "Groups you are student of" section where you
can find list of your student groups. In first column of every row there is a
brief panel describing concerning group. There is name of the group and
percentage of gained points from course. If you have enough points to
successfully complete the course then this panel has green background with tick
sign. In the second column there is a list of assigned exercises with its
deadlines. If you want to quickly get to the groups page you might want to use
provided "Show group's detail" button.
### Join Group
To be able to submit solutions you have to be a member of the right group. Each
instance has its own group hierarchy, so you can choose only those within your
instance. That is why a list of groups is available from under an instance link
located in the sidebar. This link brings you to instance detail page.
In there you can see a description of the instance and most importantly in
"Groups hierarchy" box there is a hierarchical list of all public groups in the
instance. Please note that groups with plus sign are collapsible and can be
further extended. When you find a group you would like to join, continue by
clicking on "See group's page" link following with "Join group" link.
**Note:** Some groups can be marked as private and these groups are not visible
in hierarchy and membership cannot be established by students themselves.
Management of students in this type of groups is in the hands of supervisors.
On the group detail page there are multiple interesting things for you. The
first one is a brief overview containing the information describing the group,
there is a list of supervisors and also the hierarchy of the subgroups. The most
important section is the "Student's dashboard" section. This section contains
the list of assignments and the list of fellow students. If the supervisors of
the group allowed students to see the statistic of their fellow students then
there will also be the number of points each of the students has gained so far.
### Start Solving Assignments
In the "Assignments" box on the group detail page there is a list of assigned
exercises which students are supposed to solve. The assignments are displayed
with their names and deadlines. There are possibly two deadlines, the first one
means that till this datetime student will receive full amount of points in case
of successful solution. Second deadline does not have to be set, but in case it
is, the maximum number of points for successful solution between these two
deadlines can be different.
An assignment link will lead you to assignment detail page where are presented
all known details about assignment. There are of course both deadlines, limit of
submissions which you can make and also full-range description of assignment,
which can be localized. The localization can be on demand switched between all
language variants in tab like box.
Further on the page you can find "Submitted solutions" box where is a list of
submissions with links to result details. But most importantly there is a
"Submit new solution" button on the assignment page which provides an interface
to submit solution of the assignment.
After clicking on submit button, dialog window will show up. In here you can
upload files representing your solution, you can even add some notes to mark the
solution. Your supervisor can also access this note. After you successfully
upload all files necessary for your solution, click the "Submit your solution"
button and let ReCodEx evaluate the solution.
During the execution ReCodEx backend might send evaluation progress state to
your browser which will be displayed in another dialog window. When the whole
execution is finished then a "See the results" button will appear and you can
look at the results of your solution.
### View Results of Submission
On the results detail page there are a lot of information. Apart from assignment
description, which is not connected to your results, there is also the solution
submitter name (supervisor can submit a solution on your behalf), further there
are files which were uploaded on submission and most importantly "Evaluation
details" and "Test results" boxes.
Evaluation details contains overall results of your solution. There are
information such as whether the solution was provided before deadlines, if the
evaluation process successfully finished or if compilation succeeded. After that
you can find a lot of values, most important one is the last, "Total score",
consisting of your score, slash and the maximum number of points for this
assignment. Interestingly the your score value can be higher than the maximum,
which is caused by "Bonus points" item above. If your solution is nice and
supervisor notices it, he/she can assign you additional points for effort. On
the other hand, points can be also subtracted for bad coding habits or even
cheating.
In test results box there is a table of all exercise tests results. Columns
represents these information:
- test case overall result, symbol of yes/no option
- test case name
- percentage of correctness of this particular test
- evaluation status, if test was successfully executed or failed
- memory limit, if supervisor allowed it then percentual memory usage is
displayed
- time limit, if supervisor allowed it then percentual time usage is displayed
A new feature of web application is "Comments and notes" box where you can
communicate with your supervisors or just write random private notes to your
submission. Adding a note is quite simple, you just write it to text field in
the bottom of box and click on the "Send" button. The button with lock image
underneath can switch visibility of newly created comments.
In case you think the ReCodEx evaluation of your solution is wrong, please use
the comments system described above, or even better notify your supervisor by
another channel (email). Unfortunately there is currently no notification
mechanism for new comment messages.
## Group Supervisor
Group supervisor is typically the lecturer of a course. A user in this role can
modify group description and properties, assign exercises or manage list of
students. Further permissions like managing subgroups or supervisors is
available only for group administrators.
On "Dashboard" page you can find "Groups you supervise" section. Here there are
boxes representing your groups with the list of students attending course and
their points. Student names are clickable with redirection to the profile of the
user where further information about his/hers assignments and solution can be
found. To quickly jump onto groups page, use "Show group's detail" button at
the bottom of the matching group box.
### Manage Group
Locate group you supervise and you want to manage. All your supervised groups
are available in sidebar under "Groups -- supervisor" collapsible menu. If you
click on one of those you will be redirected to group detail page. In addition
to basic group information you can also see "Supervisor's controls" section. In
this section there are lists of current students and assignments.
As a supervisor of group you are able to see "Edit group settings" button
at the top of the page. Following this link will take you to group editation
page with form containing these fields:
- group name which is visible to other users
- external identification which may be used for pairing with entries in an
information system
- description of group which will be available to users in instance (in
Markdown)
- set if group is publicly visible (and joinable by students) or private
- options to set if students should be able see statistics of each other
- minimal points threshold which students have to gain to successfully complete
the course
After filling all necessary fields the form can be sent by clicking on "Edit
group" button and all changes will be applied.
For students management there are "Students" and "Add student" boxes. The first
one is simple list of all students which are attending the course with the
possibility of delete them from the group. That can be done by hitting "Leave
group" button near particular user. The second box is for adding students to the
group. There is a text field for typing name of the student and after clicking
on the magnifier image or pressing enter key there will appear list of matched
users. At this moment just click on the "Join group" button and student will be
signed in to your group.
### Assigning Exercises
Before assigning an exercise, you obviously have to know what exercises are
available. A list of all exercises in the system can be found under "Exercises"
link in sidebar. This page contains a table with exercises names, difficulties
and names of the exercise authors. Further information about exercise is
available by clicking on its name.
On the exercise details page are numerous information about it. There is a box
with all possible localized descriptions and also a box with some additional
information of exercise author, its difficulty, version, etc. There is also a
description for supervisors by exercise author under "Exercise overview" option,
where some important information can be found. And most notably there is an
information about available programming languages for this exercise, under
"Supported runtime environments" section.
If you decide that the exercise is suitable for one of your groups, look for the
"Groups" box at the bottom of the page. There is a list of all groups you
supervise with an "Assign" button which will assign the exercise to the
selected group.
After clicking on the "Assign" button you should be redirected to assignment
editation page. In there you can find two forms, one for editation of assignment
meta information and the second one for setting exercise time and memory limits.
In meta information form you can fill these options:
- name of the assignment which will be visible in a group
- visibility (if an assignment is under construction then you can mark it as not
visible and students will not see it)
- subform for localized descriptions (new localization can be added by clicking
on "Add language variant" button, current one can be deleted with "Remove this
language" button)
- language of description from dropdown field (English, Czech, German)
- description in selected language
- score configuration which will be used on students solution evaluation, you
can find some very simple one already in here, description of score
configuration can be found further in "Writing score configuration" chapter
- first submission deadline
- maximum points that can be gained before the first deadline; if you want to
manage all points manually, set it to 0 and then use bonus points, which are
described in the next subchapter
- second submission deadline, after that students still can submit exercises but
they are given no points no points (must be after the first deadline)
- maximum points that can be gained between first deadline and second deadline
- submission count limit for the solutions of the students -- this limits the amount of
attempts a student has at solving the problem
- visibility of memory and time ratios; if true students can see the percentage
of used memory and time (with respect to the limit) for each test
- minimum percentage of points which each submission must gain to be considered
correct (if it gets less, it will gain no points)
- whether the assignment is marked as bonus one and points from solving it are
not included into group threshold limit (that means solving it can get you
additional points over the limit)
The form has to be submitted with "Edit settings" button otherwise changes will
not be saved.
The same editation page serves also for the purpose of assignment editation, not
only creation. That is why on bottom of the page "Delete the assignment" box
can be found. Clearly the button "Delete" in there can be used to unassign
exercise from group.
The last unexplored area is the time and memory limits form. The whole form is
situated in a box with tabs which are leading to particular runtime
environments. If you wish not to use one of those, locate "Remove" button at the
bottom of the box tab which will delete this environment from the assignment.
Please note that this action is irreversible.
In general, every tab in environments box contains some basic information about
runtime environment and another nested tabbed box. In there you can find all
hardware groups which are available for the exercise and set limits for all test
cases. The time limits have to be filled in seconds (float), memory limits are
in bytes (int). If you are interested in some reference values to particular
test case then you can take a peek on collapsible "Reference solutions'
evaluations" items. If you are satisfied with changes you made to the limits,
save the form with "Change limits" button right under environments box.
### Management of The Solutions of The Students
One of the most important tasks of a group supervisor is checking student
solutions. As automatic evaluation of them cannot catch all problems in the
source code, it is advisable to do a brief manual review of the coding
style of the student and reflect that in assignment bonus points.
On "Assignment detail" page there is an "View student results" button near top
of the page (next to "Edit assignment settings" button). This will redirect you
to a page where is a list of boxes, one box per student. Each student box
contains a list of submissions for this assignment. The row structure of
submission list is the same as the structure in the "Submitted solution"
box. More information about every solution can be shown by clicking on "Show
details" link on the end of solution row.
This page is the same as for students with one exception -- there is an
additional collapsed box "Set bonus points". In unfolded state, there is an
input field for one number (positive or negative integer) and confirmation
button "Set bonus points". After filling intended amount of points and
submitting the form, the data in "Evaluation details" box get immediately
updated. To remove assigned bonus points, submit just the zero number. The bonus
points are not additive, newer value overrides older values.
It is useful to give a feedback about the solution back to the user. For this
you can use the "Comments and notes" box. Make sure that the messages are not
private, so that the student can see them. More detailed description of this box
can be nicely used the "Comments and notes" box. Make sure that the messages are
not private, so the student can see them. More detailed description of this box
is available in student part of user documentation.
One of the discussed concept was marking one solution as accepted. However, due
to lack of frontend developers it is not yet prepared in user interface. We
hope, it will be ready as soon as possible. The button for accepting a solution
will be most probably also on this page.
### Creating Exercises
Link to exercise creation can be found in exercises list which is accessible
through "Exercises" link in sidebar. On the bottom of the exercises list page
you can find "Add exercise" button which will redirect you to exercise editation
page. In this moment exercise is already created so if you just leave this page
exercise will stay in the database. This is also reason why exercise creation
form is the same as the exercise editation form.
Exercise editation page is divided into three separate forms. First one is
supposed to contain meta information about exercise, second one is used for
uploading and management of supplementary files and third one manages runtime
configuration in which exercise can be executed.
First form is located in "Edit exercise settings" and generally contains meta
information needed by frontend which are somehow somewhere visible. In here you
can define:
- exercise name which will be visible to other supervisors
- difficulty of exercise (easy, medium, hard)
- description which will be available only for visitors, may be used for further
description of exercise (for example information about test cases and how they
could be scored)
- private/public switch, if exercise is private then only you as author can see
it, assign it or modify it
- subform containing localized descriptions of exercise, new one can be added
with "Add language variant" button and current one deleted with "Remove this
language"
- language in which this particular description is in (Czech, English,
German)
- actual localized description of exercise
After all information is properly set form has to be submitted with "Edit
settings" button.
Management of supplementary files can be found in "Supplementary files" box.
Supplementary files are files which you can use further in job configurations
which have to be provided in all runtime configurations. These files are
uploaded directly to fileserver from where worker can download them and use
during execution according to job configuration.
Files can be uploaded either by drag and drop mechanism or by standard "Add a
file" button. In opened dialog window choose file which should be uploaded. All
chosen files are immediately uploaded to server but to save supplementary files
list you have to hit "Save supplementary files" button. All previously uploaded
files are visible right under drag and drop area, please note that files are
stored on fileserver and cannot be deleted after upload.
The last form on exercise editation page is runtime configurations editation
form. Exercise can have multiple runtime configurations according to the number
of programming languages in which it can be run. Every runtime configuration
corresponds to one programming language because all of them has to have a bit
different job configuration.
New runtime configuration can be added with "Add new runtime configuration"
button this will spawn new tab in runtime configurations box. In here you can
fill following:
- human readable identifier of runtime configuration
- runtime environment which corresponds to programming language
- job configuration in YAML, detailed description of job configuration can be
found further in this chapter in "Writing job configuration" section
If you are done with changes to runtime configurations save form with "Change
runtime configurations" button. If you want to delete some particular runtime
just hit "Remove" button in the right tab, please note that after this operation
runtime configurations form has to be again saved to apply changes.
All runtime configurations which were added to exercise will be visible to
supervisors and all can be used in assignment, so please be sure that all of the
languages and job configurations are working.
If you choose to delete exercise, at the bottom of the exercise editation page
you can find "Delete the exercise" box where "Delete" button is located. By
clicking on it exercise will be delete from the exercises list and will no
longer be available.
### Reference Solutions of An Exercise
Each exercise should have a set of reference solutions, which are used to tune
time and memory limits of assignments. Values of used time and memory for each
solution are displayed in yellow boxes under forms for setting assignment limits
as described earlier.
However, there is currently no user interface to upload and evaluate reference
solutions. It is possible to use direct REST API calls, but it is not much user
friendly. If you are interested, please look at [API
documentation](https://recodex.github.io/api/), notably sections
_Uploaded-Files_ and _Reference-Exercise-Solutions_. You need to upload the
reference solution files, create a new reference solution and then evaluate the
solution. After that, measured data will be available in the box at assignment
editing page (setting limits section).
We are now working on a better user interface, which will be available soon.
Then its description will be added here.
## Group Administrator
Group administrator is the group supervisor with some additional permissions in
particular group. Namely group administrator is capable of creating a subgroups
in managed group and also adding and deleting supervisors. Administrator of the
particular group can be only one person.
### Creating Subgroups And Managing Supervisors
There is no special link which will get you to groups in which you are
administrator. So you have to get there through "Groups - supervisor" link in
sidebar and choose the right group detail page. If you are there you can see
"Administrator controls" section, here you can either add supervisor to group or
create new subgroup.
Form for creating a subgroup is present right on the group detail page in "Add
subgroup" box. Group can be created with following options:
- name which will be visible in group hierarchy
- external identification, can be for instance ID of group from school system
- some brief description about group
- allow or deny users to see each others statistics from assignments
After filling all the information a group can be created by clicking on "Create
new group" button. If creation is successful then the group is visible in
"Groups hierarchy" box on the top of page. All information filled during
creation can be later modified.
Adding a supervisor to a group is rather easy, on group detail page is an "Add
supervisor" box which contains text field. In there you can type name or
username of any user from system. After filling user name, click on the
magnifier image or press the enter key and all suitable users are searched. If
your chosen supervisor is in the updated list then just click on the "Make
supervisor" button and new supervisor should be successfully set.
Also, existing supervisor can be removed from the group. On the group detail
page there is "Supervisors" box in which all supervisors of the group are
visible. If you are the group administrator, you can see there "Remove
supervisor" buttons right next to supervisors names. After clicking on it some
particular supervisor should not to be supervisor of the group anymore.
## Instance Administrator
Instance administrator can be only one person per instance. In addition to
previous roles this administrator should be able to modify the instance details,
manage licences and take care of top level groups which belong to the instance.
### Instance Management
List of all instances in the system can be found under "Instances" link in the
sidebar. On that page there is a table of instances with their respective
admins. If you are one of them, you can visit its page by clicking on the
instance name. On the instance details page you can find a description of the
instance, current groups hierarchy and a form for creating a new group.
If you want to change some of the instance settings, follow "Edit instance" link
on the instance details page. This will take you to the instance editation page
with corresponding form. In there you can fill following information:
- name of the instance which will be visible to every other user
- brief description of instance and for whom it is intended
- checkbox if instance is open or not which means public or private (hidden from
potential users)
If you are done with your editation, save filled information by clicking on
"Update instance" button.
If you go back to the instance details page you can find there a "Create new
group" box which is able to add a group to the instance. This form is the same
as the one for creating subgroup in already existing group so we can skip
description of the form fields. After successful creation of the group it will
appear in "Groups hierarchy" box at the top of the page.
### Licences
On the instance details page, there is a box "Licences". On the first line, it
shows it this instance has currently valid licence or not. Then, there are
multiple lines with all licences assigned to this instance. Each line consists
of a note, validity status (if it is valid or revoked by superadministrator) and
the last date of licence validity.
A box "Add new licence" is used for creating new licences. Required fields are
the note and the last day of validity. It is not possible to extend licence
lifetime, a new one should be generated instead. It is possible to have more
than one valid licence at a time. Currently there is no user interface for
revoking licences, this is done manually by superadministrator. If an instance
is to be disabled, all valid licences have to be revoked.
## Superadministrator
Superadministrator is a user with the most privileges and as such superadmin
should be quite a unique role. Ideally, there should be only one user of this
kind, used with special caution and adequate security. With this stated it is
obvious that superadmin can perform any action the API is capable of.
### Management of Users
There are only a few user roles in ReCodEx. Basically there are only three:
_student_, _supervisor_, and _superadmin_. Base role is student which is
assigned to every registered user. Roles are stored in database alongside other
information about user. One user always has only one role at the time. At first
startup of ReCodEx, the administrator has to change the role for his/her account
manually in the database. After that manual intervention into database should
never be needed.
There is a little catch in groups and instances management. Groups can have
admins and supervisors. This setting is valid only per one particular group and
has to be separated from basic role system. This implies that supervisor in one
group can be student in another and simultaneously have global supervisor role.
Changing role from student to supervisor and back is done automatically when the
new privileges are granted to the user, so managing roles by hand in database is
not needed. Previously stated information can be applied to instances as well,
but instances can only have admins.
Roles description:
- Student -- Default role which is used for newly created accounts. Student can
join or leave public groups and submit solutions of assigned exercises.
- Supervisor -- Inherits all permissions from student role. Can manage groups to
which he/she belongs to. Supervisor can also view and change groups details,
manage assigned exercises, view students in group and their solutions for
assigned exercises. On top of that supervisor can create/delete groups too,
but only as subgroup of groups he/she belongs to.
- Superadmin -- Inherits all permissions from supervisor role. Most powerful
user in ReCodEx who should be able to do access any functionality provided by
the application.
## Writing Score Configuration
An important thing about assignment is how to assign points to particular
solutions. As mentioned previously, the whole job is composed of logical tests.
All of these tests have to contain one essential "evaluation" task. Evaluation
task should output one float number which can be further used for scoring of
particular tests.
Total resulting score of the students solution is then calculated according to a
supplied score config (described below) and using specified calculator. Total
score is also a float between 0 and 1. This number is then multiplied by the
maximum of points awarded for the assignment by the teacher assigning the
exercise -- not the exercise author.
For now, there is only one way how to write score configuration using only
simple score calculator. But the implementation in API is agile enough to handle
upcoming score calculators which might use some more complex scoring algorithms.
This also means that future calculators do not have to use the YAML format for
configuration. In fact, the configuration can be a string in any format.
### Simple Score Calculation
First implemented calculator is simple score calculator with test weights. This
calculator just looks at the score of each test and put them together according
to the test weights specified in assignment configuration. Resulting score is
calculated as a sum of products of score and weight of each test divided by the
sum of all weights. The algorithm in Python would look something like this:
```
sum = 0
weightSum = 0
for t in tests:
sum += t.score * t.weight
weightSum += t.weight
score = sum / weightSum
```
Sample score config in YAML format:
```{.yml}
testWeights:
a: 300 # test with id 'a' has a weight of 300
b: 200
c: 100
d: 100
```
## Writing Job Configuration
To run and evaluate an exercise the backend needs to know the steps how to do
that. This is different for each environment (operation system, programming
language, etc.), so each of the environments needs to have separate
configuration.
Backend works with a powerful, but quite low level description of simple
connected tasks written in YAML syntax. More about the syntax and general task
overview can be found on [separate
page](https://github.com/ReCodEx/wiki/wiki/Assignments). One of the planned
features was user friendly configuration editor, but due to tight deadline and
team composition it did not make it to the first release. However, writing
configuration in the basic format will be always available and allows users to
use the full expressive power of the system.
This section walks through creation of job configuration for _hello world_
exercise. The goal is to compile file _source.c_ and check if it prints `Hello
World!` to the standard output. This is the only test case named **A**.
The problem can be split into several tasks:
- compile _source.c_ into _helloworld_ with `/usr/bin/gcc`
- run _helloworld_ and save standard output into _out.txt_
- fetch predefined output (suppose it is already uploaded to fileserver) with
hash `a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b` to _reference.txt_
- compare _out.txt_ and _reference.txt_ by `/usr/bin/diff`
The absolute path of tools can be obtained from system administrator. However,
`/usr/bin/gcc` is location, where the GCC binary is available almost everywhere,
so location of some tools can be (professionally) guessed.
First, write header of the job to the configuration file.
```{.yml}
submission:
job-id: hello-word-job
hw-groups:
- group1
```
Basically it means, that the job _hello-world-job_ needs to be run on workers
that belong to the `group_1` hardware group . Reference files are downloaded
from the default location configured in API (such as
`http://localhost:9999/exercises`) if not stated explicitly otherwise. Job
execution log will not be saved to result archive.
Next the tasks have to be constructed under _tasks_ section. In this demo job,
every task depends only on previous one. The first task has input file
_source.c_ (if submitted by user) already available in working directory, so
just call the GCC. Compilation is run in sandbox as any other external program
and should have relaxed time and memory limits. In this scenario, worker
defaults are used. If compilation fails, the whole job is immediately terminated
(because the _fatal-failure_ bit is set). Because _bound-directories_ option in
sandbox limits section is mostly shared between all tasks, it can be set in
worker configuration instead of job configuration (suppose this for following
tasks). For configuration of workers please contact your administrator.
```{.yml}
- task-id: "compilation"
type: "initiation"
fatal-failure: true
cmd:
bin: "/usr/bin/gcc"
args:
- "source.c"
- "-o"
- "helloworld"
sandbox:
name: "isolate"
limits:
- hw-group-id: group1
chdir: ${EVAL_DIR}
bound-directories:
- src: ${SOURCE_DIR}
dst: ${EVAL_DIR}
mode: RW
```
The compiled program is executed with time and memory limit set and the standard
output is redirected to a file. This task depends on _compilation_ task, because
the program cannot be executed without being compiled first. It is important to
mark this task with _execution_ type, so exceeded limits will be reported in
frontend.
Time and memory limits set directly for a task have higher priority than worker
defaults. One important constraint is, that these limits cannot exceed limits
set by workers. Worker defaults are present as a safety measure so that a
malformed job configuration cannot block the worker forever. Worker default
limits should be reasonably high, like a gigabyte of memory and several hours of
execution time. For exact numbers please contact your administrator.
It is important to know that if the output of a program (both standard and
error) is redirected to a file, the sandbox disk quotas apply to that file, as
well as the files created directly by the program. In case the outputs are
ignored, they are redirected to `/dev/null`, which means there is no limit on
the output length (as long as the printing fits in the time limit).
```{.yml}
- task-id: "execution_1"
test-id: "A"
type: "execution"
dependencies:
- compilation
cmd:
bin: "helloworld"
sandbox:
name: "isolate"
stdout: ${EVAL_DIR}/out.txt
limits:
- hw-group-id: group1
chdir: ${EVAL_DIR}
bound-directories:
- src: ${SOURCE_DIR}
dst: ${EVAL_DIR}
mode: RW
time: 0.5
memory: 8192
```
Fetch sample solution from file server. Base URL of file server is in the header
of the job configuration, so only the name of required file (its `sha1sum` in
our case) is necessary.
```{.yml}
- task-id: "fetch_solution_1"
test-id: "A"
dependencies:
- execution
cmd:
bin: "fetch"
args:
- "a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b"
- "${SOURCE_DIR}/reference.txt"
```
Comparison of results is quite straightforward. It is important to set the task
type to _evaluation_, so that the return code is set to 0 if the program is
correct and 1 otherwise. We do not set our own limits, so the default limits are
used.
```{.yml}
- task-id: "judge_1"
test-id: "A"
type: "evaluation"
dependencies:
- fetch_solution_1
cmd:
bin: "/usr/bin/diff"
args:
- "out.txt"
- "reference.txt"
sandbox:
name: "isolate"
limits:
- hw-group-id: group1
chdir: ${EVAL_DIR}
bound-directories:
- src: ${SOURCE_DIR}
dst: ${EVAL_DIR}
mode: RW
```
<!---
// vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
-->

90
Web-API.md
Unescape Escape View File

@ -1,90 +0,0 @@
# Web API
## Description
The Web API provides a controlled access to the evaluation backend. It also
enables the use of different user frontends such as default web application,
mobile applications, commandline tools and possibly others. The communication
goes as HTTP(S) requests in predefined format, nowadays mostly known as
[REST](https://en.wikipedia.org/wiki/Representational_state_transfer) format.
Results from the API are in plain text in JSON format to be easily parsed in
various languages (notably JavaScript).
This component must be publicly visible on the internet, so it is important to
care about security and follow our recommendations. Security and user access
restriction are among our primary concerns, so proper roles with permission
separation are introduced and maintained. Also some additional checks are made
directly in the code, so that a user cannot access information which are out of
their authorization.
## Architecture
Web API is written in PHP using [Nette framework](https://nette.org/en/). This framework provides useful components like _Tracy_ for logging and showing errors, _Tester_ for productive unit testing or _Latte_ templating engine. Nette is modern, widely used and great performing software with active developers and user community. Nette can help eliminate security holes, simplify debugging and make coding easier with numerous plugins and extensions. Also, it is published under permissive BSD license.
API architecture consists of several parts:
- **router** -- component handling mapping from URL addresses to methods in presenter classes (called endpoints)
- **presenters** -- classes containing one method per endpoint responsible for
fetching and parsing request arguments and performing desired actions
- **entities** -- classes persisted using a database with an ORM framework
- **repositories** -- common operations on entities of one type, mostly finding entity by identifier or persisting changes to the database
- **helpers** -- set of classes solving more complicated internal logic, used from presenters to keep them reasonably small
Each presenter method has several annotations. They are used for generating REST API documentation in [Swagger](http://swagger.io/), specifying request type and its parameters and specifying one level of access restrictions. Also, there is simple description of the endpoint.
For specifying the request type (_GET_, _POST_, _DELETE_) annotations with exactly these names without any parameters are used. To describe request parameters `@Param` annotation is used with following arguments:
- type -- the type of argument, one of _post_ of _query_
- name -- name of the argument (the key)
- validation -- validation of the value, see [Nette validation rules](https://doc.nette.org/en/2.4/validators#toc-rules)
- msg -- description for users about the values this parameter can contain
- required -- specifies if this option is mandatory (`true`, default) or optional (`false`)
- description -- description for documentation of the API
Another annotation is `@LoggedIn` which takes no arguments. It can be placed before a whole class or before a method, so requests from unauthorized users are forbidden. Permissions can be granted or prohibitted by `@UserIsAllowed` annotation. This one is only per method and takes one argument in `key="value"` format. The value specifies which action (_value_) of a resource (_key_) the user needs to be allowed to perform this request. An example of how an annotated endpoint can look like:
```{.php}
/**
* Create a user account
* @POST
* @LoggedIn
* @UserIsAllowed(users="create")
* @Param(type="post", name="email", validation="email",
* description="An email that will serve as a login name")
* @Param(type="post", name="name", validation="string:2..",
* description="First name")
* @Param(type="post", name="password", validation="string:1..",
* msg="Password cannot be empty.",
* description="A password for authentication")
*/
public function actionCreateAccount() {
...
}
```
The [Doctrine](https://github.com/Kdyby/Doctrine)
[ORM](https://en.wikipedia.org/wiki/Object-relational_mapping) framework is used
as an object persistence layer. It provides simple to use annotations to specify
columns of database tables including types, indexes and also it is possible to
make mapping between entities. For detailed info refer to [official
documentation](http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/).
The API is capable of sending email messages. They can inform an administrator about errors and users about submission evaluation or a temporary link to change forgotten password. The [Nette Mail](https://doc.nette.org/en/2.4/mailing) extension provides nice interface for sending messages through external SMTP server (**preferred**) or builtin PHP function `mail`. It is important to set up the mailserver properly to ensure message delivery to the clients. The messages are rendered in HTML format via simple _Latte_ templates.
### Authentication
Instead of relying on PHP sessions, we decided to use an authentication flow
based on JWT tokens (RFC 7519). On successful login, the user is issued an
access token that they have to send with subsequent requests using the HTTP
Authorization header (`Authorization: Bearer <token>`). The token has a limited
validity period and has to be renewed periodically using a dedicated API
endpoint.
To implement this behavior in Nette framework, a new `IUserStorage`
implementation was created, along with an `IIdentity` and authenticators for
both our internal login service and CAS.
An advantage of this approach is being able control the authentication process
completely instead of just receiving session data through a global variable.

155
Web-application.md
Unescape Escape View File

@ -1,155 +0,0 @@
# Web application
## Description
The *WebApp* is the default user interface of the ReCodEx project. It is a Single Page Application (SPA) which users access through their web browsers.
This user interface allows students to join *groups* corresponding to their school classes, submit new *solutions* and see the results as soon as the solution is evaluated by the backend.
Teachers can manage their *groups* by adding and removing *students*, assigning *exercises* and examining the *automatically evaluated solutions* of their students and their total score gained throughout the semester.
The actions performed by the user are sent to the *API server* over the HTTPS protocol and the web application does not modify the database or access files on their server directly and all the inputs sent
from the web application are additionally *validated* on the *API server*.
The *WebApp* is written in *ECMAScript* (often referred to as JavaScript) and HTML5. The [React](https://facebook.github.io/react) library developed by Facebook is used to manage the user interface and the [Redux](http://redux.js.org/) framework
created by Dan Abramov is used to manage the application state. Both of these frameworks are industry standards used in production by many companies worldwide. Many other smaller libraries and components are used, notably the [Immutable.js](https://facebook.github.io/immutable-js) and [Format.js](http://formatjs.io/) by Yahoo via *redact-intl* integration library.
### Multiplatform approach
Since the runtime environment of *WebApp* is the user's web browser it does not need any installation on the enduser's device and does not require any particular platform or non-standard software installed on the user's device.
This application is designed and implemented in a way which should be suitable for modern web browsers on both desktop and mobile devices.
## Architecture
The whole project is written using the next generation of JavaScript referred to as *ECMAScript 6* (also known as *ES6*, *ES.next*, or *Harmony*). Since not all the features introduced in this standard are implemented in today's modern web browsers (like classes and spread operators) and hardly any are implemented in older versions of web browsers which are currently still in use, the source code is transpiled into the older standard *ES5* using [Babel.js](https://babeljs.io/) transpiler and bundled into a single script file using the [webpack](https://webpack.github.io/) moudle bundler. The need for a transpiler also arises from the usage of the *JSX* syntax for declaring React components. To read more about these these tools and their usage please refer to the [installation section](#Installation). The whole bundling process takes place at deployment and is not repeated afterwards.
### Routing and server side rendering
The routing mechanism uses the [react-router](https://github.com/ReactTraining/react-router) and [react-router-redux](https://github.com/reactjs/react-router-redux) libraries. The routing mechanism works both on the server and the client.
The routes are defined in the `/src/pages/routes.js` file.
### Server rendering and the 'loadAsync' convention
The server matches the URL of the request with the router and pre-renders the contents of the page. The components, which are selected by the router based on the URL, are given the parameters and are attached to the component tree.
The server script examines all the components matched with the route and looks for a static function called `loadAsync` and if it is found, it will be executed with the params extracted from the URL, the store's dispatch method and the ID of the logged in user if he is logged in. It is expected that this function will return an instance of `Promise`. Server then waits until all the promises are resolved and only after then it goes on and sends a HTTP response to the client.
### Client
Each time the user clicks on a link to a different location of the application the page is not reloaded but the router rather changes the location of the browser to the new URL and replaces the components in the component tree with the ones matched against the new URL the same way the server did. Components should call the `loadAsync` methods themselves in their lifecycle methods.
### Redux middleware and helpers
The redux [store](https://github.com/ReCodEx/web-app/blob/master/src/redux/store.js) is extended with several middlewares - [redux-thunk](https://github.com/gaearon/redux-thunk), [redux-promise-middleware](https://github.com/pburtchaell/redux-promise-middleware), router middleware, accessTokenMiddleware, and apiMiddleware.
#### apiMiddleware
The API middleware catches all the `CALL_API` actions and turns them into a HTTP request via the [fetch API](https://developer.mozilla.org/en/docs/Web/API/Fetch_API). The middleware deeply converts data in the *body* part of the request to `FormData` and sets correctly the headers and creates endpoints for the API server before each request is sent. The middleware also handle the responses - converts them to the format expected by the reducers and catches requests' failures.
#### accessTokenMiddleware
[Access token middleware](https://github.com/ReCodEx/web-app/blob/master/src/redux/middleware/accessTokenMiddleware.js) stores and keeps the access token once the user authenticates and receives a token from the API server. The middleware listens for dispatched API actions and inserts the token to the request automatically.
#### Modules
Modules (sometimes called "ducks") contain a reducer and related action types and action creators. There are many separate reducers and actions for each individual resource and task to make the store tree as flat as possible. Vast majority of the modules is derived from the *resource manager* and are very similar to each other. Modules are placed in the `/src/redux/modules` directory.
#### Selectors
Selectors are used from the *containers* to return a specific information from the global redux state. It would be very inconvenient if the containers needed to know the whole structure of the state, so they can rely on the functions provided by the selectors for each module. The [reselect](https://github.com/reactjs/reselect) library is used for creating efficient selectors. Selectors are placed in the `/src/redux/selectors` directory.
#### Resource manager
The [resource manager](https://github.com/ReCodEx/web-app/blob/master/src/redux/helpers/resourceManager/index.js) is a set of factory functions which was created to extract the repeating functionality of obtaining and managing the loading phases of a resource or multiple resources at once.
The factory methods create the reducer, action types, and action creators. It strongly relies on the *redux-promise-middleware*. It was inspired by the [Reddit API example](http://redux.js.org/docs/advanced/ExampleRedditAPI.html) from the Redux library official website.
Resource manager is placed in the `/src/redux/helpers/resourceManager` directory.
### Declarative UI components
The implementation of *WebApp* is split across more than a hundred small components which are composed together. There are two basic types of React components - presentational and stateful components.
- **Presentational components** are in fact pure functions which yield their output based solely on the input *props* and application's *context* (e. g. current language has effect on URLs). These components are preferred and should be as simple as possible. These components are placed in the subfolders of `src/components` directory.
- **Stateful components** are *connected* to the *redux store* and yield information based on the either the *props* and *context*, but also on the current state of the application. These components are more complicated and often make use of React's components' life cycle methods. *Stateful components* are often referred to as *containers* and the source code files are placed in a separate directories called `src/containers` and `src/pages`.
- There is no technical difference between *containers* and *pages*. *Pages* are used as "root" components for different pages and are registered in the app's [router](https://github.com/ReactTraining/react-router).
As was mentioned earlier in the text, there is over a hundred components in the whole application and most of them are very simple. Some of the most important and widely used components are described in the following paragraphs.
#### App
[App](https://github.com/ReCodEx/web-app/blob/master/src/containers/App/App.js) container is meant to be the root of the *react-router* tree. It checks whether a user is logged in and loads his profile and settings from the API. App container also handles access token refreshing.
#### LayoutContainer
[LayoutContainer](https://github.com/ReCodEx/web-app/blob/master/src/containers/LayoutContainer/LayoutContainer.js) handles the dependency injection of the localized links based on the current language stated in the URL. It also controls the state of the sidebar - collapsing and showing the sidebar.
##### Layout, Header, Sidebar, Footer
The [Layout](https://github.com/ReCodEx/web-app/blob/master/src/components/Layout/Layout.js), [Header](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Header/Header.js), [Header](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Footer/Footer.js), and [Sidebar](https://github.com/ReCodEx/web-app/blob/master/src/components/Sidebar/Sidebar/Sidebar.js) components hold the main HTML structure of the page which is displayed to the user and hold links to different parts of the application.
#### PageContent
The [PageContent](https://github.com/ReCodEx/web-app/blob/master/src/components/PageContent/PageContent.js) component holds the main content of a page with the common structure for all pages - the *title*, *description*, *[breadcrumbs](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Breadcrumbs/Breadcrumbs.js),*, *content*. The component passes the title and description to the [Helmet](https://github.com/nfl/react-helmet) library which reflects these into the `<head>` section of the HTML document.
#### ResourceRenderer
[ResourceRenderer](https://github.com/ReCodEx/web-app/blob/master/src/components/ResourceRenderer/ResourceRenderer.js) component is given a *resource* managed by the *resourceManager* as a *prop* and displays different content based on the state of the given *resource* - still loading, loading failed, fully loaded.
Passing content for the *loading* and *failed* states though *props* is optional; however, the content for the *loaded* state is required and must be passed as a child to the `ResourceManager`.
Multiple resources can be passed as an array to the component and it will wait in the *loading* state until some of the resources are still loading. If one of the resources fails to load the component will switch to the *failed* state. When all the files are fully loaded then the component displays the content for the *loaded* state.
It is worth mentioning that the component is completely stateless and "switching of the states" is the effect of several renderings of the component over time.
#### Page
[Page](https://github.com/ReCodEx/web-app/blob/master/src/components/Page/Page.js) combines the two previously described components, since they are often used together for displaying a page content which is dependant on a resource.
#### Pages
The main pages of the application are placed in the `/src/pages/` directory. These pages are referenced from *the router* and are responsible mainly for initiating loading of the main resources when the component will be mounted to the DOM.
#### EvaluationProgressContainer
#### CommentThreadContainer
[CommentThreadContainer](https://github.com/ReCodEx/web-app/blob/master/src/containers/CommentThreadContainer/CommentThreadContainer.js) can be used on any page of the *webapp* to create a discussion thread. A unique identification must be assigned to every thread so the comments in the distinct threads do not interleave.
#### Bootstrap and AdminLTE theme
The UI of the application is built using the stylesheets from the [Bootstrap framework](http://getbootstrap.com/) from Twitter and the [AdminLTE](https://almsaeedstudio.com/) theme from Abdullah Almsaeed. None of the JavaScript plugins from these libraries is used.
A package [react-bootstrap](https://react-bootstrap.github.io/) is used. Components specific to the *AdminLTE* theme are implemented and stored in the `src/components/AdminLTE` folder and its subfolders.
##### Box, FormBox
[Box](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Box/Box.js) is a frequently used component for bounding other components like text paragraphs or tables inside. It is in fact a re-styled `Panel` component from Bootstrap. It can be collapsable and can be displayed in different colors and types. [FormBox](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/FormBox/FormBox.js) is just a wrapper of `Box` adjusted for holding forms inside of the body.
##### CommentThread
[CommentThread](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/CommentThread/CommentThread.js) component is used directly by the earlier described `CommentThreadContainer` and uses the HTML markup introduced by the theme author to achieve nice comment threads.
##### Avatar, LoadingAvatar, FailedAvatar
[Avatar](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Avatar/Avatar.js) component is used on many places for displaying a round profile picture of a user. `LoadingAvatar` and `FailedAvatar` are used to mock the visual appearance of the avatar while the image is being downloaded or if the download failed for some reason.
#### Icons
The free [FontAwesome](http://fontawesome.io/) icon pack is used in the application via the [react-fontawesome](https://github.com/danawoodman/react-fontawesome) library. Many types of [icons](https://github.com/ReCodEx/web-app/blob/master/src/components/AdminLTE/Header/Header.js) are defined as components (e. g. `LoadingIcon`, `WarningIcon`, `SuccessIcon`, `DeleteIcon`, and many others) to be used throughout the application so the same symbols are used for the same purposes.
#### Forms
The [redux-form](http://redux-form.com/) library is used for the management of forms' states. Several of the implemented components can be used as form fields, namely the [CheckboxField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/CheckboxField.js), [MarkdownTextAreaField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/MarkdownTextAreaField.js), [SourceCodeField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/SourceCodeField.js), [DatetimeField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/DatetimeField.js), [TabbedArrayField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/TabbedArrayField.js), [LanguageSelectField](https://github.com/ReCodEx/web-app/blob/master/src/components/Forms/Fields/LanguageSelectField.js), and a few others.
### Localization and globalization
The whole application is prepared for localization. All the text can be exported from the user interface and translated into several languages. The numbers, dates, and time values are also formatted with respect to the selected language. The [react-intl](https://github.com/yahoo/react-intl) and [Moment.js](http://momentjs.com/) libraries are used to achieve this.
All the strings can be extracted from the application using a command:
```
$ npm run exportStrings
```
This will create *JSON* files with the exported strings for the *'en'* and *'cs'* locale. If you want to export strings for more languages, you must edit the `/manageTranslations.js` script. The exported strings are placed in the `/src/locales` directory.
Write Preview
Loading…
Cancel
Save