master
Simon Rozsival 8 years ago
parent 3e7b68882a
commit 6bd580a735

@ -1,4 +0,0 @@
# FAQ
// todo: readonly files which are created by internal tasks
// todo: job identification cannot contain underscore '_' character by default

@ -5,7 +5,6 @@
* [[Home]] * [[Home]]
## Separated pages ## Separated pages
* [[FAQ]]
* [[Logo]] * [[Logo]]
* [[Database schema]] * [[Database schema]]

@ -1,212 +0,0 @@
# Introduction
Generally, there are a lot of different ways and opinions on how to teach people
something new. However, most people agree that a hands-on experience is one of
the best ways to make the human brain remember a new skill. Learning must be
entertaining and interactive, with fast and frequent feedback. Some kinds of
knowledge are more suitable for this practical type of learning than others, and
fortunately, programming is one of them.
University education system is one of the areas where this knowledge can be
applied. In computer programming, there are several requirements such as the
code being syntactically correct, efficient and easy to read, maintain and
extend. Correctness and efficiency can be tested automatically to help teachers
save time for their research, but checking for bad design, habits and mistakes
is really hard to automate and requires manpower.
Checking programs written by students takes a lot of time and requires a lot of
mechanical, repetitive work. The first idea of an automatic evaluation system
comes from Stanford University profesors in 1965. They implemented a system
which evaluated code in Algol submitted on punch cards. In following years, many
similar products were written.
There are two basic ways of automatically evaluating code -- statically (check
the code without running it; safe, but not much precise) or dynamically (run the
code on testing inputs with checking the outputs against reference ones; needs
sandboxing, but provides good real world experience).
This project focuses on the machine-controlled part of source code evaluation.
First, problems of present software at our university were discussed and similar
projects at other educational institutions were examined. With acquired
knowledge from such projects in production, we set up goals for the new
evaluation system, designed the architecture and implemented a fully operational
solution. The system is now ready for production testing at our university.
## Current solution at MFF UK
The ideas presented above are not completely new. There was a group of students,
who already implemented an evaluation solution for student's homeworks in 2006.
Its name is [CodEx - The Code Examiner](http://codex.ms.mff.cuni.cz/project/)
and it has been used with some improvements since then. The original plan was to
use the system only for basic programming courses, but there is demand for
adapting it for many different subjects.
CodEx is based on dynamic analysis. It features a web-based interface, where
supervisors assign exercises to their students and the students have a time
window to submit the solution. Each solution is compiled and run in sandbox
(MO-Eval). The metrics which are checked are: corectness of the output, time and
memory limits. It supports programs written in C, C++, C#, Java, Pascal, Python
and Haskell.
Current system is old, but robust. There were no major security incidents during its production usage. However, from today's perspective there are several drawbacks. The main ones are:
- **web interface** -- The web interface is simple and fully functional. But
rapid development in web technologies opens new horizons of how web interface
can be made.
- **web api** -- CodEx offers a very limited XML API based on outdated
technologies that is not sufficient for users who would like to create custom
interfaces such as a command line tool or mobile application.
- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system
calls and blocking the bad ones. This can be easily done for single-threaded
applications, but proves difficult with multi-threaded ones. In present day,
parallelism is a very important area of computing, so there is requirement to
test multi-threaded applications too.
- **instances** -- Different ways of CodEx usage scenarios requires separate
instances (Programming I and II, Java, C#, etc.). This configuration is not
user friendly (students have to register in each instance separately) and
burdens administrators with unnecessary work. CodEx architecture does not
allow sharing hardware between instances, which results in an inefficient use
of hardware for evaluation.
- **task extensibility** -- There is a need to test and evaluate complicated
programs for classes such as Parallel programming or Compiler principles,
which have a more difficult evaluation chain than simple
compilation/execution/evaluation provided by CodEx.
After considerring all these facts, it is clear that CodEx cannot be used
anymore. The project is too old to just maintain it and extend for modern
technologies. Thus, it needs to be completely rewritten or another solution must
be found.
## Analysis of related projects
First of all, some code evaluating projects were found and examined. It is not a complete list of such evaluators, but just a few projects which are used these days and can be an inspiration for our project.
### Progtest
[Progtest](https://progtest.fit.cvut.cz/) is private project from FIT ČVUT in
Prague. As far as we know it is used for C/C++, Bash programming and
knowledge-based quizzes. There are several bonus points and penalties and also a
few hints what is failing in submitted solution. It is very strict on source
code quality, for example `-pedantic` option of GCC, Valgrind for memory leaks
or array boundaries checks via `mudflap` library.
### Codility
[Codility](https://codility.com/) is web based solution primary targeted to company recruiters. It is commercial product of SaaS type supporting 16 programming languages. The [UI](http://1.bp.blogspot.com/-_isqWtuEvvY/U8_SbkUMP-I/AAAAAAAAAL0/Hup_amNYU2s/s1600/cui.png) of Codility is [opensource](https://github.com/Codility/cui), the rest of source code is not available. One interesting feature is 'task timeline' -- captured progress of writing code for each user.
### CMS
[CMS](http://cms-dev.github.io/index.html) is an opensource distributed system
for running and organizing programming contests. It is written in Python and
contain several modules. CMS supports C/C++, Pascal, Python, PHP and Java.
PostgreSQL is a single point of failure, all modules heavily depend on database
connection. Task evaluation can be only three step pipeline -- compilation,
execution, evaluation. Execution is performed in
[Isolate](https://github.com/ioi/isolate), sandbox written by consultant of our
project, Mgr. Martin Mareš, Ph.D.
### MOE
[MOE](http://www.ucw.cz/moe/) is a grading system written in Shell scripts, C
and Python. It does not provide a default GUI interface, all actions have to be
performed from command line. The system does not evaluate submissions in real
time, results are computed in batch mode after exercise deadline, using Isolate
for sandboxing. Parts of MOE are used in other systems like CodEx or CMS, but
the system is generally obsolete.
### Kattis
[Kattis](http://www.kattis.com/) is another SaaS solution. It provides a clean
and functional web UI, but the rest of the application is too simple. A nice
feature is the usage of a [standardized
format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for
exercises. Kattis is primarily used by programming contest organizators, company
recruiters and also some universities.
## ReCodEx goals
From the research above, we set up several goals, which a new system should
have. They mostly reflect drawbacks of current version of CodEx. No existing
tool fits our needs, for example no examined project provides complex
execution/evaluation pipeline to support needs of courses like Compiler
principles. Modifying CodEx is also not an option -- the required scope of a new
solution is too big. To sum up, a new evaluation system has to be written, with
only small parts of reused code from CodEx (for example judges).
The new project is **ReCodEx -- ReCodEx Code Examiner**. The name should point to
CodEx, previous evaluation solution, but also reflect new approach to solve
issues. **Re** as part of the name means redesigned, rewritten, renewed or
restarted.
Official assignment of the project is available at [web of software project committee](http://www.ksi.mff.cuni.cz/sw-projekty/zadani/recodex.pdf) (only in Czech). Most notable features are following:
- modern HTML5 web frontend written in Javascript using a suitable framework
- REST API implemented in PHP, communicating with database, backend and file server
- backend is implemented as distributed system on top of message queue framework (ZeroMQ) with master-worker architecture
- worker with basic support of Windows environment (without sandbox, no general purpose suitable tool available yet)
- evaluation procedure configured in YAML file, compound of small tasks connected into arbitrary oriented acyclic graph
## Terminology
Official terminology of ReCodEx which will be used in documentation and within code.
* **Exercise** -- Exercise is a template of programming problem including
detailed text description, evaluation instructions, sample implementation and
reference inputs and outputs. Typically, an author of exercise is a lecturer
of a programming class.
* **Assignment** -- Assignment is basically an instance of an exercise which was
assigned to a group of students by their supervisor. Supervisor can alter
predefined restrictions for resulting code (execution time limit, etc.),
deadlines and maximal amount of points for correct solutions.
* **Reference solution** -- Solution of exercise provided by author. This
solution should pass all test cases and could be also used for
auto-calibration of the exercise. One exercise could have more reference
solutions, for example in different programming languages or with varied
levels of efficiency.
* **Submission** -- Submission is one student's solution of an assignment
received by ReCodEx API. Submission can contain submitted source code and
additional information about assignment, exercise or submitter.
* **Job** -- Piece of work for a worker, generally corresponding to evaluation
of one submission. There are also other types of jobs like benchmarking
submission for memory and time limits configuration, but this classification
has no effect for evaluation. Internally, job is a set of small tasks defined
in exercise configuration. Job itself is transfered in the form of an archive
with submitted source codes and a configuration file written in YAML.
* **Task** -- Atomic piece of work defined in job configuration which can
execute external program or some internal command. External program execution
is (mostly) performed in sandboxed environment, internal commands are executed
directly. For example, one task could make a new directory, copy a file or
compile source codes using GCC.
* **Test** -- Test is a logical part of a job that checks the correctness of a
program. There can be multiple tests inside a job, which together prove the
validity and correctness of all aspects of the solution. In the simplest case,
testing is done by providing reference inputs to the tested program and
results are compared with reference outputs. One test consists of multiple
tasks.
* **Judge** -- Judge is a standalone comparision program that compares sample
outputs against output from tested programs.
* **Limits** -- Tasks executing external programs are usually executed in a
sandbox with defined limits on execution time, allocated memory, used disk
space and others. These limits are specified in job configuration. The term
_limits_ in this context means all the restrictions together.
* **Hwgroup** -- Hardware group is a set of workers with similar hardware. Its
purpose is to group workers that are likely to run a program using the same
amount of resources. Test limits are defined separately for each group. A
group has a unique string identifier and every worker in a group has this
identifier in its configuration file. Hardware group management is done
manually by the system administrator. Jobs can be routed to workers based on
hwgroup.

@ -1010,9 +1010,70 @@ existing parsers for most of the programming languages and it is easy enough to
learn and understand. Another choice which make sense is JSON but at the end learn and understand. Another choice which make sense is JSON but at the end
YAML seemed to be better. YAML seemed to be better.
Job configuration as it was implemented and designed is described in 'Job
configuration' appendix where list of all task types is present alongside with
whole configuration structure and much more.
#### Task Types
From the low-level point of view there are only two types of tasks in the job.
First ones are doing some internal operation which should work on all platforms
or operating systems same way. Second type of tasks are external ones which are
executing external binary.
Internal tasks should handle at least these operations:
- *fetch* -- fetch single file from fileserver
- *copy* -- copy file between directories
- *remove* -- remove single file or folder
- *extract* -- extract files from downloaded archive
These internal operations are essential but many more can be eventually
implemented.
External tasks executing external binary should be optionally runnable in
sandbox. But for security sake there is no reason to execute them outside of
sandbox. So all external tasks are executed within sandbox which should be
general and configurable. Configuration options for sandboxes will be called
limits and there can be specified for example time or memory limits.
#### Configuration File Content #### Configuration File Content
@todo: discuss what should be in configuration: limits, dependencies, priorities... whatever Content of configuration file can be divided in two parts, first concerns about
job in general and its metadata, second one relates to tasks and their
specification.
There is not much to express in general job metadata. There can be
identification of job and some general options, like enable/disable logging. But
really necessary item is address of fileserver from where supplementary files
should be downloaded. This option is crucial because there can be more
fileservers and worker have no other way how to figure out where the files might
be.
More interesting situation is about metadata of tasks. From the initial analysis
of evaluation unit and its structure there can be derived at least these
generally needed items:
- *task identification* -- identificator used at least for specifying
dependencies
- *type* -- as described before, one of: 'initiation', 'execution', 'evaluation'
or 'inner'
- *priority* -- priority can additionally control execution flow in task graph
- *dependencies* -- necessary item for constructing hierarchy of tasks into DAG
- *execution command* -- command which should be executed withing this tasks
with possible parameters parameters
Previous list of items is applicable both for internal and external tasks.
Internal tasks do not need any more items but external do. Additional items are
exclusively related to sandboxing and limitation:
- *sandbox name* -- there should be possibility to have multiple sandboxes, so
identification of the right one is needed
- *limits* -- hardware and software resources limitations
- *time limit* -- limits time of execution
- *memory limit* -- maximum memory which can be consumed by external program
- *I/O operations* -- limitation concerning disk operations
- *restrict filesystem* -- restrict or enable access to directories
#### Supplementary Files #### Supplementary Files
@ -1442,8 +1503,9 @@ also not a problem, file is firstly downloaded to working folder and after that
copied to cache. copied to cache.
And even if something else unexpectedly fails and because of that fetch task And even if something else unexpectedly fails and because of that fetch task
will fail during execution, even that should be fine as mentioned previsously. will fail during execution, even that should be fine as mentioned previously.
This should be the last salvation in case everything else goes wrong. Reassigning of job should be the last salvation in case everything else goes
wrong.
### Monitor ### Monitor
@ -2931,7 +2993,7 @@ and then ends. This means that the cleaner has to be run repeatedly, for example
using cron, systemd timer or Windows task scheduler. For proper function of the using cron, systemd timer or Windows task scheduler. For proper function of the
cleaner a suitable cronning interval has to be used. It is recommended to use cleaner a suitable cronning interval has to be used. It is recommended to use
24 hour interval which is sufficient enough for intended usage. The value is set 24 hour interval which is sufficient enough for intended usage. The value is set
in hte configuration file of the cleaner. in the configuration file of the cleaner.
## REST API ## REST API

@ -1,78 +0,0 @@
# User documentation
## Student
### From registration to solution submit
- Registration can be made on separate page, _Create acount_ link from left menu. There are two forms, one for ReCodEx authentication and one for CAS (Charles University Authentication System) authentication. During the process instance has to be chosen. After successful registration you will be automatically logged in.
- Login forms are located on page accessible through _Sign in_ link from menu. The same type of form as the registration must be used. After login occurs redirection to _Dashboard_ -- page with overview of achievements for each group user is member of.
- New page with details about used instance is accessible from menu, specificaly containing list public groups. From group detail page is possible to join (or leave) the group. Private groups are hidden and only supervisor can add students to that group.
- Pages of joined groups are accessible from menu. There is a list of assigned exercises. Detail of each assignment contains description of problem, deadlines, already submitted solution and other information.
- Solution is submitted from assignment detail page through _Submit new solution_ button. After uploading source files the evaluation will start showing current progress. When finished, button _See results_ redirects to page with results summary.
### Password reset
- Password reset could by done only for ReCodEx authentication, CAS has its own system.
- Under login form there is a link leading to forgotten password page. After providing an email address a message with short term link to password change page is sent.
- On that page the new password can be set. Please note that password change link is time limited and is functional only for few minutes.
## Group supervisor
### Create and update exercise
- For management of exercises there is whole page which can be visited from _Exercises_ link in main menu.
- In there you can list all public exercises and the ones you created.
- If you choose to create new exercise, there is a button _Add exercise_ for that purpose.
- After clicking on it you will be redirected to exercise editation page.
- In here there are three forms, in the first one you can add localized descriptions of exercise, change difficulty, name and some description which can be seen by supervisors.
- Second form serves for the purpose of uploading supplementary files to exercise. These files are handed over directly to fileserver and afterwards can be used in job configuration with their hashed names. Note that these files are not additional files for exercise and are not visible to outside world!
- The most important thing on exercise editation page is runtime configurations, in here you can select environment which will be used for processing of this exercise and most notably you should provide textual job configuration.
- Job configuration is instruction set for workers which is used for detailed description of execution flow. It should be provided in right format which can be found in separate [article](https://github.com/ReCodEx/wiki/wiki/Assignments#job-configuration).
### Assign exercise to group
- On group detail page is search box for exercises. Every match has a button _Assign_ which redirects to configuration page. Assignment text, visibility to students, time and memoty limits for tests, deadlines and other informations can be set there.
- Assigned exercises are shown in separate box in group detail page, where is also button for editing each assignment.
### Add student to group
- Groups can be public or private. Public groups are visible to all instance users and students can join them. Private ones are hidden and students can be only added to them.
- Supervisor of group can add students to it by hand on group detail page through _Add student_ search box.
### View students results
- Agregate results for each assignment are accessible through _View student results_ button on assignment detail page. There is a list of submissions for every group member.
- Each submission can be viewed in detail including submitted files on special page.
## Group administrator
### Create sub-group
- Group administrator can create sub-groups of his own groups.
- On group detail page there is a form for creating new subgroup, including fields for brief description in Markdown and other basic information.
- Newly created group is shown in refreshed group hierarchy.
### Add and delete supervisor
- On group detail page is list of supervisors and search form for adding new ones.
- Supervisors are searched by their names and surnames. Adding supervisor immediately changes supervisor list from where the removal is possible.
## Superadministrator
Superadmin is user with the most priviledges and as such superadmin should be quite unique role. Ideally there should be only one of this kind, used with special caution and adequate security. With this stated it is obvious that superadmin can perform any action the API is capable of.
### Users management
There are only few roles to which users can belong in ReCodEx. Basically there are only three: _student_, _supervisor_, and _superadmin_. Base role is student which is assigned to every registered user. Roles are stored in database alongside other information about user. One user always has only one role at the time. At first startup of ReCodEx administrator should create his account and then change role in database by hand. After that manual intervention into database should never be needed.
There is a little catch in groups and instances management. Groups can have admins and supervisors. This setting is valid only per one particular group and has to be separated from basic role system. This implies that supervisor in one group can be student in another and simultaneously have global supervisor role. Changing role from student to supervisor and back is done automatically by application and should not be managed by hand in database! Previously stated information can be applied to instances as well, but instances can only have admins.
Roles description:
- Student -- Default role which is used for newly created accounts. Student can join or leave public groups and submit solutions of assigned exercises.
- Supervisor -- Inherits all permissions from student role. Can manage groups to which he/she belongs to. Supervisor can also view and change groups details, manage assigned exercises, view students in group and their solutions for assigned exercises. On top of that supervisor can create/delete groups too, but only as subgroup of groups he/she belongs to.
- Superadmin -- Inherits all permissions from supervisor role. Most powerful user in ReCodEx who should be able to do everything which is provided by application.
Loading…
Cancel
Save