@ -17,11 +17,11 @@ Ideas presented above are not completely new. There was a group of students, who
CodEx is based on dynamic analysis. It is a system with web-based interface, where supervisors assign exercises to their students and the students have a time window to submit the solution. Each solution is compiled and run in sandbox (MO-Eval). The metrics which are checked are: corectness of the output, time and memory limits. Supported languages are C, C++, C#, Java, Pascal, Python and Haskel.
Current system is old, but robust. There were no major security incident during its production usage. However, from today's perspective there are several drawbacks. The main ones are:
Current system is old, but robust. There were no major security incidents during its production usage. However, from today's perspective there are several drawbacks. The main ones are:
- **web interface** -- The web interface is simple and fully functional. But rapid development in web technologies opens new horizons of how web interface can be made.
- **web api** -- There is only very limited XML API based on outdated technologies in current CodEx. This locks users from creating custom interfaces like command line tool or mobile application.
- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system calls into operation system and blocking the bad ones. This could be easily done only for single-threaded applications. These days parallelism is very important part of computing, so there is requirement to test multi-threaded applications too.
- **sandboxing** -- MO-Eval sandbox is based on principle of monitoring system calls into operating system and blocking the bad ones. This could be easily done only for single-threaded applications. These days parallelism is very important part of computing, so there is requirement to test multi-threaded applications too.
- **instances** -- Different ways of CodEx usage scenarios requires separate instances (Programming I and II, Java, C#, etc.). This configuration is not user friendly (students have to register to each instance again) and puts unnecessary work to administrators. CodEx architecture does not allow to share hardware between instances, so not trivial amount of additional hardware is occupied.
- **task extensibility** -- There is a need to test and evaluate complicated programs from Parallel programming or Compiler principles classes, which have more difficult evaluation chain than simple compilation/execution/evaluation. CodEx is only capable of the simple solution without possibility of easy extension.
@ -46,11 +46,11 @@ First of all, some code evaluating projects were found and examined. It is not a
### MOE
[MOE](http://www.ucw.cz/moe/) is grading system written in Shell scripts, C and Python. It does not provide default GUI interface, all actions have to be performed from command line. The system does not evaluate submissions in real time, results are computated in batch mode after exercise deadline. Used sandboxing environment is Isolate. Parts of MOE are used in other systems like CodEx or CMS, but the system is generally obsolete.
[MOE](http://www.ucw.cz/moe/) is grading system written in Shell scripts, C and Python. It does not provide default GUI interface, all actions have to be performed from command line. The system does not evaluate submissions in real time, results are computed in batch mode after exercise deadline. Used sandboxing environment is Isolate. Parts of MOE are used in other systems like CodEx or CMS, but the system is generally obsolete.
### Kattis
[Kattis](http://www.kattis.com/) is another SaaS solution. It provides pretty nice web UI, but the rest of this application is too much simple. Nice point is used [standartized format](http://www.problemarchive.org/wiki/index.php/Problem_Format) of exercises. Kattis is primarily used by programming contest organizators, company recruiters and also some universities.
[Kattis](http://www.kattis.com/) is another SaaS solution. It provides pretty nice web UI, but the rest of the application is too simple. Nice feature is usage of [standartized format](http://www.problemarchive.org/wiki/index.php/Problem_Format) for exercises. Kattis is primarily used by programming contest organizators, company recruiters and also some universities.
## ReCodEx goals
@ -74,21 +74,25 @@ Official terminology of ReCodEx which will be used in documentation and within c
* **Exercise** -- Exercise is a template of programming problem including detailed text description, evaluation instructions, sample implementation and reference inputs and outputs. Author of exercise is mostly lecturer of a programming class.
* **Assignment** -- Assignment is basically instance of exercise which was assigned to a group of students by their supervisor. He can alter predefined restrictions for resulting code (execution time limit, etc.), deadlines and maximal amount of points for correct solutions.
* **Assignment** -- Assignment is basically instance of exercise which was assigned to a group of students by their supervisor. Supervisor can alter predefined restrictions for resulting code (execution time limit, etc.), deadlines and maximal amount of points for correct solutions.
* **Reference solution** -- Solution of exercise provided by author. This solution should pass all test cases and could be also used for auto-calibration of the exercise. One exercise could have more reference solutions, for example in different programming languages or with various level of complexity.
* **Submission** -- Submission is one student solution of an assignment received by ReCodEx API. Submission can contain submitted source code and additional informations about assignment, exercise or submitter.
* **Submission** -- Submission is one student solution of an assignment received by ReCodEx API. Submission can contain submitted source code and additional information about assignment, exercise or submitter.
* **Job** -- Piece of work for worker, generally corresponding to evaluation of one submission. There are also other types of jobs like benchmarking submission for memory and time limits configuration, but this classification has no effect for evaluation. Internally job is set of small tasks defined in exercise configuration. Job itself is transfered in form of archive with submitted source codes and configuration file written in YAML.
* **Task** -- Atomic piece of work which can execute external program or some internal command. External program execution is (mostly) performed in sandboxed environment, internal commands are executed directly. For example, one task could make a new directory, copy a file or compile source codes using GCC.
* **Task** -- Atomic piece of work defined in job configuration which can execute external program or some internal command. External program execution is (mostly) performed in sandboxed environment, internal commands are executed directly. For example, one task could make a new directory, copy a file or compile source codes using GCC.
* **Test** -- Test is a piece of work to check correctness of a program. There are multiple tests inside job, which together checks validity and correctness of all aspects of exercise solution. In easiest case, testing is done by providing reference inputs to the tested program and results are compared with reference outputs. One test consist of multiple tasks.
* **Test** -- Test is a piece of work to check correctness of a program. There are multiple tests inside job, which together checks validity and correctness of all aspects of exercise solution. In easiest case, testing is done by providing reference inputs to the tested program and results are compared with reference outputs. One test consists of multiple tasks.
* **Judge** -- Judge is a standalone comparision program which compares sample outputs against output from program.
* **Judge** -- Judge is a standalone comparision program which compares sample outputs against output from tested program.
* **Limits** -- Tasks executing external program are usually using sandbox with defined limits on running time, consumed memory, used disk space and others. The term 'limits' in this context means all these restrictions for program execution together.
* **Limits** -- Tasks executing external programs are usually executed in sandbox, to limit these programs predefined limits are used. These limits are specified in job configuration and can limit execution time, consumed memory, used disk space and some other worth-to-watch values.
* **Hwgroup** -- Hardware group is set of workers with similar hardware capabilities. Each group has unique string identifier and every worker in particular group has that identifier inside its configuration. Hardware group management is done manualy by system administrator. Jobs are routed to the workers according to hwgroup, limits are also tied up with specific hwgroup.
// TODO: which one, the above or the below?
Tasks executing external programs are usually using sandbox with defined limits on run time, consumed memory, used disk space and others. The term 'limits' in this context means all these restrictions for program execution together.
* **Hwgroup** -- Hardware group is set of workers with similar hardware capabilities. Each group has unique string identifier and every worker in particular group has that identifier inside its configuration. Hardware group management is done manually by system administrator. Jobs are routed to the workers according to hwgroup, limits are also tied up with specific hwgroup.
Martin Polanka
8 years ago