|
|
@ -46,15 +46,15 @@ extendable, so other kinds of usage will be possible.
|
|
|
|
This project have a great starting point -- there is an old grading system used
|
|
|
|
This project have a great starting point -- there is an old grading system used
|
|
|
|
these days at out university, so its mistakes and weaknesses can be adressed, and
|
|
|
|
these days at out university, so its mistakes and weaknesses can be adressed, and
|
|
|
|
a lot of teachers willing to use the new system. The ideas were collected both
|
|
|
|
a lot of teachers willing to use the new system. The ideas were collected both
|
|
|
|
from our personal experience with old system and from requests of some teachers.
|
|
|
|
from our personal experience with old system and from teachers' requests.
|
|
|
|
|
|
|
|
|
|
|
|
**Common grading system features:**
|
|
|
|
**Common grading system features:**
|
|
|
|
|
|
|
|
|
|
|
|
- creating exercises including textual description, sample inputs and correct
|
|
|
|
- creating exercises including textual description, sample inputs and correct
|
|
|
|
reference outputs (for example "sum all numbers from given file and write the
|
|
|
|
reference outputs (for example "sum all numbers from given file and write the
|
|
|
|
result to the standard output")
|
|
|
|
result to the standard output")
|
|
|
|
- assign the exercise to a group of users with some additional properties like
|
|
|
|
- assign the exercise to a group of users with some additional properties set
|
|
|
|
deadlines set
|
|
|
|
(deadlines, etc.)
|
|
|
|
- user interface for interaction with the system, mainly for showing assigned
|
|
|
|
- user interface for interaction with the system, mainly for showing assigned
|
|
|
|
exercises, uploading solution sources and presenting evaluated results
|
|
|
|
exercises, uploading solution sources and presenting evaluated results
|
|
|
|
- safe environment to execute student solutions withing prescribed time and
|
|
|
|
- safe environment to execute student solutions withing prescribed time and
|
|
|
@ -63,7 +63,7 @@ from our personal experience with old system and from requests of some teachers.
|
|
|
|
- user management with support of roles (at least two -- _student_ and
|
|
|
|
- user management with support of roles (at least two -- _student_ and
|
|
|
|
_supervisor_)
|
|
|
|
_supervisor_)
|
|
|
|
- administrative interface for manual checking of solutions, overriding
|
|
|
|
- administrative interface for manual checking of solutions, overriding
|
|
|
|
automatically obtained amount of points and view overall statistics about
|
|
|
|
automatically assigned amount of points and view overall statistics about
|
|
|
|
users
|
|
|
|
users
|
|
|
|
|
|
|
|
|
|
|
|
Schools and educating institutions needs a bit specific setup. System
|
|
|
|
Schools and educating institutions needs a bit specific setup. System
|
|
|
@ -72,21 +72,20 @@ configuration needs to reflect structure of courses and user hierarchy
|
|
|
|
with currently used system at the university, but there are some new ones, which
|
|
|
|
with currently used system at the university, but there are some new ones, which
|
|
|
|
escalated during long production lifetime of old system. From student
|
|
|
|
escalated during long production lifetime of old system. From student
|
|
|
|
perspective there are not many thing to improve, but a lot of them come from
|
|
|
|
perspective there are not many thing to improve, but a lot of them come from
|
|
|
|
administrator and supervisors. Collected ideas are from meeting with faculty
|
|
|
|
administrator and supervisors. Collected ideas are mainly from meeting with
|
|
|
|
staff involved in current system.
|
|
|
|
faculty staff involved in current system.
|
|
|
|
|
|
|
|
|
|
|
|
**Wanted features for new system:**
|
|
|
|
**Wanted features for new system:**
|
|
|
|
|
|
|
|
|
|
|
|
- keep it simple as possible
|
|
|
|
|
|
|
|
- login through university authentication system
|
|
|
|
- login through university authentication system
|
|
|
|
- support of multiple programming environments at once to avoid unacceptable
|
|
|
|
- support for multiple programming environments at once to avoid unacceptable
|
|
|
|
workload for administrator and hardware occupation
|
|
|
|
workload for administrator and hardware occupation
|
|
|
|
- localization (both UI and exercises)
|
|
|
|
- localization (both UI and exercises)
|
|
|
|
- Markdown support for exercise texts
|
|
|
|
- Markdown support for exercise texts
|
|
|
|
- tagging exercises and search by tags
|
|
|
|
- tagging exercises and search by tags
|
|
|
|
- comments, comments, comments (exercises, tests, solutions, ...)
|
|
|
|
- comments, comments, comments (exercises, tests, solutions, ...)
|
|
|
|
- edit student solution and privately resubmit it
|
|
|
|
- edit student solution and privately resubmit it
|
|
|
|
- resubmit solution with saving all results
|
|
|
|
- resubmit solution with saving all (including temporary) results
|
|
|
|
- mark one student solution as accepted (used for grading this assignment)
|
|
|
|
- mark one student solution as accepted (used for grading this assignment)
|
|
|
|
- web and command-line submit tool
|
|
|
|
- web and command-line submit tool
|
|
|
|
- SIS integration for fetching personal user data
|
|
|
|
- SIS integration for fetching personal user data
|
|
|
@ -95,7 +94,15 @@ staff involved in current system.
|
|
|
|
layer for ordinary configuration cases
|
|
|
|
layer for ordinary configuration cases
|
|
|
|
- use of modern technologies with state-of-the-art compilers
|
|
|
|
- use of modern technologies with state-of-the-art compilers
|
|
|
|
|
|
|
|
|
|
|
|
@todo: some conclusion
|
|
|
|
The survey shows that the system is used in many different ways, but the core
|
|
|
|
|
|
|
|
functionality is the same for all. When the system is ready, it is likely that
|
|
|
|
|
|
|
|
new ideas are developed. Thus the system must be designed to be easily
|
|
|
|
|
|
|
|
extendable, so everyone can develop his dream feature. This also means, that
|
|
|
|
|
|
|
|
widely used programming languages and techniques should be used, so users can
|
|
|
|
|
|
|
|
quickly understand the code and make changes.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
To find out current state in the field of automatic grading systems, let's do a
|
|
|
|
|
|
|
|
short survey at universities, programming contests or online tools.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Related projects
|
|
|
|
### Related projects
|
|
|
@ -107,8 +114,8 @@ list has a brief description and some key features mentioned.
|
|
|
|
|
|
|
|
|
|
|
|
#### CodEx
|
|
|
|
#### CodEx
|
|
|
|
|
|
|
|
|
|
|
|
There already is a grading solution at MFF UK, which was inplemented in 2006 by
|
|
|
|
There already is a grading solution at MFF UK, which was implemented in 2006 by
|
|
|
|
group of students. Its name is [CodEx - The Code
|
|
|
|
group of students. Its name is [CodEx -- The Code
|
|
|
|
Examiner](http://codex.ms.mff.cuni.cz/project/) and it has been used with some
|
|
|
|
Examiner](http://codex.ms.mff.cuni.cz/project/) and it has been used with some
|
|
|
|
improvements since then. The original plan was to use the system only for basic
|
|
|
|
improvements since then. The original plan was to use the system only for basic
|
|
|
|
programming courses, but there is demand for adapting it for many different
|
|
|
|
programming courses, but there is demand for adapting it for many different
|
|
|
@ -198,28 +205,23 @@ recruiters and also some universities.
|
|
|
|
|
|
|
|
|
|
|
|
### ReCodEx goals
|
|
|
|
### ReCodEx goals
|
|
|
|
|
|
|
|
|
|
|
|
After considering all these facts, it is clear that CodEx cannot be used
|
|
|
|
From the survey above it is clear, that none of the existing systems is capable
|
|
|
|
anymore. The project is too old to just maintain it and extend for modern
|
|
|
|
of all the features collected for the new system. No grading system is designed
|
|
|
|
technologies. Thus, it needs to be completely rewritten or another solution
|
|
|
|
to support complicated evaluation pipeline, so this part is unexplored field and
|
|
|
|
must be found.
|
|
|
|
has to be designed with caution. Also, no project is modern and extendable in a
|
|
|
|
|
|
|
|
way that it can be used as a base for ReCodEx. After considering all these
|
|
|
|
From the research above, we set up several goals, which a new system should
|
|
|
|
facts, it is clear that the new system has to be written from scratch. This
|
|
|
|
have. They mostly reflect drawbacks of current version of CodEx and wishes of
|
|
|
|
implies, that only subset of all features will be implemented in the first
|
|
|
|
MFF users. No existing tool fits our needs, for example no examined project
|
|
|
|
version, the others following later.
|
|
|
|
provides complex execution/evaluation pipeline to support needs of courses like
|
|
|
|
|
|
|
|
Compiler principles. Modifying CodEx is also not an option -- the required
|
|
|
|
|
|
|
|
scope of a new solution is too big. To sum up, a new evaluation system has to
|
|
|
|
|
|
|
|
be written, with only small parts of reused code from CodEx (for example
|
|
|
|
|
|
|
|
judges).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The new project is **ReCodEx -- ReCodEx Code Examiner**. The name should point
|
|
|
|
The new project is **ReCodEx -- ReCodEx Code Examiner**. The name should point
|
|
|
|
to CodEx, previous evaluation solution, but also reflect new approach to solve
|
|
|
|
to CodEx, previous evaluation solution, but also reflect new approach to solve
|
|
|
|
issues. **Re** as part of the name means redesigned, rewritten, renewed or
|
|
|
|
issues. **Re** as part of the name means redesigned, rewritten, renewed or
|
|
|
|
restarted.
|
|
|
|
restarted.
|
|
|
|
|
|
|
|
|
|
|
|
Official assignment of the project is available at [web of software project
|
|
|
|
From the previous research, we set up several goals, which a new system should
|
|
|
|
committee](http://www.ksi.mff.cuni.cz/sw-projekty/zadani/recodex.pdf) (only in
|
|
|
|
have. They mostly reflect drawbacks of current version of CodEx and reasonable
|
|
|
|
Czech). Most notable features are following:
|
|
|
|
wishes of university users. Most notable features are following:
|
|
|
|
|
|
|
|
|
|
|
|
- modern HTML5 web frontend written in Javascript using a suitable framework
|
|
|
|
- modern HTML5 web frontend written in Javascript using a suitable framework
|
|
|
|
- REST API implemented in PHP, communicating with database, backend and file
|
|
|
|
- REST API implemented in PHP, communicating with database, backend and file
|
|
|
@ -231,6 +233,11 @@ Czech). Most notable features are following:
|
|
|
|
- evaluation procedure configured in YAML file, compound of small tasks
|
|
|
|
- evaluation procedure configured in YAML file, compound of small tasks
|
|
|
|
connected into arbitrary oriented acyclic graph
|
|
|
|
connected into arbitrary oriented acyclic graph
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Usage design
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@todo: describe detailed usage ... grading and this kind of stuff
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Implementation analysis
|
|
|
|
### Implementation analysis
|
|
|
|
|
|
|
|
|
|
|
|
@todo: GENERALLY... what problems were solved, how they can be solved and what was the final solution
|
|
|
|
@todo: GENERALLY... what problems were solved, how they can be solved and what was the final solution
|
|
|
|