|
|
@ -36,11 +36,79 @@ solution. The system is now ready for production testing at our university.
|
|
|
|
Analysis
|
|
|
|
Analysis
|
|
|
|
--------
|
|
|
|
--------
|
|
|
|
|
|
|
|
|
|
|
|
### Current solution at MFF UK
|
|
|
|
### Assignment
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The major goal of this project is to create grading application that will be
|
|
|
|
|
|
|
|
used for programming classes at the Faculty of Mathematics and Physics, Charles
|
|
|
|
|
|
|
|
University. However, the application should have modular design and be easily
|
|
|
|
|
|
|
|
extendable, so other kinds of usage will be possible.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This project have a great starting point -- there is an old grading system used
|
|
|
|
|
|
|
|
these days at out university, so its mistakes and weaknesses can be adressed, and
|
|
|
|
|
|
|
|
a lot of teachers willing to use the new system. The ideas were collected both
|
|
|
|
|
|
|
|
from our personal experience with old system and from requests of some teachers.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Common grading system features:**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- creating exercises including textual description, sample inputs and correct
|
|
|
|
|
|
|
|
reference outputs (for example "sum all numbers from given file and write the
|
|
|
|
|
|
|
|
result to the standard output")
|
|
|
|
|
|
|
|
- assign the exercise to a group of users with some additional properties like
|
|
|
|
|
|
|
|
deadlines set
|
|
|
|
|
|
|
|
- user interface for interaction with the system, mainly for showing assigned
|
|
|
|
|
|
|
|
exercises, uploading solution sources and presenting evaluated results
|
|
|
|
|
|
|
|
- safe environment to execute student solutions withing prescribed time and
|
|
|
|
|
|
|
|
memory limits and check corectness of outputs
|
|
|
|
|
|
|
|
- assigning points to users depending of correctness of his/her solution
|
|
|
|
|
|
|
|
- user management with support of roles (at least two -- _student_ and
|
|
|
|
|
|
|
|
_supervisor_)
|
|
|
|
|
|
|
|
- administrative interface for manual checking of solutions, overriding
|
|
|
|
|
|
|
|
automatically obtained amount of points and view overall statistics about
|
|
|
|
|
|
|
|
users
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Schools and educating institutions needs a bit specific setup. System
|
|
|
|
|
|
|
|
configuration needs to reflect structure of courses and user hierarchy
|
|
|
|
|
|
|
|
(_student_, _supervisor_ and _administrator_). These requirement are satisfied
|
|
|
|
|
|
|
|
with currently used system at the university, but there are some new ones, which
|
|
|
|
|
|
|
|
escalated during long production lifetime of old system. From student
|
|
|
|
|
|
|
|
perspective there are not many thing to improve, but a lot of them come from
|
|
|
|
|
|
|
|
administrator and supervisors. Collected ideas are from meeting with faculty
|
|
|
|
|
|
|
|
staff involved in current system.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Wanted features for new system:**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- keep it simple as possible
|
|
|
|
|
|
|
|
- login through university authentication system
|
|
|
|
|
|
|
|
- support of multiple programming environments at once to avoid unacceptable
|
|
|
|
|
|
|
|
workload for administrator and hardware occupation
|
|
|
|
|
|
|
|
- localization (both UI and exercises)
|
|
|
|
|
|
|
|
- Markdown support for exercise texts
|
|
|
|
|
|
|
|
- tagging exercises and search by tags
|
|
|
|
|
|
|
|
- comments, comments, comments (exercises, tests, solutions, ...)
|
|
|
|
|
|
|
|
- edit student solution and privately resubmit it
|
|
|
|
|
|
|
|
- resubmit solution with saving all results
|
|
|
|
|
|
|
|
- mark one student solution as accepted (used for grading this assignment)
|
|
|
|
|
|
|
|
- web and command-line submit tool
|
|
|
|
|
|
|
|
- SIS integration for fetching personal user data
|
|
|
|
|
|
|
|
- plagiarism detection
|
|
|
|
|
|
|
|
- advanced low-level evaluation flow configuration with high-level abstraction
|
|
|
|
|
|
|
|
layer for ordinary configuration cases
|
|
|
|
|
|
|
|
- use of modern technologies with state-of-the-art compilers
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@todo: some conclusion
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The ideas presented above are not completely new. There was a group of
|
|
|
|
### Related projects
|
|
|
|
students, who already implemented an evaluation solution for student's
|
|
|
|
|
|
|
|
homeworks in 2006. Its name is [CodEx - The Code
|
|
|
|
First of all, some code evaluating projects were found and examined. It is not
|
|
|
|
|
|
|
|
a complete list of such evaluators, but just a few projects which are used
|
|
|
|
|
|
|
|
these days and can be an inspiration for our project. Each project from the
|
|
|
|
|
|
|
|
list has a brief description and some key features mentioned.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### CodEx
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
There already is a grading solution at MFF UK, which was inplemented in 2006 by
|
|
|
|
|
|
|
|
group of students. Its name is [CodEx - The Code
|
|
|
|
Examiner](http://codex.ms.mff.cuni.cz/project/) and it has been used with some
|
|
|
|
Examiner](http://codex.ms.mff.cuni.cz/project/) and it has been used with some
|
|
|
|
improvements since then. The original plan was to use the system only for basic
|
|
|
|
improvements since then. The original plan was to use the system only for basic
|
|
|
|
programming courses, but there is demand for adapting it for many different
|
|
|
|
programming courses, but there is demand for adapting it for many different
|
|
|
@ -79,33 +147,6 @@ several drawbacks. The main ones are:
|
|
|
|
which have a more difficult evaluation chain than simple
|
|
|
|
which have a more difficult evaluation chain than simple
|
|
|
|
compilation/execution/evaluation provided by CodEx.
|
|
|
|
compilation/execution/evaluation provided by CodEx.
|
|
|
|
|
|
|
|
|
|
|
|
Supervisors and current users of CodEx gives us following ideas for
|
|
|
|
|
|
|
|
improvements:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- keep it simple
|
|
|
|
|
|
|
|
- localization (both UI and exercises)
|
|
|
|
|
|
|
|
- Markdown support for exercise texts
|
|
|
|
|
|
|
|
- tagging exercises and search by tags
|
|
|
|
|
|
|
|
- comments, comments, comments (exercises, tests, solutions, ...)
|
|
|
|
|
|
|
|
- edit student solution and privately resubmit it
|
|
|
|
|
|
|
|
- resubmit solution with saving all results
|
|
|
|
|
|
|
|
- mark one student solution as accepted (used for grading this assignment)
|
|
|
|
|
|
|
|
- command-line submit tool
|
|
|
|
|
|
|
|
- SIS integration for personal user data
|
|
|
|
|
|
|
|
- plagiarism detection
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
After considering all these facts, it is clear that CodEx cannot be used
|
|
|
|
|
|
|
|
anymore. The project is too old to just maintain it and extend for modern
|
|
|
|
|
|
|
|
technologies. Thus, it needs to be completely rewritten or another solution
|
|
|
|
|
|
|
|
must be found.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Related projects
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First of all, some code evaluating projects were found and examined. It is not
|
|
|
|
|
|
|
|
a complete list of such evaluators, but just a few projects which are used
|
|
|
|
|
|
|
|
these days and can be an inspiration for our project. Each project from the
|
|
|
|
|
|
|
|
list has a brief description and some key features mentioned.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Progtest
|
|
|
|
#### Progtest
|
|
|
|
|
|
|
|
|
|
|
|
[Progtest](https://progtest.fit.cvut.cz/) is private project from FIT ČVUT in
|
|
|
|
[Progtest](https://progtest.fit.cvut.cz/) is private project from FIT ČVUT in
|
|
|
@ -157,6 +198,11 @@ recruiters and also some universities.
|
|
|
|
|
|
|
|
|
|
|
|
### ReCodEx goals
|
|
|
|
### ReCodEx goals
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
After considering all these facts, it is clear that CodEx cannot be used
|
|
|
|
|
|
|
|
anymore. The project is too old to just maintain it and extend for modern
|
|
|
|
|
|
|
|
technologies. Thus, it needs to be completely rewritten or another solution
|
|
|
|
|
|
|
|
must be found.
|
|
|
|
|
|
|
|
|
|
|
|
From the research above, we set up several goals, which a new system should
|
|
|
|
From the research above, we set up several goals, which a new system should
|
|
|
|
have. They mostly reflect drawbacks of current version of CodEx and wishes of
|
|
|
|
have. They mostly reflect drawbacks of current version of CodEx and wishes of
|
|
|
|
MFF users. No existing tool fits our needs, for example no examined project
|
|
|
|
MFF users. No existing tool fits our needs, for example no examined project
|
|
|
|