|
|
@ -78,18 +78,13 @@ the code without running it; safe, but not very precise) or dynamically (run the
|
|
|
|
code on testing inputs with checking the outputs against reference ones; needs
|
|
|
|
code on testing inputs with checking the outputs against reference ones; needs
|
|
|
|
sandboxing, but provides good real world experience).
|
|
|
|
sandboxing, but provides good real world experience).
|
|
|
|
|
|
|
|
|
|
|
|
<!---
|
|
|
|
|
|
|
|
*Simon*: I am not very sure about the formulation of 'our university' - shouldn't
|
|
|
|
|
|
|
|
we rather say 'The Charles University in Prague' instead?
|
|
|
|
|
|
|
|
-->
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This project focuses on the machine-controlled part of source code evaluation.
|
|
|
|
This project focuses on the machine-controlled part of source code evaluation.
|
|
|
|
First, the problems of the software used at our university previously were
|
|
|
|
First, the problems of the software used at Charles University in Prague
|
|
|
|
discussed and similar projects at other educational institutions were examined.
|
|
|
|
previously were discussed and similar projects at other educational institutions
|
|
|
|
With acquired knowledge from such projects in production, we set up goals for
|
|
|
|
were examined. With acquired knowledge from such projects in production, we set
|
|
|
|
the new evaluation system, designed the architecture and implemented a fully
|
|
|
|
up goals for the new evaluation system, designed the architecture and
|
|
|
|
operational solution. The system is now ready for production testing
|
|
|
|
implemented a fully operational solution. The system is now ready for production
|
|
|
|
at our university.
|
|
|
|
testing at our university.
|
|
|
|
|
|
|
|
|
|
|
|
## Assignment
|
|
|
|
## Assignment
|
|
|
|
|
|
|
|
|
|
|
@ -102,80 +97,87 @@ of using it possible.
|
|
|
|
The project has a great starting point -- there is an old grading system
|
|
|
|
The project has a great starting point -- there is an old grading system
|
|
|
|
currently used at the university (CodEx), so its flaws and weaknesses can be
|
|
|
|
currently used at the university (CodEx), so its flaws and weaknesses can be
|
|
|
|
addressed. Furthermore, many teachers are willing to use and test the new
|
|
|
|
addressed. Furthermore, many teachers are willing to use and test the new
|
|
|
|
system. Following requirements were collected both from our personal experience
|
|
|
|
system. Following requirements were collected both from our personal experience
|
|
|
|
with CodEx and from teachers' requests.
|
|
|
|
with CodEx and from teachers' requests.
|
|
|
|
|
|
|
|
|
|
|
|
## Requirements
|
|
|
|
## Requirements
|
|
|
|
|
|
|
|
|
|
|
|
@todo: sekce kde budou ty pozadavky (o neco podrobneji nez ted). Navrhoval bych
|
|
|
|
There are bunch of different requirements for the system. Some of them are
|
|
|
|
to rozdelit minimalne na funkcni a nefunkcni pozadavky, ale i ty funkcni je
|
|
|
|
features which are necessary for any system for evaluation of programming coding
|
|
|
|
mozna vhodne rozdelit na ciste uzivatelske (ulohy se maji bodovat, studenti
|
|
|
|
assignments. Some of them are specific for university deployment and some are
|
|
|
|
dostanou od cviciho komentar, ...) a administracne-technicke (napojeni na SIS a
|
|
|
|
wishes for new features collected for period of CodEx operation.
|
|
|
|
LDAP, rizeni pristupu pomoci roli,...).
|
|
|
|
|
|
|
|
|
|
|
|
CodEx satisfies all the basic requirements and a few more that originate from
|
|
|
|
### Basic grading system requirements:
|
|
|
|
the way courses are organized at university environment -- for example students
|
|
|
|
|
|
|
|
are divided into groups that correspond to lab groups. New wishes arose during
|
|
|
|
|
|
|
|
the ten year long lifetime of the old system. There are not many ways to improve
|
|
|
|
|
|
|
|
it from the perspective of a student, but a lot of feature requests came from
|
|
|
|
|
|
|
|
administrators and supervisors. The ideas were mostly gathered from meetings
|
|
|
|
|
|
|
|
with faculty staff involved with the current system.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
For clear arragement all the requirements and wishes are presented grouped by
|
|
|
|
|
|
|
|
categories.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### System features
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
System features are requirements directly accessible to users of the system.
|
|
|
|
|
|
|
|
They describe the evaluation system in general and also university addons
|
|
|
|
|
|
|
|
(mostly administrative features).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Pure user requirements
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- system users can be members of multiple groups (reflecting courses or labs)
|
|
|
|
|
|
|
|
- there is a database of exercises; teachers can create exercises including
|
|
|
|
|
|
|
|
textual description, sample inputs and correct reference outputs (for example
|
|
|
|
|
|
|
|
"sum all numbers from given file and write the result to the standard
|
|
|
|
|
|
|
|
output")
|
|
|
|
|
|
|
|
- there is a list of assigned exercises in each group and interface to submit a
|
|
|
|
|
|
|
|
solution; teachers can assign an existing exercise to their class with some
|
|
|
|
|
|
|
|
specific properties set (deadlines, etc.)
|
|
|
|
|
|
|
|
- there is a list of submitted solutions for each assignment with corresponding
|
|
|
|
|
|
|
|
results
|
|
|
|
|
|
|
|
- teachers can specify scale of points which will be awarted to the students
|
|
|
|
|
|
|
|
depending on the correctness of his/her solution for each assignment extra
|
|
|
|
|
|
|
|
(expressed in percentage points)
|
|
|
|
|
|
|
|
- teachers can view detailed data about their students (users of a their groups)
|
|
|
|
|
|
|
|
including all submitted solutions; also, each of the solution can be manually
|
|
|
|
|
|
|
|
reviewed, commented and assigned additional points (positive or negative)
|
|
|
|
|
|
|
|
- one particular solution can be marked as accepted (used for grading this
|
|
|
|
|
|
|
|
assignment)
|
|
|
|
|
|
|
|
- teacher can edit student solution and privately resubmit it; optionaly saving
|
|
|
|
|
|
|
|
all results (including temporary ones)
|
|
|
|
|
|
|
|
- localization of all texts (UI and exercises)
|
|
|
|
|
|
|
|
- Markdown support for creating exercise texts
|
|
|
|
|
|
|
|
- tagging exercises in database and search by these tags
|
|
|
|
|
|
|
|
- comments, comments, comments (exercises, tests, solutions, ...)
|
|
|
|
|
|
|
|
- plagiarism detection
|
|
|
|
|
|
|
|
|
|
|
|
These are the features which are necessary for any system for evaluation of
|
|
|
|
#### Administrative requirements
|
|
|
|
programming coding assignments used in any university programming course:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- students can use an intuitive user interface for interaction with the system,
|
|
|
|
- users can use an intuitive user interface for interaction with the system,
|
|
|
|
mainly for viewing assigned exercises, uploading their own solutions to the
|
|
|
|
mainly for viewing assigned exercises, uploading their own solutions to the
|
|
|
|
assignments, and viewing the results of the solutions after an automatic
|
|
|
|
assignments, and viewing the results of the solutions after an automatic
|
|
|
|
evaluation is finished
|
|
|
|
evaluation is finished; wanted two interfaces are web and command-line based
|
|
|
|
- teachers can create exercises including textual description, sample inputs and
|
|
|
|
|
|
|
|
correct reference outputs (for example "sum all numbers from given file and
|
|
|
|
|
|
|
|
write the result to the standard output")
|
|
|
|
|
|
|
|
- teachers can assign an existing exercise to their class with some specific
|
|
|
|
|
|
|
|
properties set (deadlines, etc.)
|
|
|
|
|
|
|
|
- teachers can specify their scale of points which will be awarted to the
|
|
|
|
|
|
|
|
students depending on the correctness of his/her solution (expressed in
|
|
|
|
|
|
|
|
percentage points)
|
|
|
|
|
|
|
|
- teachers can view all of the solutions their students submitted and also the
|
|
|
|
|
|
|
|
results of the evaluations and they can override the automatically assigned
|
|
|
|
|
|
|
|
points to the solutions manually
|
|
|
|
|
|
|
|
- teachers can see the statistics of their classes and individual students of
|
|
|
|
|
|
|
|
these claseese
|
|
|
|
|
|
|
|
- administrators can depend on a safe environment in which the students'
|
|
|
|
|
|
|
|
solutions will be executed
|
|
|
|
|
|
|
|
- administrators can manage users with support of roles (at least two --
|
|
|
|
- administrators can manage users with support of roles (at least two --
|
|
|
|
_student_ and _supervisor_)
|
|
|
|
_student_ and _supervisor_)
|
|
|
|
|
|
|
|
|
|
|
|
CodEx satisfies all these requirements and a few more that originate from the
|
|
|
|
|
|
|
|
way courses are organized at our university -- for example, users have roles
|
|
|
|
|
|
|
|
(_student_, _supervisor_ and _administrator_) that determine their capabilities
|
|
|
|
|
|
|
|
in the system and students are divided into groups that correspond to lab
|
|
|
|
|
|
|
|
groups.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
However, further requirements arose during the ten year long lifetime of the old
|
|
|
|
|
|
|
|
system. There are not many ways to improve it from the perspective of a student,
|
|
|
|
|
|
|
|
but a lot of feature requests came from administrators and supervisors. The
|
|
|
|
|
|
|
|
ideas were mostly gathered from meetings with faculty staff involved with the
|
|
|
|
|
|
|
|
current system.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Requested features for the new system:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First, there is a list of ideas influencing system capabilities:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- logging in through a university authentication system (e.g. LDAP)
|
|
|
|
- logging in through a university authentication system (e.g. LDAP)
|
|
|
|
|
|
|
|
- SIS (university information system) integration for fetching personal user
|
|
|
|
|
|
|
|
data
|
|
|
|
|
|
|
|
- administrators can depend on a safe environment in which the students'
|
|
|
|
|
|
|
|
solutions will be executed
|
|
|
|
- support for multiple programming environments at once to avoid unacceptable
|
|
|
|
- support for multiple programming environments at once to avoid unacceptable
|
|
|
|
workload for administrator (maintain separate installations for many courses)
|
|
|
|
workload for administrator (maintain separate installations for many courses)
|
|
|
|
and high hardware occupation
|
|
|
|
and high hardware occupation
|
|
|
|
- localization (both UI and exercises)
|
|
|
|
|
|
|
|
- Markdown support for exercise texts
|
|
|
|
|
|
|
|
- tagging exercises and search by tags
|
|
|
|
|
|
|
|
- comments, comments, comments (exercises, tests, solutions, ...)
|
|
|
|
|
|
|
|
- edit student solution and privately resubmit it
|
|
|
|
|
|
|
|
- resubmit solution with saving all results (including temporary ones)
|
|
|
|
|
|
|
|
- mark one student's solution as accepted (used for grading this assignment)
|
|
|
|
|
|
|
|
- web and command-line submission tool
|
|
|
|
|
|
|
|
- SIS (university information system) integration for fetching personal user data
|
|
|
|
|
|
|
|
- plagiarism detection
|
|
|
|
|
|
|
|
- advanced low-level evaluation flow configuration with high-level abstraction
|
|
|
|
- advanced low-level evaluation flow configuration with high-level abstraction
|
|
|
|
layer for ordinary configuration cases
|
|
|
|
layer for ordinary configuration cases
|
|
|
|
- use of modern technologies with state-of-the-art compilers
|
|
|
|
- use of modern technologies with state-of-the-art compilers
|
|
|
|
|
|
|
|
|
|
|
|
Then, there are also some requirements of technical character with no direct
|
|
|
|
### Technical details
|
|
|
|
mapping to visible parts of system. Most notably they are these ones:
|
|
|
|
|
|
|
|
|
|
|
|
Technical details are requirements of technical character with no direct mapping
|
|
|
|
|
|
|
|
to visible parts of system. In ideal word, users should not know about these if
|
|
|
|
|
|
|
|
they work properly, but would be at least annoyed if these requirements were not
|
|
|
|
|
|
|
|
met. Most notably they are these ones:
|
|
|
|
|
|
|
|
|
|
|
|
- user interface of the system accessible on users' computers without
|
|
|
|
- user interface of the system accessible on users' computers without
|
|
|
|
installation of any kind of additional software
|
|
|
|
installation of any kind of additional software
|
|
|
@ -188,16 +190,19 @@ melo byt lepe popsano, ze je potreba oddelit UI od aplikacni logiky, aby si
|
|
|
|
nekdo mohl napsat jine UI, ze je potreba mit backend, ktery bude skalovat na
|
|
|
|
nekdo mohl napsat jine UI, ze je potreba mit backend, ktery bude skalovat na
|
|
|
|
vykon a dovoli pridavani dalsich platforem atd.
|
|
|
|
vykon a dovoli pridavani dalsich platforem atd.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Conclusion
|
|
|
|
|
|
|
|
|
|
|
|
The survey shows that the system is used in many different ways, but the core
|
|
|
|
The survey shows that the system is used in many different ways, but the core
|
|
|
|
functionality is the same for all of them. When the system is ready it is
|
|
|
|
functionality is the same for all of them. When the system is ready it is likely
|
|
|
|
likely that there will be new ideas of how to use the system and thus the system
|
|
|
|
that there will be new ideas of how to use the system and thus the system must
|
|
|
|
must be designed to be easily extendable, so everyone can develop their own feature.
|
|
|
|
be designed to be easily extendable, so everyone can develop their own feature.
|
|
|
|
This also means that widely used programming languages and techniques should be used,
|
|
|
|
This also means that widely used programming languages and techniques should be
|
|
|
|
so users can quickly understand the code and make changes.
|
|
|
|
used, so users can quickly understand the code and make changes.
|
|
|
|
|
|
|
|
|
|
|
|
To find out the current state in the field of automatic grading systems we did a
|
|
|
|
To find out the current state in the field of automatic grading systems we did a
|
|
|
|
short survey at universities, programming contests, and other available tools.
|
|
|
|
short survey at universities, programming contests, and other available tools.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Related work
|
|
|
|
## Related work
|
|
|
|
|
|
|
|
|
|
|
|
This is not a complete list of available evaluators, but only a few projects
|
|
|
|
This is not a complete list of available evaluators, but only a few projects
|
|
|
@ -1663,9 +1668,9 @@ used.
|
|
|
|
name: "isolate"
|
|
|
|
name: "isolate"
|
|
|
|
limits:
|
|
|
|
limits:
|
|
|
|
- hw-group-id: group1
|
|
|
|
- hw-group-id: group1
|
|
|
|
chdir: ${EVAL_DIR}
|
|
|
|
chdir: ${EVAL_DIR}
|
|
|
|
```
|
|
|
|
```
|
|
|
|
<!---
|
|
|
|
<!---
|
|
|
|
// vim: set formatoptions=ant textwidth=80 colorcolumn=+1:
|
|
|
|
// vim: set formatoptions=tqan flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
|
|
|
|
-->
|
|
|
|
-->
|
|
|
|
|
|
|
|
|
|
|
|