|
|
|
@ -92,7 +92,7 @@ the university.
|
|
|
|
The major goal of this project is to create a grading application that will be
|
|
|
|
The major goal of this project is to create a grading application that will be
|
|
|
|
used for programming classes at the Faculty of Mathematics and Physics of the
|
|
|
|
used for programming classes at the Faculty of Mathematics and Physics of the
|
|
|
|
Charles University in Prague. However, the application should be designed in a
|
|
|
|
Charles University in Prague. However, the application should be designed in a
|
|
|
|
modular fashion to be easily extended ori even modified to make other ways of
|
|
|
|
modular fashion to be easily extended or even modified to make other ways of
|
|
|
|
usage possible.
|
|
|
|
usage possible.
|
|
|
|
|
|
|
|
|
|
|
|
The system should be capable of dynamic analysis of programming code. It means,
|
|
|
|
The system should be capable of dynamic analysis of programming code. It means,
|
|
|
|
@ -106,7 +106,7 @@ that following four basic steps have to be supported:
|
|
|
|
The project has a great starting point -- there is an old grading system
|
|
|
|
The project has a great starting point -- there is an old grading system
|
|
|
|
currently used at the university (CodEx), so its flaws and weaknesses can be
|
|
|
|
currently used at the university (CodEx), so its flaws and weaknesses can be
|
|
|
|
addressed. Furthermore, many teachers desire to use and test the new system and
|
|
|
|
addressed. Furthermore, many teachers desire to use and test the new system and
|
|
|
|
they are willing to consult ideas or problems during development.
|
|
|
|
they are willing to consult ideas or problems during development with us.
|
|
|
|
|
|
|
|
|
|
|
|
### Intended usage
|
|
|
|
### Intended usage
|
|
|
|
|
|
|
|
|
|
|
|
@ -114,7 +114,7 @@ The whole system is intended to help both teachers (supervisors) and students.
|
|
|
|
To achieve this, it is crucial to keep in mind typical usage scenarios of the
|
|
|
|
To achieve this, it is crucial to keep in mind typical usage scenarios of the
|
|
|
|
system and try to make these tasks as simple as possible.
|
|
|
|
system and try to make these tasks as simple as possible.
|
|
|
|
|
|
|
|
|
|
|
|
The system has a database of users. Each user has assigned one role, which
|
|
|
|
The system has a database of users. Each user has assigned a role, which
|
|
|
|
corresponds to his/her privileges. There are user groups reflecting structure of
|
|
|
|
corresponds to his/her privileges. There are user groups reflecting structure of
|
|
|
|
lectured courses. Groups can be hierarchically ordered to reflect additional
|
|
|
|
lectured courses. Groups can be hierarchically ordered to reflect additional
|
|
|
|
metadata such as the academic year. For example, a reasonable group hierarchy
|
|
|
|
metadata such as the academic year. For example, a reasonable group hierarchy
|
|
|
|
@ -135,8 +135,8 @@ entities are just for keeping the related groups together. The hierarchy
|
|
|
|
structure can be modified and altered to fit specific needs of the university or
|
|
|
|
structure can be modified and altered to fit specific needs of the university or
|
|
|
|
any other organization, even the flat structure (i.e., no hierarchy) is
|
|
|
|
any other organization, even the flat structure (i.e., no hierarchy) is
|
|
|
|
possible. One user can be part of multiple groups and on the other hand one
|
|
|
|
possible. One user can be part of multiple groups and on the other hand one
|
|
|
|
group can have multiple users. Each user has a specific role for every group in
|
|
|
|
group can have multiple users. Each user can have a specific role for every
|
|
|
|
which is a member.
|
|
|
|
group in which is a member, overriding his/her default role in this context.
|
|
|
|
|
|
|
|
|
|
|
|
Database of exercises (algorithmic problems) is another part of the project.
|
|
|
|
Database of exercises (algorithmic problems) is another part of the project.
|
|
|
|
Each exercise consists of a text in multiple language variants, an evaluation
|
|
|
|
Each exercise consists of a text in multiple language variants, an evaluation
|
|
|
|
@ -146,7 +146,7 @@ choose one of the available exercises and specifying additional properties. An
|
|
|
|
assignment has a deadline (optionally a second deadline), a maximum amount of
|
|
|
|
assignment has a deadline (optionally a second deadline), a maximum amount of
|
|
|
|
points, a configuration for calculating the final score, a maximum number of
|
|
|
|
points, a configuration for calculating the final score, a maximum number of
|
|
|
|
submissions, and a list of supported runtime environements (e.g., programming
|
|
|
|
submissions, and a list of supported runtime environements (e.g., programming
|
|
|
|
languages) including specific time and memory limits for the sandboxed tasks.
|
|
|
|
languages) including specific time and memory limits for each one.
|
|
|
|
|
|
|
|
|
|
|
|
Typical use cases for supported user roles are illustrated on following UML
|
|
|
|
Typical use cases for supported user roles are illustrated on following UML
|
|
|
|
diagram:
|
|
|
|
diagram:
|
|
|
|
@ -169,9 +169,9 @@ Java languages). Matching exercise configuration is then used for taking care of
|
|
|
|
evaluation process.
|
|
|
|
evaluation process.
|
|
|
|
|
|
|
|
|
|
|
|
There is a pool of worker computers dedicated to processing jobs. Some of them
|
|
|
|
There is a pool of worker computers dedicated to processing jobs. Some of them
|
|
|
|
may have different environment to allow testing programs in different
|
|
|
|
may have different environment to allow testing programs in more conditions.
|
|
|
|
conditions. Incoming jobs are scheduled to particular worker depending on its
|
|
|
|
Incoming jobs are scheduled to particular worker depending on its capabilities
|
|
|
|
capabilities and job requirements.
|
|
|
|
and job requirements.
|
|
|
|
|
|
|
|
|
|
|
|
Job processing itself stars with obtaining source files and job configuration.
|
|
|
|
Job processing itself stars with obtaining source files and job configuration.
|
|
|
|
The configuration is parsed into small tasks with simple piece of work.
|
|
|
|
The configuration is parsed into small tasks with simple piece of work.
|
|
|
|
@ -183,19 +183,19 @@ are saved.
|
|
|
|
Results from worker contains only output data from processed tasks (this could
|
|
|
|
Results from worker contains only output data from processed tasks (this could
|
|
|
|
be return value, consumed time, ...). On top of that, one value is calculated to
|
|
|
|
be return value, consumed time, ...). On top of that, one value is calculated to
|
|
|
|
express overall quality of the tested job. It is used as points for final
|
|
|
|
express overall quality of the tested job. It is used as points for final
|
|
|
|
student grading. Calculation method of this value may be different for each
|
|
|
|
student grading. Calculation method of this value may be different for each
|
|
|
|
assignment. Data presented back to users include overview of job parts (which
|
|
|
|
assignment. Data presented back to users include overview of job parts (which
|
|
|
|
succeeded and which failed, optionally with reason like "memory limit exceeded")
|
|
|
|
succeeded and which failed, optionally with reason like "memory limit exceeded")
|
|
|
|
and achieved score (amount of awarded points).
|
|
|
|
and achieved score (amount of awarded points).
|
|
|
|
|
|
|
|
|
|
|
|
## Requirements
|
|
|
|
## Requirements
|
|
|
|
|
|
|
|
|
|
|
|
There are bunch of different requirements for the system. Some of them are
|
|
|
|
There are bunch of different formal requirements for the system. Some of them
|
|
|
|
necessary for any system for source code evaluation. Some of them are specific
|
|
|
|
are necessary for any system for source code evaluation, some of them are
|
|
|
|
for university deployment and some of them arose during the ten year long
|
|
|
|
specific for university deployment and some of them arose during the ten year
|
|
|
|
lifetime of the old system. There are not many ways to improve CodEx
|
|
|
|
long lifetime of the old system. There are not many ways to improve CodEx
|
|
|
|
experience from the perspective of a student, but a lot of feature requests
|
|
|
|
experience from the perspective of a student, but a lot of feature requests come
|
|
|
|
came from administrators and supervisors. The ideas were gathered mostly our
|
|
|
|
from administrators and supervisors. The ideas were gathered mostly from our
|
|
|
|
personal experience with the system and from meetings with faculty staff
|
|
|
|
personal experience with the system and from meetings with faculty staff
|
|
|
|
involved with the current system.
|
|
|
|
involved with the current system.
|
|
|
|
|
|
|
|
|
|
|
|
@ -204,12 +204,13 @@ categories.
|
|
|
|
|
|
|
|
|
|
|
|
### System features
|
|
|
|
### System features
|
|
|
|
|
|
|
|
|
|
|
|
System features are requirements directly accessible to users of the system.
|
|
|
|
System features represents directly accessible functionality to users of the
|
|
|
|
They describe the evaluation system in general and also university addons
|
|
|
|
system. They describe the evaluation system in general and also university
|
|
|
|
(mostly administrative features).
|
|
|
|
addons (mostly administrative features).
|
|
|
|
|
|
|
|
|
|
|
|
#### Pure user requirements
|
|
|
|
#### Pure user requirements
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- users have their own account in the system
|
|
|
|
- system users can be members of multiple groups (reflecting courses or labs)
|
|
|
|
- system users can be members of multiple groups (reflecting courses or labs)
|
|
|
|
- there is a database of exercises; teachers can create exercises including
|
|
|
|
- there is a database of exercises; teachers can create exercises including
|
|
|
|
textual description, sample inputs and correct reference outputs (for example
|
|
|
|
textual description, sample inputs and correct reference outputs (for example
|
|
|
|
@ -218,8 +219,8 @@ They describe the evaluation system in general and also university addons
|
|
|
|
- there is a list of assigned exercises in each group and interface to submit a
|
|
|
|
- there is a list of assigned exercises in each group and interface to submit a
|
|
|
|
solution; teachers can assign an existing exercise to their class with some
|
|
|
|
solution; teachers can assign an existing exercise to their class with some
|
|
|
|
specific properties set (deadlines, etc.)
|
|
|
|
specific properties set (deadlines, etc.)
|
|
|
|
- there is a list of submitted solutions for each assignment with corresponding
|
|
|
|
- user can see a list of submitted solutions for each assignment with
|
|
|
|
results
|
|
|
|
corresponding results
|
|
|
|
- teachers can specify way of computation grading points which will be awarted
|
|
|
|
- teachers can specify way of computation grading points which will be awarted
|
|
|
|
to the students depending on the quality of his/her solution for each
|
|
|
|
to the students depending on the quality of his/her solution for each
|
|
|
|
assignment extra
|
|
|
|
assignment extra
|
|
|
|
@ -227,7 +228,7 @@ They describe the evaluation system in general and also university addons
|
|
|
|
including all submitted solutions; also, each of the solution can be manually
|
|
|
|
including all submitted solutions; also, each of the solution can be manually
|
|
|
|
reviewed, commented and assigned additional points (positive or negative)
|
|
|
|
reviewed, commented and assigned additional points (positive or negative)
|
|
|
|
- one particular solution can be marked as accepted (used for grading this
|
|
|
|
- one particular solution can be marked as accepted (used for grading this
|
|
|
|
assignment)
|
|
|
|
assignment); otherwise, the solution with most points is used
|
|
|
|
- teacher can edit student solution and privately resubmit it; optionally saving
|
|
|
|
- teacher can edit student solution and privately resubmit it; optionally saving
|
|
|
|
all results (including temporary ones)
|
|
|
|
all results (including temporary ones)
|
|
|
|
- localization of all texts (UI and exercises)
|
|
|
|
- localization of all texts (UI and exercises)
|
|
|
|
@ -241,14 +242,15 @@ They describe the evaluation system in general and also university addons
|
|
|
|
- users can use an intuitive user interface for interaction with the system,
|
|
|
|
- users can use an intuitive user interface for interaction with the system,
|
|
|
|
mainly for viewing assigned exercises, uploading their own solutions to the
|
|
|
|
mainly for viewing assigned exercises, uploading their own solutions to the
|
|
|
|
assignments, and viewing the results of the solutions after an automatic
|
|
|
|
assignments, and viewing the results of the solutions after an automatic
|
|
|
|
evaluation is finished; wanted two interfaces are web and command-line based
|
|
|
|
evaluation is finished; the two wanted interfaces are web and command-line
|
|
|
|
|
|
|
|
based
|
|
|
|
- user privilege separation (at least two roles -- _student_ and _supervisor_)
|
|
|
|
- user privilege separation (at least two roles -- _student_ and _supervisor_)
|
|
|
|
- logging in through a university authentication system (e.g. LDAP)
|
|
|
|
- logging in through a university authentication system (e.g. LDAP)
|
|
|
|
- SIS (university information system) integration for fetching personal user
|
|
|
|
- SIS (university information system) integration for fetching personal user
|
|
|
|
data
|
|
|
|
data
|
|
|
|
- safe environment in which the students' solutions are executed
|
|
|
|
- safe environment in which the students' solutions are executed
|
|
|
|
- support for multiple programming environments at once to avoid unacceptable
|
|
|
|
- support for multiple programming environments at once to avoid unacceptable
|
|
|
|
workload for administrator (maintain separate installations for many courses)
|
|
|
|
workload for administrator (maintain separate installation for every course)
|
|
|
|
and high hardware occupation
|
|
|
|
and high hardware occupation
|
|
|
|
- advanced low-level evaluation flow configuration with high-level abstraction
|
|
|
|
- advanced low-level evaluation flow configuration with high-level abstraction
|
|
|
|
layer for ordinary configuration cases
|
|
|
|
layer for ordinary configuration cases
|
|
|
|
@ -266,23 +268,23 @@ met. Most notably they are these ones:
|
|
|
|
- easy implementation of different user interfaces
|
|
|
|
- easy implementation of different user interfaces
|
|
|
|
- be ready for workload hundreds of students and tens of supervisors
|
|
|
|
- be ready for workload hundreds of students and tens of supervisors
|
|
|
|
- automated installation of all components
|
|
|
|
- automated installation of all components
|
|
|
|
|
|
|
|
- source code with permissive license allowing further development; this also
|
|
|
|
@todo: fill some nonfunctional requirements;
|
|
|
|
applies on used libraries and frameworks
|
|
|
|
melo byt lepe popsano, ze je potreba oddelit UI od aplikacni logiky, aby si
|
|
|
|
- multi-platform worker supporting at least two major operating systems
|
|
|
|
nekdo mohl napsat jine UI, ze je potreba mit backend, ktery bude skalovat na
|
|
|
|
|
|
|
|
vykon a dovoli pridavani dalsich platforem atd.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Conclusion
|
|
|
|
### Conclusion
|
|
|
|
|
|
|
|
|
|
|
|
The survey shows that the system is used in many different ways, but the core
|
|
|
|
The survey shows that there are a lot of different requirements and wishes for
|
|
|
|
functionality is the same for all of them. When the system is ready it is likely
|
|
|
|
the new system. When the system is ready it is likely that there will be new
|
|
|
|
that there will be new ideas of how to use the system and thus the system must
|
|
|
|
ideas of how to use the system and thus the system must be designed to be easily
|
|
|
|
be designed to be easily extendable, so everyone can develop their own feature.
|
|
|
|
extendable, so everyone can develop their own feature. This also means that
|
|
|
|
This also means that widely used programming languages and techniques should be
|
|
|
|
widely used programming languages and techniques should be used, so users can
|
|
|
|
used, so users can quickly understand the code and make changes.
|
|
|
|
quickly understand the code and make changes.
|
|
|
|
|
|
|
|
|
|
|
|
To find out the current state in the field of automatic grading systems we did a
|
|
|
|
To find out the current state in the field of automatic grading systems we did a
|
|
|
|
short survey at universities, programming contests, and other available tools.
|
|
|
|
short market survey on the field of automatic grading systems at universities,
|
|
|
|
|
|
|
|
programming contests, and possibly other places where similat tools are
|
|
|
|
|
|
|
|
available.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Related work
|
|
|
|
## Related work
|
|
|
|
|
Petr Stefan
8 years ago