Intro finished

master
Petr Stefan 8 years ago
parent 91fa5d0af1
commit b86567be3f

@ -92,7 +92,7 @@ the university.
The major goal of this project is to create a grading application that will be The major goal of this project is to create a grading application that will be
used for programming classes at the Faculty of Mathematics and Physics of the used for programming classes at the Faculty of Mathematics and Physics of the
Charles University in Prague. However, the application should be designed in a Charles University in Prague. However, the application should be designed in a
modular fashion to be easily extended ori even modified to make other ways of modular fashion to be easily extended or even modified to make other ways of
usage possible. usage possible.
The system should be capable of dynamic analysis of programming code. It means, The system should be capable of dynamic analysis of programming code. It means,
@ -106,7 +106,7 @@ that following four basic steps have to be supported:
The project has a great starting point -- there is an old grading system The project has a great starting point -- there is an old grading system
currently used at the university (CodEx), so its flaws and weaknesses can be currently used at the university (CodEx), so its flaws and weaknesses can be
addressed. Furthermore, many teachers desire to use and test the new system and addressed. Furthermore, many teachers desire to use and test the new system and
they are willing to consult ideas or problems during development. they are willing to consult ideas or problems during development with us.
### Intended usage ### Intended usage
@ -114,7 +114,7 @@ The whole system is intended to help both teachers (supervisors) and students.
To achieve this, it is crucial to keep in mind typical usage scenarios of the To achieve this, it is crucial to keep in mind typical usage scenarios of the
system and try to make these tasks as simple as possible. system and try to make these tasks as simple as possible.
The system has a database of users. Each user has assigned one role, which The system has a database of users. Each user has assigned a role, which
corresponds to his/her privileges. There are user groups reflecting structure of corresponds to his/her privileges. There are user groups reflecting structure of
lectured courses. Groups can be hierarchically ordered to reflect additional lectured courses. Groups can be hierarchically ordered to reflect additional
metadata such as the academic year. For example, a reasonable group hierarchy metadata such as the academic year. For example, a reasonable group hierarchy
@ -135,8 +135,8 @@ entities are just for keeping the related groups together. The hierarchy
structure can be modified and altered to fit specific needs of the university or structure can be modified and altered to fit specific needs of the university or
any other organization, even the flat structure (i.e., no hierarchy) is any other organization, even the flat structure (i.e., no hierarchy) is
possible. One user can be part of multiple groups and on the other hand one possible. One user can be part of multiple groups and on the other hand one
group can have multiple users. Each user has a specific role for every group in group can have multiple users. Each user can have a specific role for every
which is a member. group in which is a member, overriding his/her default role in this context.
Database of exercises (algorithmic problems) is another part of the project. Database of exercises (algorithmic problems) is another part of the project.
Each exercise consists of a text in multiple language variants, an evaluation Each exercise consists of a text in multiple language variants, an evaluation
@ -146,7 +146,7 @@ choose one of the available exercises and specifying additional properties. An
assignment has a deadline (optionally a second deadline), a maximum amount of assignment has a deadline (optionally a second deadline), a maximum amount of
points, a configuration for calculating the final score, a maximum number of points, a configuration for calculating the final score, a maximum number of
submissions, and a list of supported runtime environements (e.g., programming submissions, and a list of supported runtime environements (e.g., programming
languages) including specific time and memory limits for the sandboxed tasks. languages) including specific time and memory limits for each one.
Typical use cases for supported user roles are illustrated on following UML Typical use cases for supported user roles are illustrated on following UML
diagram: diagram:
@ -169,9 +169,9 @@ Java languages). Matching exercise configuration is then used for taking care of
evaluation process. evaluation process.
There is a pool of worker computers dedicated to processing jobs. Some of them There is a pool of worker computers dedicated to processing jobs. Some of them
may have different environment to allow testing programs in different may have different environment to allow testing programs in more conditions.
conditions. Incoming jobs are scheduled to particular worker depending on its Incoming jobs are scheduled to particular worker depending on its capabilities
capabilities and job requirements. and job requirements.
Job processing itself stars with obtaining source files and job configuration. Job processing itself stars with obtaining source files and job configuration.
The configuration is parsed into small tasks with simple piece of work. The configuration is parsed into small tasks with simple piece of work.
@ -183,19 +183,19 @@ are saved.
Results from worker contains only output data from processed tasks (this could Results from worker contains only output data from processed tasks (this could
be return value, consumed time, ...). On top of that, one value is calculated to be return value, consumed time, ...). On top of that, one value is calculated to
express overall quality of the tested job. It is used as points for final express overall quality of the tested job. It is used as points for final
student grading. Calculation method of this value may be different for each student grading. Calculation method of this value may be different for each
assignment. Data presented back to users include overview of job parts (which assignment. Data presented back to users include overview of job parts (which
succeeded and which failed, optionally with reason like "memory limit exceeded") succeeded and which failed, optionally with reason like "memory limit exceeded")
and achieved score (amount of awarded points). and achieved score (amount of awarded points).
## Requirements ## Requirements
There are bunch of different requirements for the system. Some of them are There are bunch of different formal requirements for the system. Some of them
necessary for any system for source code evaluation. Some of them are specific are necessary for any system for source code evaluation, some of them are
for university deployment and some of them arose during the ten year long specific for university deployment and some of them arose during the ten year
lifetime of the old system. There are not many ways to improve CodEx long lifetime of the old system. There are not many ways to improve CodEx
experience from the perspective of a student, but a lot of feature requests experience from the perspective of a student, but a lot of feature requests come
came from administrators and supervisors. The ideas were gathered mostly our from administrators and supervisors. The ideas were gathered mostly from our
personal experience with the system and from meetings with faculty staff personal experience with the system and from meetings with faculty staff
involved with the current system. involved with the current system.
@ -204,12 +204,13 @@ categories.
### System features ### System features
System features are requirements directly accessible to users of the system. System features represents directly accessible functionality to users of the
They describe the evaluation system in general and also university addons system. They describe the evaluation system in general and also university
(mostly administrative features). addons (mostly administrative features).
#### Pure user requirements #### Pure user requirements
- users have their own account in the system
- system users can be members of multiple groups (reflecting courses or labs) - system users can be members of multiple groups (reflecting courses or labs)
- there is a database of exercises; teachers can create exercises including - there is a database of exercises; teachers can create exercises including
textual description, sample inputs and correct reference outputs (for example textual description, sample inputs and correct reference outputs (for example
@ -218,8 +219,8 @@ They describe the evaluation system in general and also university addons
- there is a list of assigned exercises in each group and interface to submit a - there is a list of assigned exercises in each group and interface to submit a
solution; teachers can assign an existing exercise to their class with some solution; teachers can assign an existing exercise to their class with some
specific properties set (deadlines, etc.) specific properties set (deadlines, etc.)
- there is a list of submitted solutions for each assignment with corresponding - user can see a list of submitted solutions for each assignment with
results corresponding results
- teachers can specify way of computation grading points which will be awarted - teachers can specify way of computation grading points which will be awarted
to the students depending on the quality of his/her solution for each to the students depending on the quality of his/her solution for each
assignment extra assignment extra
@ -227,7 +228,7 @@ They describe the evaluation system in general and also university addons
including all submitted solutions; also, each of the solution can be manually including all submitted solutions; also, each of the solution can be manually
reviewed, commented and assigned additional points (positive or negative) reviewed, commented and assigned additional points (positive or negative)
- one particular solution can be marked as accepted (used for grading this - one particular solution can be marked as accepted (used for grading this
assignment) assignment); otherwise, the solution with most points is used
- teacher can edit student solution and privately resubmit it; optionally saving - teacher can edit student solution and privately resubmit it; optionally saving
all results (including temporary ones) all results (including temporary ones)
- localization of all texts (UI and exercises) - localization of all texts (UI and exercises)
@ -241,14 +242,15 @@ They describe the evaluation system in general and also university addons
- users can use an intuitive user interface for interaction with the system, - users can use an intuitive user interface for interaction with the system,
mainly for viewing assigned exercises, uploading their own solutions to the mainly for viewing assigned exercises, uploading their own solutions to the
assignments, and viewing the results of the solutions after an automatic assignments, and viewing the results of the solutions after an automatic
evaluation is finished; wanted two interfaces are web and command-line based evaluation is finished; the two wanted interfaces are web and command-line
based
- user privilege separation (at least two roles -- _student_ and _supervisor_) - user privilege separation (at least two roles -- _student_ and _supervisor_)
- logging in through a university authentication system (e.g. LDAP) - logging in through a university authentication system (e.g. LDAP)
- SIS (university information system) integration for fetching personal user - SIS (university information system) integration for fetching personal user
data data
- safe environment in which the students' solutions are executed - safe environment in which the students' solutions are executed
- support for multiple programming environments at once to avoid unacceptable - support for multiple programming environments at once to avoid unacceptable
workload for administrator (maintain separate installations for many courses) workload for administrator (maintain separate installation for every course)
and high hardware occupation and high hardware occupation
- advanced low-level evaluation flow configuration with high-level abstraction - advanced low-level evaluation flow configuration with high-level abstraction
layer for ordinary configuration cases layer for ordinary configuration cases
@ -266,23 +268,23 @@ met. Most notably they are these ones:
- easy implementation of different user interfaces - easy implementation of different user interfaces
- be ready for workload hundreds of students and tens of supervisors - be ready for workload hundreds of students and tens of supervisors
- automated installation of all components - automated installation of all components
- source code with permissive license allowing further development; this also
@todo: fill some nonfunctional requirements; applies on used libraries and frameworks
melo byt lepe popsano, ze je potreba oddelit UI od aplikacni logiky, aby si - multi-platform worker supporting at least two major operating systems
nekdo mohl napsat jine UI, ze je potreba mit backend, ktery bude skalovat na
vykon a dovoli pridavani dalsich platforem atd.
### Conclusion ### Conclusion
The survey shows that the system is used in many different ways, but the core The survey shows that there are a lot of different requirements and wishes for
functionality is the same for all of them. When the system is ready it is likely the new system. When the system is ready it is likely that there will be new
that there will be new ideas of how to use the system and thus the system must ideas of how to use the system and thus the system must be designed to be easily
be designed to be easily extendable, so everyone can develop their own feature. extendable, so everyone can develop their own feature. This also means that
This also means that widely used programming languages and techniques should be widely used programming languages and techniques should be used, so users can
used, so users can quickly understand the code and make changes. quickly understand the code and make changes.
To find out the current state in the field of automatic grading systems we did a To find out the current state in the field of automatic grading systems we did a
short survey at universities, programming contests, and other available tools. short market survey on the field of automatic grading systems at universities,
programming contests, and possibly other places where similat tools are
available.
## Related work ## Related work

Loading…
Cancel
Save