Intro finished

master
Petr Stefan 8 years ago
parent 91fa5d0af1
commit b86567be3f

@ -92,7 +92,7 @@ the university.
The major goal of this project is to create a grading application that will be
used for programming classes at the Faculty of Mathematics and Physics of the
Charles University in Prague. However, the application should be designed in a
modular fashion to be easily extended ori even modified to make other ways of
modular fashion to be easily extended or even modified to make other ways of
usage possible.
The system should be capable of dynamic analysis of programming code. It means,
@ -106,7 +106,7 @@ that following four basic steps have to be supported:
The project has a great starting point -- there is an old grading system
currently used at the university (CodEx), so its flaws and weaknesses can be
addressed. Furthermore, many teachers desire to use and test the new system and
they are willing to consult ideas or problems during development.
they are willing to consult ideas or problems during development with us.
### Intended usage
@ -114,7 +114,7 @@ The whole system is intended to help both teachers (supervisors) and students.
To achieve this, it is crucial to keep in mind typical usage scenarios of the
system and try to make these tasks as simple as possible.
The system has a database of users. Each user has assigned one role, which
The system has a database of users. Each user has assigned a role, which
corresponds to his/her privileges. There are user groups reflecting structure of
lectured courses. Groups can be hierarchically ordered to reflect additional
metadata such as the academic year. For example, a reasonable group hierarchy
@ -135,8 +135,8 @@ entities are just for keeping the related groups together. The hierarchy
structure can be modified and altered to fit specific needs of the university or
any other organization, even the flat structure (i.e., no hierarchy) is
possible. One user can be part of multiple groups and on the other hand one
group can have multiple users. Each user has a specific role for every group in
which is a member.
group can have multiple users. Each user can have a specific role for every
group in which is a member, overriding his/her default role in this context.
Database of exercises (algorithmic problems) is another part of the project.
Each exercise consists of a text in multiple language variants, an evaluation
@ -146,7 +146,7 @@ choose one of the available exercises and specifying additional properties. An
assignment has a deadline (optionally a second deadline), a maximum amount of
points, a configuration for calculating the final score, a maximum number of
submissions, and a list of supported runtime environements (e.g., programming
languages) including specific time and memory limits for the sandboxed tasks.
languages) including specific time and memory limits for each one.
Typical use cases for supported user roles are illustrated on following UML
diagram:
@ -169,9 +169,9 @@ Java languages). Matching exercise configuration is then used for taking care of
evaluation process.
There is a pool of worker computers dedicated to processing jobs. Some of them
may have different environment to allow testing programs in different
conditions. Incoming jobs are scheduled to particular worker depending on its
capabilities and job requirements.
may have different environment to allow testing programs in more conditions.
Incoming jobs are scheduled to particular worker depending on its capabilities
and job requirements.
Job processing itself stars with obtaining source files and job configuration.
The configuration is parsed into small tasks with simple piece of work.
@ -183,19 +183,19 @@ are saved.
Results from worker contains only output data from processed tasks (this could
be return value, consumed time, ...). On top of that, one value is calculated to
express overall quality of the tested job. It is used as points for final
student grading. Calculation method of this value may be different for each
student grading. Calculation method of this value may be different for each
assignment. Data presented back to users include overview of job parts (which
succeeded and which failed, optionally with reason like "memory limit exceeded")
and achieved score (amount of awarded points).
## Requirements
There are bunch of different requirements for the system. Some of them are
necessary for any system for source code evaluation. Some of them are specific
for university deployment and some of them arose during the ten year long
lifetime of the old system. There are not many ways to improve CodEx
experience from the perspective of a student, but a lot of feature requests
came from administrators and supervisors. The ideas were gathered mostly our
There are bunch of different formal requirements for the system. Some of them
are necessary for any system for source code evaluation, some of them are
specific for university deployment and some of them arose during the ten year
long lifetime of the old system. There are not many ways to improve CodEx
experience from the perspective of a student, but a lot of feature requests come
from administrators and supervisors. The ideas were gathered mostly from our
personal experience with the system and from meetings with faculty staff
involved with the current system.
@ -204,12 +204,13 @@ categories.
### System features
System features are requirements directly accessible to users of the system.
They describe the evaluation system in general and also university addons
(mostly administrative features).
System features represents directly accessible functionality to users of the
system. They describe the evaluation system in general and also university
addons (mostly administrative features).
#### Pure user requirements
- users have their own account in the system
- system users can be members of multiple groups (reflecting courses or labs)
- there is a database of exercises; teachers can create exercises including
textual description, sample inputs and correct reference outputs (for example
@ -218,8 +219,8 @@ They describe the evaluation system in general and also university addons
- there is a list of assigned exercises in each group and interface to submit a
solution; teachers can assign an existing exercise to their class with some
specific properties set (deadlines, etc.)
- there is a list of submitted solutions for each assignment with corresponding
results
- user can see a list of submitted solutions for each assignment with
corresponding results
- teachers can specify way of computation grading points which will be awarted
to the students depending on the quality of his/her solution for each
assignment extra
@ -227,7 +228,7 @@ They describe the evaluation system in general and also university addons
including all submitted solutions; also, each of the solution can be manually
reviewed, commented and assigned additional points (positive or negative)
- one particular solution can be marked as accepted (used for grading this
assignment)
assignment); otherwise, the solution with most points is used
- teacher can edit student solution and privately resubmit it; optionally saving
all results (including temporary ones)
- localization of all texts (UI and exercises)
@ -241,14 +242,15 @@ They describe the evaluation system in general and also university addons
- users can use an intuitive user interface for interaction with the system,
mainly for viewing assigned exercises, uploading their own solutions to the
assignments, and viewing the results of the solutions after an automatic
evaluation is finished; wanted two interfaces are web and command-line based
evaluation is finished; the two wanted interfaces are web and command-line
based
- user privilege separation (at least two roles -- _student_ and _supervisor_)
- logging in through a university authentication system (e.g. LDAP)
- SIS (university information system) integration for fetching personal user
data
- safe environment in which the students' solutions are executed
- support for multiple programming environments at once to avoid unacceptable
workload for administrator (maintain separate installations for many courses)
workload for administrator (maintain separate installation for every course)
and high hardware occupation
- advanced low-level evaluation flow configuration with high-level abstraction
layer for ordinary configuration cases
@ -266,23 +268,23 @@ met. Most notably they are these ones:
- easy implementation of different user interfaces
- be ready for workload hundreds of students and tens of supervisors
- automated installation of all components
@todo: fill some nonfunctional requirements;
melo byt lepe popsano, ze je potreba oddelit UI od aplikacni logiky, aby si
nekdo mohl napsat jine UI, ze je potreba mit backend, ktery bude skalovat na
vykon a dovoli pridavani dalsich platforem atd.
- source code with permissive license allowing further development; this also
applies on used libraries and frameworks
- multi-platform worker supporting at least two major operating systems
### Conclusion
The survey shows that the system is used in many different ways, but the core
functionality is the same for all of them. When the system is ready it is likely
that there will be new ideas of how to use the system and thus the system must
be designed to be easily extendable, so everyone can develop their own feature.
This also means that widely used programming languages and techniques should be
used, so users can quickly understand the code and make changes.
The survey shows that there are a lot of different requirements and wishes for
the new system. When the system is ready it is likely that there will be new
ideas of how to use the system and thus the system must be designed to be easily
extendable, so everyone can develop their own feature. This also means that
widely used programming languages and techniques should be used, so users can
quickly understand the code and make changes.
To find out the current state in the field of automatic grading systems we did a
short survey at universities, programming contests, and other available tools.
short market survey on the field of automatic grading systems at universities,
programming contests, and possibly other places where similat tools are
available.
## Related work

Loading…
Cancel
Save