Updated structure of system requirements

master
Petr Stefan 8 years ago
parent 948c3ebccb
commit 58aeaf5cce

@ -78,18 +78,13 @@ the code without running it; safe, but not very precise) or dynamically (run the
code on testing inputs with checking the outputs against reference ones; needs code on testing inputs with checking the outputs against reference ones; needs
sandboxing, but provides good real world experience). sandboxing, but provides good real world experience).
<!---
*Simon*: I am not very sure about the formulation of 'our university' - shouldn't
we rather say 'The Charles University in Prague' instead?
-->
This project focuses on the machine-controlled part of source code evaluation. This project focuses on the machine-controlled part of source code evaluation.
First, the problems of the software used at our university previously were First, the problems of the software used at Charles University in Prague
discussed and similar projects at other educational institutions were examined. previously were discussed and similar projects at other educational institutions
With acquired knowledge from such projects in production, we set up goals for were examined. With acquired knowledge from such projects in production, we set
the new evaluation system, designed the architecture and implemented a fully up goals for the new evaluation system, designed the architecture and
operational solution. The system is now ready for production testing implemented a fully operational solution. The system is now ready for production
at our university. testing at our university.
## Assignment ## Assignment
@ -102,80 +97,87 @@ of using it possible.
The project has a great starting point -- there is an old grading system The project has a great starting point -- there is an old grading system
currently used at the university (CodEx), so its flaws and weaknesses can be currently used at the university (CodEx), so its flaws and weaknesses can be
addressed. Furthermore, many teachers are willing to use and test the new addressed. Furthermore, many teachers are willing to use and test the new
system. Following requirements were collected both from our personal experience system. Following requirements were collected both from our personal experience
with CodEx and from teachers' requests. with CodEx and from teachers' requests.
## Requirements ## Requirements
@todo: sekce kde budou ty pozadavky (o neco podrobneji nez ted). Navrhoval bych There are bunch of different requirements for the system. Some of them are
to rozdelit minimalne na funkcni a nefunkcni pozadavky, ale i ty funkcni je features which are necessary for any system for evaluation of programming coding
mozna vhodne rozdelit na ciste uzivatelske (ulohy se maji bodovat, studenti assignments. Some of them are specific for university deployment and some are
dostanou od cviciho komentar, ...) a administracne-technicke (napojeni na SIS a wishes for new features collected for period of CodEx operation.
LDAP, rizeni pristupu pomoci roli,...).
CodEx satisfies all the basic requirements and a few more that originate from
### Basic grading system requirements: the way courses are organized at university environment -- for example students
are divided into groups that correspond to lab groups. New wishes arose during
the ten year long lifetime of the old system. There are not many ways to improve
it from the perspective of a student, but a lot of feature requests came from
administrators and supervisors. The ideas were mostly gathered from meetings
with faculty staff involved with the current system.
For clear arragement all the requirements and wishes are presented grouped by
categories.
### System features
System features are requirements directly accessible to users of the system.
They describe the evaluation system in general and also university addons
(mostly administrative features).
#### Pure user requirements
- system users can be members of multiple groups (reflecting courses or labs)
- there is a database of exercises; teachers can create exercises including
textual description, sample inputs and correct reference outputs (for example
"sum all numbers from given file and write the result to the standard
output")
- there is a list of assigned exercises in each group and interface to submit a
solution; teachers can assign an existing exercise to their class with some
specific properties set (deadlines, etc.)
- there is a list of submitted solutions for each assignment with corresponding
results
- teachers can specify scale of points which will be awarted to the students
depending on the correctness of his/her solution for each assignment extra
(expressed in percentage points)
- teachers can view detailed data about their students (users of a their groups)
including all submitted solutions; also, each of the solution can be manually
reviewed, commented and assigned additional points (positive or negative)
- one particular solution can be marked as accepted (used for grading this
assignment)
- teacher can edit student solution and privately resubmit it; optionaly saving
all results (including temporary ones)
- localization of all texts (UI and exercises)
- Markdown support for creating exercise texts
- tagging exercises in database and search by these tags
- comments, comments, comments (exercises, tests, solutions, ...)
- plagiarism detection
These are the features which are necessary for any system for evaluation of #### Administrative requirements
programming coding assignments used in any university programming course:
- students can use an intuitive user interface for interaction with the system, - users can use an intuitive user interface for interaction with the system,
mainly for viewing assigned exercises, uploading their own solutions to the mainly for viewing assigned exercises, uploading their own solutions to the
assignments, and viewing the results of the solutions after an automatic assignments, and viewing the results of the solutions after an automatic
evaluation is finished evaluation is finished; wanted two interfaces are web and command-line based
- teachers can create exercises including textual description, sample inputs and
correct reference outputs (for example "sum all numbers from given file and
write the result to the standard output")
- teachers can assign an existing exercise to their class with some specific
properties set (deadlines, etc.)
- teachers can specify their scale of points which will be awarted to the
students depending on the correctness of his/her solution (expressed in
percentage points)
- teachers can view all of the solutions their students submitted and also the
results of the evaluations and they can override the automatically assigned
points to the solutions manually
- teachers can see the statistics of their classes and individual students of
these claseese
- administrators can depend on a safe environment in which the students'
solutions will be executed
- administrators can manage users with support of roles (at least two -- - administrators can manage users with support of roles (at least two --
_student_ and _supervisor_) _student_ and _supervisor_)
CodEx satisfies all these requirements and a few more that originate from the
way courses are organized at our university -- for example, users have roles
(_student_, _supervisor_ and _administrator_) that determine their capabilities
in the system and students are divided into groups that correspond to lab
groups.
However, further requirements arose during the ten year long lifetime of the old
system. There are not many ways to improve it from the perspective of a student,
but a lot of feature requests came from administrators and supervisors. The
ideas were mostly gathered from meetings with faculty staff involved with the
current system.
### Requested features for the new system:
First, there is a list of ideas influencing system capabilities:
- logging in through a university authentication system (e.g. LDAP) - logging in through a university authentication system (e.g. LDAP)
- SIS (university information system) integration for fetching personal user
data
- administrators can depend on a safe environment in which the students'
solutions will be executed
- support for multiple programming environments at once to avoid unacceptable - support for multiple programming environments at once to avoid unacceptable
workload for administrator (maintain separate installations for many courses) workload for administrator (maintain separate installations for many courses)
and high hardware occupation and high hardware occupation
- localization (both UI and exercises)
- Markdown support for exercise texts
- tagging exercises and search by tags
- comments, comments, comments (exercises, tests, solutions, ...)
- edit student solution and privately resubmit it
- resubmit solution with saving all results (including temporary ones)
- mark one student's solution as accepted (used for grading this assignment)
- web and command-line submission tool
- SIS (university information system) integration for fetching personal user data
- plagiarism detection
- advanced low-level evaluation flow configuration with high-level abstraction - advanced low-level evaluation flow configuration with high-level abstraction
layer for ordinary configuration cases layer for ordinary configuration cases
- use of modern technologies with state-of-the-art compilers - use of modern technologies with state-of-the-art compilers
Then, there are also some requirements of technical character with no direct ### Technical details
mapping to visible parts of system. Most notably they are these ones:
Technical details are requirements of technical character with no direct mapping
to visible parts of system. In ideal word, users should not know about these if
they work properly, but would be at least annoyed if these requirements were not
met. Most notably they are these ones:
- user interface of the system accessible on users' computers without - user interface of the system accessible on users' computers without
installation of any kind of additional software installation of any kind of additional software
@ -188,16 +190,19 @@ melo byt lepe popsano, ze je potreba oddelit UI od aplikacni logiky, aby si
nekdo mohl napsat jine UI, ze je potreba mit backend, ktery bude skalovat na nekdo mohl napsat jine UI, ze je potreba mit backend, ktery bude skalovat na
vykon a dovoli pridavani dalsich platforem atd. vykon a dovoli pridavani dalsich platforem atd.
### Conclusion
The survey shows that the system is used in many different ways, but the core The survey shows that the system is used in many different ways, but the core
functionality is the same for all of them. When the system is ready it is functionality is the same for all of them. When the system is ready it is likely
likely that there will be new ideas of how to use the system and thus the system that there will be new ideas of how to use the system and thus the system must
must be designed to be easily extendable, so everyone can develop their own feature. be designed to be easily extendable, so everyone can develop their own feature.
This also means that widely used programming languages and techniques should be used, This also means that widely used programming languages and techniques should be
so users can quickly understand the code and make changes. used, so users can quickly understand the code and make changes.
To find out the current state in the field of automatic grading systems we did a To find out the current state in the field of automatic grading systems we did a
short survey at universities, programming contests, and other available tools. short survey at universities, programming contests, and other available tools.
## Related work ## Related work
This is not a complete list of available evaluators, but only a few projects This is not a complete list of available evaluators, but only a few projects
@ -1663,9 +1668,9 @@ used.
name: "isolate" name: "isolate"
limits: limits:
- hw-group-id: group1 - hw-group-id: group1
chdir: ${EVAL_DIR} chdir: ${EVAL_DIR}
``` ```
<!--- <!---
// vim: set formatoptions=ant textwidth=80 colorcolumn=+1: // vim: set formatoptions=tqan flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
--> -->

Loading…
Cancel
Save