System usage

master
Petr Stefan 8 years ago
parent 21f12fcd19
commit 10b06273a0

@ -44,7 +44,7 @@ University. However, the application should be designed in a modular fashion so
that it can be easily extended to make other ways of using it possible. that it can be easily extended to make other ways of using it possible.
The project has a great starting point -- there is an old grading system The project has a great starting point -- there is an old grading system
currently used at our university, so its mistakes and weaknesses can be currently used at our university (CodEx), so its mistakes and weaknesses can be
adressed. Furthermore, many teachers are willing to use and test the new system. adressed. Furthermore, many teachers are willing to use and test the new system.
Following requirements were collected both from our personal experience with Following requirements were collected both from our personal experience with
CodEx and from teachers' requests. CodEx and from teachers' requests.
@ -54,7 +54,10 @@ CodEx and from teachers' requests.
These are features that are necessary for any system for evaluation of These are features that are necessary for any system for evaluation of
programming homework assignments used in a university programming course. programming homework assignments used in a university programming course.
<!---
@todo maybe group the requirements by role (student might want to do XYZ...) @todo maybe group the requirements by role (student might want to do XYZ...)
- it's ok as is (unless requested differently)
-->
- creating exercises including textual description, sample inputs and correct - creating exercises including textual description, sample inputs and correct
reference outputs (for example "sum all numbers from given file and write the reference outputs (for example "sum all numbers from given file and write the
@ -84,27 +87,18 @@ but a lot of feature requests came from administrators and supervisors.
Collected ideas were mostly gathered from meetings with faculty staff involved Collected ideas were mostly gathered from meetings with faculty staff involved
with the current system. with the current system.
**Drawbacks of CodEx:**
@todo merge this with the CodEx entry in "Related work"
- there has to be a separate installation of the system for some courses (e.g.
Java programming and .NET programming) that do not use the same sandbox as the
majority of courses.
**Requested features for the new system:** **Requested features for the new system:**
- logging in through a university authentication system (e.g. LDAP) - logging in through a university authentication system (e.g. LDAP)
- support for multiple programming environments at once to avoid unacceptable - support for multiple programming environments at once to avoid unacceptable
workload for administrator and hardware occupation workload for administrator (maintain separate installations for many courses)
- elimination of the need to maintain separate installations for Java and C# and high hardware occupation
programming courses
- localization (both UI and exercises) - localization (both UI and exercises)
- Markdown support for exercise texts - Markdown support for exercise texts
- tagging exercises and search by tags - tagging exercises and search by tags
- comments, comments, comments (exercises, tests, solutions, ...) - comments, comments, comments (exercises, tests, solutions, ...)
- edit student solution and privately resubmit it - edit student solution and privately resubmit it
- resubmit solution with saving all (including temporary) results - resubmit solution with saving all results (including temporary ones)
- mark one student solution as accepted (used for grading this assignment) - mark one student solution as accepted (used for grading this assignment)
- web and command-line submit tool - web and command-line submit tool
- SIS (university information system) integration for fetching personal user - SIS (university information system) integration for fetching personal user
@ -115,11 +109,11 @@ with the current system.
- use of modern technologies with state-of-the-art compilers - use of modern technologies with state-of-the-art compilers
The survey shows that the system is used in many different ways, but the core The survey shows that the system is used in many different ways, but the core
functionality is the same for all. When the system is ready, it is likely that functionality is the same for all of them. When the system is ready, it is
new ideas are developed. Thus the system must be designed to be easily likely that new ideas are figured out, thus the system must be designed to be
extendable, so everyone can develop his dream feature. This also means, that easily extendable, so everyone can develop his dream feature. This also means,
widely used programming languages and techniques should be used, so users can that widely used programming languages and techniques should be used, so users
quickly understand the code and make changes. can quickly understand the code and make changes.
To find out current state in the field of automatic grading systems, let's do a To find out current state in the field of automatic grading systems, let's do a
short survey at universities, programming contests or online tools. short survey at universities, programming contests or online tools.
@ -239,9 +233,12 @@ to CodEx, previous evaluation solution, but also reflect new approach to solve
issues. **Re** as part of the name means redesigned, rewritten, renewed or issues. **Re** as part of the name means redesigned, rewritten, renewed or
restarted. restarted.
From the previous research, we set up several goals, which a new system should At this point there is a clear idea how the new system will be used and what are
have. They mostly reflect drawbacks of current version of CodEx and reasonable major enhancements for future releases. With this in mind, the overall
wishes of university users. Most notable features are following: architecture can be sketched. From the previous research, we set up several
goals, which a new system should have. They mostly reflect drawbacks of current
version of CodEx and reasonable wishes of university users. Most notable
features are following:
- modern HTML5 web frontend written in Javascript using a suitable framework - modern HTML5 web frontend written in Javascript using a suitable framework
- REST API implemented in PHP, communicating with database, backend and file - REST API implemented in PHP, communicating with database, backend and file
@ -253,9 +250,88 @@ wishes of university users. Most notable features are following:
- evaluation procedure configured in YAML file, compound of small tasks - evaluation procedure configured in YAML file, compound of small tasks
connected into arbitrary oriented acyclic graph connected into arbitrary oriented acyclic graph
#### Usage design #### Intended usage
Whole system is intended to help both supervisors and students. To achieve this,
it is crucial to keep in mind typical usage scenarios of the system and try to
make these typical tasks as simple as possible. To synchronize visions of
readers, basic concepts are recapitulated.
First of all, the system has database of users. Each user has assigned a role,
which correspond to his/her privileges. User can be logged in via local
authentication service or university system. There are groups of users, which
corresponds to lectured courses. Groups can be hierarchically ordered to reflect
additional metadata like academic year. For example, reasonable group hierarchy
is like this:
```
Summer term 2016
├── Language C# and .NET platform
│   ├── Labs Monday 10:30
│   └── Labs Thursday 9:00
├── Programming I
│   ├── Labs Monday 14:00
...
```
@todo: describe detailed usage ... grading and this kind of stuff In this example, student users are part of the leaf groups, higher groups are
just for keeping related groups together. The hierarchy tree can be modified and
altered to fit specific needs for each organization, even the flat structure is
possible.
One user can be part of multiple groups and also one group can have multiple
users. Each user in a group has a role which defines its capabilities.
Priviledged user can assign a new exercise in his/her group, change assignment
details, view results of other users and manually change them. Normal user can
join a group, get list of assigned exercises, view assignment detail, submit
his/her solution and of course view the results.
Database of exercises (algorithmic problems) is another part of the project.
Each exercise consists of text in multiple language variants, evaluation
configuration and set of inputs and reference outputs. Exercises are created by
instructed priviledged users. Assigning exercise to a group means choose one of
the exercises in the list and specify additional data. Assignment has a
deadline, maximum amount of points and configuration for calculating the final
amount, number of tries and supported runtimes (programming languages) including
specific time and memory limits for sandboxed tasks.
##### Exercise evaluation chain
The most important part of the application is evaluating exercises for solutions
submitted by users. For imaginary system architecture _UI_, _API_, _Broker_ and
_Worker_ this goes as follows.
First thing users have to do is to submit their solutions to _UI_ which provides
interface to upload files and then submit them. _UI_ sends a request to _API_
that user wants to evaluate assignment with provided files.
_API_ checks the assignment invariants (deadlines, count of submissions, ...)
and stores submitted files. The runtime environment is automatically detected
based on input files and suitable exercise configuration variant is chosen (one
exercise can have multiple variants, for example C and Java languages). Matching
exercise configuration is then send to _Broker_ alongside solution source files.
_Broker_ has to find suitable _Worker_ for execution of this particular
submission. This decission is made based on capabilities of each _Worker_ and
job requirements. When a match is found, the job is held until the _Worker_ is
jobless and can receive an evaluation request.
_Worker_ gets evaluation request with source files and job configuration. The
configuration is parsed into small tasks with simple piece of work. Evaluation
itself goes in direction of tasks ordering. It is crucial to keep _Worker_
machine secure and stable, so isolated sandboxed environment is used when
dealing with unknown source code. When the execution is finished, results are
uploaded back.
_API_ is notified about finished job from _Broker_. The results are parsed and
results of important tasks (comparing actual and expected results) saved into
database. Also, points are calculated depending on solution correctness and
assignment configuration.
_UI_ then only displays results summary fetched from the _API_. Presented data
includes overview which part succeeded and which failed (optionally with reason
like "memory limit exceeded") and amount of awarded points.
### Solution concepts analysis ### Solution concepts analysis
@ -1159,4 +1235,7 @@ used.
- hw-group-id: group1 - hw-group-id: group1
chdir: ${EVAL_DIR} chdir: ${EVAL_DIR}
``` ```
<!---
// vim: set textwidth=80:
-->

Loading…
Cancel
Save