|
|
@ -197,10 +197,12 @@ users who will be assigned the same set exercises. Each group can have
|
|
|
|
multiple supervisors who can manage the students and the list of
|
|
|
|
multiple supervisors who can manage the students and the list of
|
|
|
|
assignments.
|
|
|
|
assignments.
|
|
|
|
|
|
|
|
|
|
|
|
Groups can form a tree hierarchy. This is inspired by the hierarchy of
|
|
|
|
Groups can form a tree hierarchy of arbitrary depth. This is inspired by the
|
|
|
|
school classes belonging to the same subject over several school years.
|
|
|
|
hierarchy of school classes belonging to the same subject over several school
|
|
|
|
Supervisors can create subgroups of their groups and further manage
|
|
|
|
years. For example, there can be a top level group for a programming class that
|
|
|
|
these subgroups.
|
|
|
|
contains subgroups for every school year. These groups can then by divided into
|
|
|
|
|
|
|
|
actual student groups with respect to lab attendance. Supervisors can create
|
|
|
|
|
|
|
|
subgroups of their groups and further manage these subgroups.
|
|
|
|
|
|
|
|
|
|
|
|
### Exercise
|
|
|
|
### Exercise
|
|
|
|
|
|
|
|
|
|
|
@ -228,8 +230,8 @@ language.
|
|
|
|
### Submission
|
|
|
|
### Submission
|
|
|
|
|
|
|
|
|
|
|
|
*A submission* corresponds to a *solution* being evaluated by the
|
|
|
|
*A submission* corresponds to a *solution* being evaluated by the
|
|
|
|
Backend. A single *solution* can be submitted repeatedly (i.e., when the
|
|
|
|
Backend. A single *solution* can be submitted repeatedly (e.g., when the
|
|
|
|
Backend encounters an error).
|
|
|
|
Backend encounters an error or when the supervisor changes the assignment).
|
|
|
|
|
|
|
|
|
|
|
|
### Evaluation
|
|
|
|
### Evaluation
|
|
|
|
|
|
|
|
|
|
|
@ -249,12 +251,12 @@ runtime environment can be “*Linux + GCC*”, “*Linux + Mono*”, “*Window
|
|
|
|
|
|
|
|
|
|
|
|
### Limits
|
|
|
|
### Limits
|
|
|
|
|
|
|
|
|
|
|
|
A correct *solution* of an *assignment* must obviously master the
|
|
|
|
A correct *solution* of an *assignment* has to pass all specified tests (mostly
|
|
|
|
expected behavior and typically must also be effective in some sense.
|
|
|
|
checks that it yields the correct output for various inputs) and typically must
|
|
|
|
The Backend measures the time and memory consumption of the solution
|
|
|
|
also be effective in some sense. The Backend measures the time and memory
|
|
|
|
while running. This consumption of resources can be *limited* and the
|
|
|
|
consumption of the solution while running. This consumption of resources can be
|
|
|
|
solution will receive fewer points if it exceeds the given limits in
|
|
|
|
*limited* and the solution will receive fewer points if it exceeds the given
|
|
|
|
some test cases defined by the *exercise*.
|
|
|
|
limits in some test cases defined by the *exercise*.
|
|
|
|
|
|
|
|
|
|
|
|
User management
|
|
|
|
User management
|
|
|
|
---------------
|
|
|
|
---------------
|
|
|
@ -281,12 +283,49 @@ Exercises database
|
|
|
|
|
|
|
|
|
|
|
|
### Runtime environments and hardware groups
|
|
|
|
### Runtime environments and hardware groups
|
|
|
|
|
|
|
|
|
|
|
|
@todo: describe the concept of RE and HG
|
|
|
|
@todo read this later and see if it still makes sense
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ReCodEx is designed to utilize a rather diverse set of workers -- there can be
|
|
|
|
|
|
|
|
differences in many aspects, such as the actual hardware running the worker
|
|
|
|
|
|
|
|
(which impacts the results of measuring) or installed compilers, interpreters
|
|
|
|
|
|
|
|
and other tools needed for evaluation. To address these two examples in
|
|
|
|
|
|
|
|
particular, we assign runtime environments and hardware groups to exercises.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The purpose of runtime environments is to specify which tools (and often also
|
|
|
|
|
|
|
|
operating system) are required to evaluate a solution of the exercise -- for
|
|
|
|
|
|
|
|
example, a C# programming exercise can be evaluated on a Linux worker running
|
|
|
|
|
|
|
|
Mono or a Windows worker with the .NET runtime. Such exercise would be assigned
|
|
|
|
|
|
|
|
two runtime environments, `Linux+Mono` and `Windows+.NET` (the environment names
|
|
|
|
|
|
|
|
are arbitrary strings configured by the administrator).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
A hardware group is a set of workers that run on similar hardware (e.g. a
|
|
|
|
|
|
|
|
particular quad-core processor model and a SSD hard drive). Workers are assigned
|
|
|
|
|
|
|
|
to these groups by the administrator. If this is done correctly, performance
|
|
|
|
|
|
|
|
measurements of a submission should yield the same results. Thanks to this fact,
|
|
|
|
|
|
|
|
we can use the same resource limits on every worker in a hardware group.
|
|
|
|
|
|
|
|
However, limits can differ between runtime environments -- formally speaking,
|
|
|
|
|
|
|
|
limits are a function of three arguments: an assignment, a hardware group and a
|
|
|
|
|
|
|
|
runtime environment.
|
|
|
|
|
|
|
|
|
|
|
|
### Reference solutions
|
|
|
|
### Reference solutions
|
|
|
|
|
|
|
|
|
|
|
|
@todo: What it is, what is it used for, how to add one, how to evaluate
|
|
|
|
@todo: how to add one, how to evaluate it
|
|
|
|
it
|
|
|
|
|
|
|
|
|
|
|
|
The task of determining appropriate resource limits for exercises is difficult
|
|
|
|
|
|
|
|
to do correctly. To aid exercise authors and group supervisors, ReCodEx supports
|
|
|
|
|
|
|
|
assigning reference solutions to exercises. Those are example programs that
|
|
|
|
|
|
|
|
should cover the main approaches to the implementation. For example, searching
|
|
|
|
|
|
|
|
for an integer in an ordered array can be done with a linear search, or better,
|
|
|
|
|
|
|
|
using a binary search.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Reference solutions can be evaluated on demand, using a selected hardware group.
|
|
|
|
|
|
|
|
The evaluation results are stored and can be used later to determine limits. In
|
|
|
|
|
|
|
|
our example problem, we could configure the limits so that the linear
|
|
|
|
|
|
|
|
search-based program doesn't finish in time on larger inputs, but a binary
|
|
|
|
|
|
|
|
search does.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Note that separate reference solutions should be supplied for all supported
|
|
|
|
|
|
|
|
runtime environments.
|
|
|
|
|
|
|
|
|
|
|
|
### Exercise assignments
|
|
|
|
### Exercise assignments
|
|
|
|
|
|
|
|
|
|
|
|