|
|
|
@ -673,45 +673,21 @@ arbitrary number of tasks with other types.
|
|
|
|
|
### Evaluation progress state
|
|
|
|
|
|
|
|
|
|
Users surely want to know a progress state of their submitted solution. The very
|
|
|
|
|
first idea would be to report state based on done messages from compilation,
|
|
|
|
|
first idea would be to report a state based on done messages from compilation,
|
|
|
|
|
execution and evaluation as a lot of evaluation systems are already providing.
|
|
|
|
|
However the ReCodEx have more advanced execution pipeline where there can be
|
|
|
|
|
more compilations or more executions per test and also other technical tasks
|
|
|
|
|
controlling the job execution flow. The users do not know about these technical
|
|
|
|
|
details and data from this tasks may confuse them.
|
|
|
|
|
|
|
|
|
|
A solution is to show users only percentual completion of the job as a plain
|
|
|
|
|
progress bar without additional information about task types. This solution
|
|
|
|
|
works well for all of the jobs and is very user friendly. To make the output
|
|
|
|
|
more interesting, there is a database of random kind-of-funny statements and a
|
|
|
|
|
random new one is displayed every time a task is completed.
|
|
|
|
|
details and data from this tasks may confuse them. A solution is to show users
|
|
|
|
|
only percentual completion of the job without additional information about task
|
|
|
|
|
types. This solution works well for all of the jobs and is very user friendly.
|
|
|
|
|
|
|
|
|
|
### Results of evaluation
|
|
|
|
|
|
|
|
|
|
There are lot of things which deserves discussion concerning results of
|
|
|
|
|
evaluation, how they should be displayed, what should be visible or not and also
|
|
|
|
|
what kind of reward for users solutions should be chosen.
|
|
|
|
|
|
|
|
|
|
#### Evaluation outputs
|
|
|
|
|
|
|
|
|
|
At first let us focus on all kinds of outputs from executed programs within job.
|
|
|
|
|
Out of discussion is that supervisors should be able to view almost all outputs
|
|
|
|
|
from solutions if they choose them to be visible and recorded. This feature is
|
|
|
|
|
critical in debugging either whole exercises or users solutions. Supervisor
|
|
|
|
|
should have a choice to turn on preserving the data while the default behaviour
|
|
|
|
|
is to discard them to keep a file base around whole ReCodEx system in sensible
|
|
|
|
|
limits.
|
|
|
|
|
|
|
|
|
|
More interesting question is if students should see the logs from execution of
|
|
|
|
|
their solution. Usual approach is to keep these information private because of
|
|
|
|
|
possibility of leaking input data. This may lead students to hack their
|
|
|
|
|
solutions to pass just the ReCodEx testing cases instead of properly solving the
|
|
|
|
|
assigned problem. Martin Mareš strongly recommended to use this strategy of
|
|
|
|
|
hiding sensitive data too, so ReCodEx does. One exception are compilation
|
|
|
|
|
outputs which can help students a lot during troubleshooting. These logs shall
|
|
|
|
|
be visible unless the supervisor decides otherwise. Note, that due to lack of
|
|
|
|
|
frontend developers, this feature was not implemented in the very first release
|
|
|
|
|
of ReCodEx, but will be definitely available in the future.
|
|
|
|
|
The evaluation data have to be processed and then presented in human readable
|
|
|
|
|
form. This is done through a one numeric value called points. Also, results of
|
|
|
|
|
job tests should be available to know what kind of error is in the solution. For
|
|
|
|
|
more debugging, outputs of tasks could be optionally available for the users.
|
|
|
|
|
|
|
|
|
|
#### Scoring and assigning points
|
|
|
|
|
|
|
|
|
@ -739,8 +715,8 @@ error" which is the valid answer in two tests), a minimal point threshold can be
|
|
|
|
|
specified. It the solution is to get less points than specified, it will get
|
|
|
|
|
zero points instead. This functionality can be embedded into grading computation
|
|
|
|
|
algoritm itself, but it would have to be present in each implementation
|
|
|
|
|
separately, which is a bit ugly. So, this feature is separated from point
|
|
|
|
|
computation.
|
|
|
|
|
separately, which is not maintainable. Because of this the the threshold feature
|
|
|
|
|
is separated from point computation.
|
|
|
|
|
|
|
|
|
|
Automatic grading cannot reflect all aspects of submitted code. For example,
|
|
|
|
|
structuring the code, number and quality of comments and so on. To allow
|
|
|
|
@ -754,6 +730,25 @@ even student submitting the same code again which evaluates to more points,
|
|
|
|
|
supervisor can mark a particular solution as marked and used for grading instead
|
|
|
|
|
of solution with the most points.
|
|
|
|
|
|
|
|
|
|
#### Evaluation outputs
|
|
|
|
|
|
|
|
|
|
In addition to the exact measured values used for score calculation described in
|
|
|
|
|
previous chapter, there are also text or binary outputs of the executed tasks.
|
|
|
|
|
Knowing them helps users identify and solve their potential issues, but on the
|
|
|
|
|
other hand this can lead to possibility of leaking input data. This may lead
|
|
|
|
|
students to hack their solutions to pass just the ReCodEx testing cases instead
|
|
|
|
|
of properly solving the assigned problem. The usual approach is to keep these
|
|
|
|
|
information private and so does strongly recommended Martin Mareš, who has
|
|
|
|
|
experience with several programming contests.
|
|
|
|
|
|
|
|
|
|
The only one exception of hiding the logs are compilation outputs, which can
|
|
|
|
|
help students a lot during troubleshooting and there is only small possibility
|
|
|
|
|
of input data leakage. The supervisors have access to all of the logs and they
|
|
|
|
|
can decide if students are allowed to see the compilation outputs.
|
|
|
|
|
|
|
|
|
|
Note, that due to lack of frontend developers, showing compilation logs to the
|
|
|
|
|
students is not implemented in the very first release of ReCodEx.
|
|
|
|
|
|
|
|
|
|
### Persistence
|
|
|
|
|
|
|
|
|
|
Previous parts of analysis show that the system has to keep some state. This
|
|
|
|
|