Analysis update

master
Petr Stefan 8 years ago
parent 6048f50293
commit 7036a743e5

@ -673,45 +673,21 @@ arbitrary number of tasks with other types.
### Evaluation progress state
Users surely want to know a progress state of their submitted solution. The very
first idea would be to report state based on done messages from compilation,
first idea would be to report a state based on done messages from compilation,
execution and evaluation as a lot of evaluation systems are already providing.
However the ReCodEx have more advanced execution pipeline where there can be
more compilations or more executions per test and also other technical tasks
controlling the job execution flow. The users do not know about these technical
details and data from this tasks may confuse them.
A solution is to show users only percentual completion of the job as a plain
progress bar without additional information about task types. This solution
works well for all of the jobs and is very user friendly. To make the output
more interesting, there is a database of random kind-of-funny statements and a
random new one is displayed every time a task is completed.
details and data from this tasks may confuse them. A solution is to show users
only percentual completion of the job without additional information about task
types. This solution works well for all of the jobs and is very user friendly.
### Results of evaluation
There are lot of things which deserves discussion concerning results of
evaluation, how they should be displayed, what should be visible or not and also
what kind of reward for users solutions should be chosen.
#### Evaluation outputs
At first let us focus on all kinds of outputs from executed programs within job.
Out of discussion is that supervisors should be able to view almost all outputs
from solutions if they choose them to be visible and recorded. This feature is
critical in debugging either whole exercises or users solutions. Supervisor
should have a choice to turn on preserving the data while the default behaviour
is to discard them to keep a file base around whole ReCodEx system in sensible
limits.
More interesting question is if students should see the logs from execution of
their solution. Usual approach is to keep these information private because of
possibility of leaking input data. This may lead students to hack their
solutions to pass just the ReCodEx testing cases instead of properly solving the
assigned problem. Martin Mareš strongly recommended to use this strategy of
hiding sensitive data too, so ReCodEx does. One exception are compilation
outputs which can help students a lot during troubleshooting. These logs shall
be visible unless the supervisor decides otherwise. Note, that due to lack of
frontend developers, this feature was not implemented in the very first release
of ReCodEx, but will be definitely available in the future.
The evaluation data have to be processed and then presented in human readable
form. This is done through a one numeric value called points. Also, results of
job tests should be available to know what kind of error is in the solution. For
more debugging, outputs of tasks could be optionally available for the users.
#### Scoring and assigning points
@ -739,8 +715,8 @@ error" which is the valid answer in two tests), a minimal point threshold can be
specified. It the solution is to get less points than specified, it will get
zero points instead. This functionality can be embedded into grading computation
algoritm itself, but it would have to be present in each implementation
separately, which is a bit ugly. So, this feature is separated from point
computation.
separately, which is not maintainable. Because of this the the threshold feature
is separated from point computation.
Automatic grading cannot reflect all aspects of submitted code. For example,
structuring the code, number and quality of comments and so on. To allow
@ -754,6 +730,25 @@ even student submitting the same code again which evaluates to more points,
supervisor can mark a particular solution as marked and used for grading instead
of solution with the most points.
#### Evaluation outputs
In addition to the exact measured values used for score calculation described in
previous chapter, there are also text or binary outputs of the executed tasks.
Knowing them helps users identify and solve their potential issues, but on the
other hand this can lead to possibility of leaking input data. This may lead
students to hack their solutions to pass just the ReCodEx testing cases instead
of properly solving the assigned problem. The usual approach is to keep these
information private and so does strongly recommended Martin Mareš, who has
experience with several programming contests.
The only one exception of hiding the logs are compilation outputs, which can
help students a lot during troubleshooting and there is only small possibility
of input data leakage. The supervisors have access to all of the logs and they
can decide if students are allowed to see the compilation outputs.
Note, that due to lack of frontend developers, showing compilation logs to the
students is not implemented in the very first release of ReCodEx.
### Persistence
Previous parts of analysis show that the system has to keep some state. This

Loading…
Cancel
Save