@todo: how to display generally all outputs of executed programs to user (supervisor, student), what students can or cannot see and why
There are lot of things which deserves discussion concerning results of
evaluation, how they should be displayed, what should be visible or not and also
some what kind of reward for users solutions should be chosen.
At first lets focus on all kinds of outputs from executed programs within job.
Out of discussion is that supervisors should be able to view almost all outputs
from solutions if they choose them to be visible and recorded. This feature is
critical in debugging either whole exercises or users solutions. But should it
be default behaviour to record every output? Absolutely not, supervisor should
have choice to turn it on but defaults has to be not record it. Even without
this functionality can be file base around whole ReCodEx system quite large and
on top of that outputs from executed programs can be sometimes very extensive.
Storing this amount of data is purely nonsense to every solution. But if
requested by supervisor then this feature should be available.
Question what should regular users see from execution of their solution is more
interesting. Simple answer is of course that they should not see anything which
is partly true. Outputs from their programs can be anything and users can
somehow analyse inputs or even redirect them to output. So outputs from
execution should not be visible at all or under very special circumstances. But
what about compilation or other kinds of initiations. Well, this is another
story, it really depends on the particular case. But generally it is quite
harmless to display user some kind of compilation error which can help a lot
during troubleshooting. Of course again this kind of functionality should be
configurable by supervisors and disabled by default. There is also the last kind
of tasks which can output some information which is evaluation tasks. Output of
these tasks is somehow important to whole system and again can contain some
information about inputs or reference outputs. So outputs of evaluation tasks
should not also be visible to regular users.
@todo: judges, discuss what they possibly can do and what it can be used for (returning for instance 2 numbers instead of 1 and why we return just one)
@todo: judges, discuss what they possibly can do and what it can be used for (returning for instance 2 numbers instead of 1 and why we return just one)
Martin Polanka
8 years ago