diff --git a/Rewritten-docs.md b/Rewritten-docs.md index 9bdd59a..b2d0169 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -673,45 +673,21 @@ arbitrary number of tasks with other types. ### Evaluation progress state Users surely want to know a progress state of their submitted solution. The very -first idea would be to report state based on done messages from compilation, +first idea would be to report a state based on done messages from compilation, execution and evaluation as a lot of evaluation systems are already providing. However the ReCodEx have more advanced execution pipeline where there can be more compilations or more executions per test and also other technical tasks controlling the job execution flow. The users do not know about these technical -details and data from this tasks may confuse them. - -A solution is to show users only percentual completion of the job as a plain -progress bar without additional information about task types. This solution -works well for all of the jobs and is very user friendly. To make the output -more interesting, there is a database of random kind-of-funny statements and a -random new one is displayed every time a task is completed. +details and data from this tasks may confuse them. A solution is to show users +only percentual completion of the job without additional information about task +types. This solution works well for all of the jobs and is very user friendly. ### Results of evaluation -There are lot of things which deserves discussion concerning results of -evaluation, how they should be displayed, what should be visible or not and also -what kind of reward for users solutions should be chosen. - -#### Evaluation outputs - -At first let us focus on all kinds of outputs from executed programs within job. -Out of discussion is that supervisors should be able to view almost all outputs -from solutions if they choose them to be visible and recorded. This feature is -critical in debugging either whole exercises or users solutions. Supervisor -should have a choice to turn on preserving the data while the default behaviour -is to discard them to keep a file base around whole ReCodEx system in sensible -limits. - -More interesting question is if students should see the logs from execution of -their solution. Usual approach is to keep these information private because of -possibility of leaking input data. This may lead students to hack their -solutions to pass just the ReCodEx testing cases instead of properly solving the -assigned problem. Martin Mareš strongly recommended to use this strategy of -hiding sensitive data too, so ReCodEx does. One exception are compilation -outputs which can help students a lot during troubleshooting. These logs shall -be visible unless the supervisor decides otherwise. Note, that due to lack of -frontend developers, this feature was not implemented in the very first release -of ReCodEx, but will be definitely available in the future. +The evaluation data have to be processed and then presented in human readable +form. This is done through a one numeric value called points. Also, results of +job tests should be available to know what kind of error is in the solution. For +more debugging, outputs of tasks could be optionally available for the users. #### Scoring and assigning points @@ -739,8 +715,8 @@ error" which is the valid answer in two tests), a minimal point threshold can be specified. It the solution is to get less points than specified, it will get zero points instead. This functionality can be embedded into grading computation algoritm itself, but it would have to be present in each implementation -separately, which is a bit ugly. So, this feature is separated from point -computation. +separately, which is not maintainable. Because of this the the threshold feature +is separated from point computation. Automatic grading cannot reflect all aspects of submitted code. For example, structuring the code, number and quality of comments and so on. To allow @@ -754,6 +730,25 @@ even student submitting the same code again which evaluates to more points, supervisor can mark a particular solution as marked and used for grading instead of solution with the most points. +#### Evaluation outputs + +In addition to the exact measured values used for score calculation described in +previous chapter, there are also text or binary outputs of the executed tasks. +Knowing them helps users identify and solve their potential issues, but on the +other hand this can lead to possibility of leaking input data. This may lead +students to hack their solutions to pass just the ReCodEx testing cases instead +of properly solving the assigned problem. The usual approach is to keep these +information private and so does strongly recommended Martin Mareš, who has +experience with several programming contests. + +The only one exception of hiding the logs are compilation outputs, which can +help students a lot during troubleshooting and there is only small possibility +of input data leakage. The supervisors have access to all of the logs and they +can decide if students are allowed to see the compilation outputs. + +Note, that due to lack of frontend developers, showing compilation logs to the +students is not implemented in the very first release of ReCodEx. + ### Persistence Previous parts of analysis show that the system has to keep some state. This