Merge branch 'master' of https://github.com/ReCodEx/wiki.wiki

Rewritten-docs.md

@@ -673,45 +673,21 @@ arbitrary number of tasks with other types.
 ### Evaluation progress state
 
 Users surely want to know a progress state of their submitted solution. The very
-first idea would be to report state based on done messages from compilation,
+first idea would be to report a state based on done messages from compilation,
 execution and evaluation as a lot of evaluation systems are already providing.
 However the ReCodEx have more advanced execution pipeline where there can be
 more compilations or more executions per test and also other technical tasks
 controlling the job execution flow. The users do not know about these technical
-details and data from this tasks may confuse them.
-
-A solution is to show users only percentual completion of the job as a plain
-progress bar without additional information about task types. This solution
-works well for all of the jobs and is very user friendly. To make the output
-more interesting, there is a database of random kind-of-funny statements and a
-random new one is displayed every time a task is completed.
+details and data from this tasks may confuse them. A solution is to show users
+only percentual completion of the job without additional information about task
+types. This solution works well for all of the jobs and is very user friendly.
 
 ### Results of evaluation
 
-There are lot of things which deserves discussion concerning results of
-evaluation, how they should be displayed, what should be visible or not and also
-what kind of reward for users solutions should be chosen.
-
-#### Evaluation outputs
-
-At first let us focus on all kinds of outputs from executed programs within job.
-Out of discussion is that supervisors should be able to view almost all outputs
-from solutions if they choose them to be visible and recorded. This feature is
-critical in debugging either whole exercises or users solutions. Supervisor
-should have a choice to turn on preserving the data while the default behaviour
-is to discard them to keep a file base around whole ReCodEx system in sensible
-limits.
-
-More interesting question is if students should see the logs from execution of
-their solution. Usual approach is to keep these information private because of
-possibility of leaking input data. This may lead students to hack their
-solutions to pass just the ReCodEx testing cases instead of properly solving the
-assigned problem. Martin Mareš strongly recommended to use this strategy of
-hiding sensitive data too, so ReCodEx does. One exception are compilation
-outputs which can help students a lot during troubleshooting. These logs shall
-be visible unless the supervisor decides otherwise. Note, that due to lack of
-frontend developers, this feature was not implemented in the very first release
-of ReCodEx, but will be definitely available in the future.
+The evaluation data have to be processed and then presented in human readable
+form. This is done through a one numeric value called points. Also, results of
+job tests should be available to know what kind of error is in the solution. For
+more debugging, outputs of tasks could be optionally available for the users.
 
 #### Scoring and assigning points
 
@@ -739,8 +715,8 @@ error" which is the valid answer in two tests), a minimal point threshold can be
 specified. It the solution is to get less points than specified, it will get
 zero points instead. This functionality can be embedded into grading computation
 algoritm itself, but it would have to be present in each implementation
-separately, which is a bit ugly. So, this feature is separated from point
-computation.
+separately, which is not maintainable. Because of this the the threshold feature
+is separated from point computation.
 
 Automatic grading cannot reflect all aspects of submitted code. For example,
 structuring the code, number and quality of comments and so on. To allow
@@ -754,6 +730,25 @@ even student submitting the same code again which evaluates to more points,
 supervisor can mark a particular solution as marked and used for grading instead
 of solution with the most points.
 
+#### Evaluation outputs
+
+In addition to the exact measured values used for score calculation described in
+previous chapter, there are also text or binary outputs of the executed tasks.
+Knowing them helps users identify and solve their potential issues, but on the
+other hand this can lead to possibility of leaking input data. This may lead
+students to hack their solutions to pass just the ReCodEx testing cases instead
+of properly solving the assigned problem. The usual approach is to keep these
+information private and so does strongly recommended Martin Mareš, who has
+experience with several programming contests.
+
+The only one exception of hiding the logs are compilation outputs, which can
+help students a lot during troubleshooting and there is only small possibility
+of input data leakage. The supervisors have access to all of the logs and they
+can decide if students are allowed to see the compilation outputs.
+
+Note, that due to lack of frontend developers, showing compilation logs to the
+students is not implemented in the very first release of ReCodEx.
+
 ### Persistence
 
 Previous parts of analysis show that the system has to keep some state. This
@@ -941,20 +936,46 @@ However, all of the three options would have been possible to use.
 
 ### File transfers
 
-There has to be a way to access files stored on the fileserver from both workers
-and clients. We will present some of the possible options:
-
-@todo elaborate this stuff
-
-- HTTP(S)
-- FTP
-- SFTP
-- A network-shared file system (such as NFS)
-- A custom protocol over ZeroMQ
-
-We chose HTTPS because it is widely used and clients exist in all relevant
-environments. In addition, it is highly probable we will have to run an HTTP
-server, because it is intended for ReCodEx to have a web frontend.
+There has to be a way to access files stored on the fileserver from both worker
+and frontend server machines. The protocol used for this should handle large
+files efficiently and be resilient to network failures. Security features are
+not a primary concern, because all communication with the fileserver will happen
+in an internal network. However, a basic form of authentication can be useful to
+ensure correct configuration (if a development fileserver uses different
+credentials than production, production workers will not be able to use it by
+accident). Lastly, the protocol must have a client library for platforms
+(languages) used in the backend. We will present some of the possible options:
+
+- HTTP(S) -- a de-facto standard for web communication that has far more
+  features than just file transfers. Thanks to being used on the web, a large
+  effort has been put into the development of its servers. It supports
+  authentication and it can handle short-term network failures (thanks to being
+  built on TCP and supporting resuming interrupted transfers). We will use HTTP
+  for communication with clients, so there is no added cost in maintaining a
+  server. HTTP requests can be made using libcurl.
+- FTP -- an old protocol designed only for transferring files. It has all the
+  required features, but doesn't offer anything over HTTP. It is also supported
+  by libcurl.
+- SFTP -- a file transfer protocol most frequently used as a subsystem of the
+  SSH protocol implementations. It doesn't provide authentication, but it
+  supports working with large files and resuming failed transfers. The libcurl
+  library supports SFTP.
+- A network-shared file system (such as NFS) -- an obvious advantage of a
+  network-shared file system is that applications can work with remote files the
+  same way they would with local files. However, it brings an overhead for the
+  administrator, who has to configure access to this filesystem for every
+  machine that needs to access the storage. 
+- A custom protocol over ZeroMQ -- it is possible to design a custom file
+  transfer protocol that uses ZeroMQ for sending data, but it is not a trivial
+  task -- we would have to find a way to transfer large files efficiently, 
+  implement an acknowledgement scheme and support resuming transfers. Using
+  ZeroMQ as the underlying layer does not help a lot with this. The sole
+  advantage of this is that the backend components would not need another
+  library for communication.
+
+We chose HTTPS because it is widely used and client libraries exist in all
+relevant environments. In addition, it is highly probable we will have to run an
+HTTP server, because it is intended for ReCodEx to have a web frontend.
 
 ### Frontend - backend communication