From cd89c0bb391727fec66929e7c30e4c5ef6ba92ff Mon Sep 17 00:00:00 2001 From: Teyras Date: Sun, 22 Jan 2017 23:54:42 +0100 Subject: [PATCH] elaborate on fileserver transfer protocol --- Rewritten-docs.md | 54 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 40 insertions(+), 14 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index b2d0169..7ecc32a 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -936,20 +936,46 @@ However, all of the three options would have been possible to use. ### File transfers -There has to be a way to access files stored on the fileserver from both workers -and clients. We will present some of the possible options: - -@todo elaborate this stuff - -- HTTP(S) -- FTP -- SFTP -- A network-shared file system (such as NFS) -- A custom protocol over ZeroMQ - -We chose HTTPS because it is widely used and clients exist in all relevant -environments. In addition, it is highly probable we will have to run an HTTP -server, because it is intended for ReCodEx to have a web frontend. +There has to be a way to access files stored on the fileserver from both worker +and frontend server machines. The protocol used for this should handle large +files efficiently and be resilient to network failures. Security features are +not a primary concern, because all communication with the fileserver will happen +in an internal network. However, a basic form of authentication can be useful to +ensure correct configuration (if a development fileserver uses different +credentials than production, production workers will not be able to use it by +accident). Lastly, the protocol must have a client library for platforms +(languages) used in the backend. We will present some of the possible options: + +- HTTP(S) -- a de-facto standard for web communication that has far more + features than just file transfers. Thanks to being used on the web, a large + effort has been put into the development of its servers. It supports + authentication and it can handle short-term network failures (thanks to being + built on TCP and supporting resuming interrupted transfers). We will use HTTP + for communication with clients, so there is no added cost in maintaining a + server. HTTP requests can be made using libcurl. +- FTP -- an old protocol designed only for transferring files. It has all the + required features, but doesn't offer anything over HTTP. It is also supported + by libcurl. +- SFTP -- a file transfer protocol most frequently used as a subsystem of the + SSH protocol implementations. It doesn't provide authentication, but it + supports working with large files and resuming failed transfers. The libcurl + library supports SFTP. +- A network-shared file system (such as NFS) -- an obvious advantage of a + network-shared file system is that applications can work with remote files the + same way they would with local files. However, it brings an overhead for the + administrator, who has to configure access to this filesystem for every + machine that needs to access the storage. +- A custom protocol over ZeroMQ -- it is possible to design a custom file + transfer protocol that uses ZeroMQ for sending data, but it is not a trivial + task -- we would have to find a way to transfer large files efficiently, + implement an acknowledgement scheme and support resuming transfers. Using + ZeroMQ as the underlying layer does not help a lot with this. The sole + advantage of this is that the backend components would not need another + library for communication. + +We chose HTTPS because it is widely used and client libraries exist in all +relevant environments. In addition, it is highly probable we will have to run an +HTTP server, because it is intended for ReCodEx to have a web frontend. ### Frontend - backend communication