elaborate on fileserver transfer protocol

master
Teyras 8 years ago
parent 7036a743e5
commit cd89c0bb39

@ -936,20 +936,46 @@ However, all of the three options would have been possible to use.
### File transfers
There has to be a way to access files stored on the fileserver from both workers
and clients. We will present some of the possible options:
@todo elaborate this stuff
- HTTP(S)
- FTP
- SFTP
- A network-shared file system (such as NFS)
- A custom protocol over ZeroMQ
We chose HTTPS because it is widely used and clients exist in all relevant
environments. In addition, it is highly probable we will have to run an HTTP
server, because it is intended for ReCodEx to have a web frontend.
There has to be a way to access files stored on the fileserver from both worker
and frontend server machines. The protocol used for this should handle large
files efficiently and be resilient to network failures. Security features are
not a primary concern, because all communication with the fileserver will happen
in an internal network. However, a basic form of authentication can be useful to
ensure correct configuration (if a development fileserver uses different
credentials than production, production workers will not be able to use it by
accident). Lastly, the protocol must have a client library for platforms
(languages) used in the backend. We will present some of the possible options:
- HTTP(S) -- a de-facto standard for web communication that has far more
features than just file transfers. Thanks to being used on the web, a large
effort has been put into the development of its servers. It supports
authentication and it can handle short-term network failures (thanks to being
built on TCP and supporting resuming interrupted transfers). We will use HTTP
for communication with clients, so there is no added cost in maintaining a
server. HTTP requests can be made using libcurl.
- FTP -- an old protocol designed only for transferring files. It has all the
required features, but doesn't offer anything over HTTP. It is also supported
by libcurl.
- SFTP -- a file transfer protocol most frequently used as a subsystem of the
SSH protocol implementations. It doesn't provide authentication, but it
supports working with large files and resuming failed transfers. The libcurl
library supports SFTP.
- A network-shared file system (such as NFS) -- an obvious advantage of a
network-shared file system is that applications can work with remote files the
same way they would with local files. However, it brings an overhead for the
administrator, who has to configure access to this filesystem for every
machine that needs to access the storage.
- A custom protocol over ZeroMQ -- it is possible to design a custom file
transfer protocol that uses ZeroMQ for sending data, but it is not a trivial
task -- we would have to find a way to transfer large files efficiently,
implement an acknowledgement scheme and support resuming transfers. Using
ZeroMQ as the underlying layer does not help a lot with this. The sole
advantage of this is that the backend components would not need another
library for communication.
We chose HTTPS because it is widely used and client libraries exist in all
relevant environments. In addition, it is highly probable we will have to run an
HTTP server, because it is intended for ReCodEx to have a web frontend.
### Frontend - backend communication

Loading…
Cancel
Save