|
|
|
@ -936,20 +936,46 @@ However, all of the three options would have been possible to use.
|
|
|
|
|
|
|
|
|
|
### File transfers
|
|
|
|
|
|
|
|
|
|
There has to be a way to access files stored on the fileserver from both workers
|
|
|
|
|
and clients. We will present some of the possible options:
|
|
|
|
|
|
|
|
|
|
@todo elaborate this stuff
|
|
|
|
|
|
|
|
|
|
- HTTP(S)
|
|
|
|
|
- FTP
|
|
|
|
|
- SFTP
|
|
|
|
|
- A network-shared file system (such as NFS)
|
|
|
|
|
- A custom protocol over ZeroMQ
|
|
|
|
|
|
|
|
|
|
We chose HTTPS because it is widely used and clients exist in all relevant
|
|
|
|
|
environments. In addition, it is highly probable we will have to run an HTTP
|
|
|
|
|
server, because it is intended for ReCodEx to have a web frontend.
|
|
|
|
|
There has to be a way to access files stored on the fileserver from both worker
|
|
|
|
|
and frontend server machines. The protocol used for this should handle large
|
|
|
|
|
files efficiently and be resilient to network failures. Security features are
|
|
|
|
|
not a primary concern, because all communication with the fileserver will happen
|
|
|
|
|
in an internal network. However, a basic form of authentication can be useful to
|
|
|
|
|
ensure correct configuration (if a development fileserver uses different
|
|
|
|
|
credentials than production, production workers will not be able to use it by
|
|
|
|
|
accident). Lastly, the protocol must have a client library for platforms
|
|
|
|
|
(languages) used in the backend. We will present some of the possible options:
|
|
|
|
|
|
|
|
|
|
- HTTP(S) -- a de-facto standard for web communication that has far more
|
|
|
|
|
features than just file transfers. Thanks to being used on the web, a large
|
|
|
|
|
effort has been put into the development of its servers. It supports
|
|
|
|
|
authentication and it can handle short-term network failures (thanks to being
|
|
|
|
|
built on TCP and supporting resuming interrupted transfers). We will use HTTP
|
|
|
|
|
for communication with clients, so there is no added cost in maintaining a
|
|
|
|
|
server. HTTP requests can be made using libcurl.
|
|
|
|
|
- FTP -- an old protocol designed only for transferring files. It has all the
|
|
|
|
|
required features, but doesn't offer anything over HTTP. It is also supported
|
|
|
|
|
by libcurl.
|
|
|
|
|
- SFTP -- a file transfer protocol most frequently used as a subsystem of the
|
|
|
|
|
SSH protocol implementations. It doesn't provide authentication, but it
|
|
|
|
|
supports working with large files and resuming failed transfers. The libcurl
|
|
|
|
|
library supports SFTP.
|
|
|
|
|
- A network-shared file system (such as NFS) -- an obvious advantage of a
|
|
|
|
|
network-shared file system is that applications can work with remote files the
|
|
|
|
|
same way they would with local files. However, it brings an overhead for the
|
|
|
|
|
administrator, who has to configure access to this filesystem for every
|
|
|
|
|
machine that needs to access the storage.
|
|
|
|
|
- A custom protocol over ZeroMQ -- it is possible to design a custom file
|
|
|
|
|
transfer protocol that uses ZeroMQ for sending data, but it is not a trivial
|
|
|
|
|
task -- we would have to find a way to transfer large files efficiently,
|
|
|
|
|
implement an acknowledgement scheme and support resuming transfers. Using
|
|
|
|
|
ZeroMQ as the underlying layer does not help a lot with this. The sole
|
|
|
|
|
advantage of this is that the backend components would not need another
|
|
|
|
|
library for communication.
|
|
|
|
|
|
|
|
|
|
We chose HTTPS because it is widely used and client libraries exist in all
|
|
|
|
|
relevant environments. In addition, it is highly probable we will have to run an
|
|
|
|
|
HTTP server, because it is intended for ReCodEx to have a web frontend.
|
|
|
|
|
|
|
|
|
|
### Frontend - backend communication
|
|
|
|
|
|
|
|
|
|