From 5affa9e5efbc623f540da8a5c698cb3a51adfc43 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Wed, 22 Jun 2016 21:20:35 +0200 Subject: [PATCH] Worker - File Server communication --- Communication.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/Communication.md b/Communication.md index a89eb4d..4a976fb 100644 --- a/Communication.md +++ b/Communication.md @@ -99,9 +99,28 @@ workers. If a worker thinks the broker is dead, it tries to reconnect with a bounded, exponentially increasing delay. + ## Worker - File Server communication -**TODO:** +Worker is communicating with file server only from _execution thread_. Supported is HTTP protocol optionally with SSL encryption (**recommended**, you can get free certificate from [Let's Encrypt](https://letsencrypt.org/) if you haven't one yet). If supported by server and used version of libcurl, HTTP/2 standard is also available. File server should be set up to require basic HTTP authentication and worker is capable to send corresponding credentials with each request. + +### Worker point of view + +Worker is cabable of 2 things - download file and upload file. Internally, worker is using libcurl C library with very similar setup. In both cases it can verify HTTPS certificate (on Linux against system cert list, on Windows against downloaded one from their website during installation), support basic HTTP authentication, offer HTTP/2 with fallback to HTTP/1.1 and fail on error (returned HTTP status code is >= 400). Worker have list of credentials to all available file servers in it's config file. + +- download file - standard HTTP GET request to given URL expecting content as response +- upload file - standard HTTP PUT request to given URL with file data as body - same as command line tool `curl` with option `--upload-file` + +### File server point of view + +File server has it's internal directory structure, where all the files are stored. It provides REST API to get them or create new ones. File server doesn't provide authentication or secured connection by itself, but it's supposed to run file server as WSGI script inside a web server (like Apache) with proper configuration. For communication with worker are relevant these commands: + +- **GET /submission_archives/.** - gets an archive with submitted source code and corresponding configuration of this job evaluation +- **GET /tasks/** - gets a file, common usage is for input files or reference result files +- **PUT /results/.** - upload archive with evaluation results under specified name (should be same _id_ as name of submission archive) + +If not specified otherview, `zip` format of archives is used. Symbol `/` in API description is root of file server's domain. If the domain is for example `fs.recodex.org` with SSL support, getting input file for one task could look as GET request to `https://fs.recodex.org/tasks/8b31e12787bdae1b5766ebb8534b0adc10a1c34c`. + ## Broker - Monitor communication