From 5d9fa4254736307794cd5b177cf009e8731913cb Mon Sep 17 00:00:00 2001 From: Teyras Date: Fri, 25 Dec 2015 18:17:21 +0100 Subject: [PATCH] updated the messagigng protocol and file server dir structure --- Architecture.md | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/Architecture.md b/Architecture.md index e838fe9..051cfd2 100644 --- a/Architecture.md +++ b/Architecture.md @@ -9,7 +9,9 @@ **Worker's** main role is securely _compile_, _run_ and _evaluate_ given submit against model solutions provided by author of each task. It is logicaly divided into three objects: - **Message Frontend** communicates with **WebApp** using messaging queue [ZeroMQ](http://zeromq.org/). It receives new submits, operates the evaluation through **Work API** and reports progress back. - **Worker Core** can do all evaluating steps and is responsible for security of them. Sandbox [Isolate](https://github.com/ioi/isolate) is used. -- **File Server Frontend** ensures via **File API** access to files on **File Server**, where are stored testing inputs and corresponding outputs for each task and other required files. It's possible to upload files, too. +- **File Server Frontend** provides access to files on **File Server** via + **File API**, where testing inputs and corresponding outputs for each task and + other required files are stored. It's possible to upload files, too. ### Default worker configuration Worker should have some default configuration which is applied to worker itself or may be used in given jobs (implicitly if something is missing, or explicitly with special variables). This configuration should be hardcoded and can be rewritten by explicitly declared configuration file. Format of this configuration is yaml like in the job config. @@ -71,8 +73,11 @@ To meet these requirements, **Storage** and **Database** must be set as bellow. ├── submits │   └── user_id │   └── advanced_dot_net_1 -│   ├── bf216fa9274261628f4d952a103c6cfd1cbbc587 -│   └── e6ae49bbfda4a8bb57aceeb64fb117990b226ca5 +│ └── submit_id +│   ├── eval.yml +│   └── source.cs +├── submit_archives +│ └── submit_id.tar.gz ├── tasks │   ├── a │   │   ├── a014ed2abb56371bfaf2b4298a85d5dfb56509ed @@ -86,10 +91,11 @@ To meet these requirements, **Storage** and **Database** must be set as bellow. There are subdirectories _user_id_ and _advanced_dot_net_1_ which groups submits by users and courses the submits are for. This structure is easy to maintain for new and deleted users. - -- **tasks** contains all files for all existing task in ReCodEx. To avoid too - many files in one directory, files are separated to subfolders by first - character of their name. +- **submit_archives** contains the student submissions in compressed archives so + that they can be easily downloaded by workers. +- **tasks** contains supplementary files (such as test inputs or helper + programs) for all existing task in ReCodEx. To avoid too many files in one + directory, files are separated to subfolders by first character of their name. - **temp** directory is dedicated to temporary storing outputs of programs on teachers' demand. This directory will be erased by cron job on daily basis. ### Database @@ -117,11 +123,12 @@ that passes jobs to workers capable of processing them. The frontend must send a multipart message that contains the following frames: - The `eval` command -- The job id (ASCII or network byte order - to be specified) +- The job id (in ASCII representation -- we avoid endianness issues and also + support alphabetic ids) - A frame for each header (e.g. `hwgroup=group_1`) -- An URL of the file server where the submitted files are stored -- A hash code of the assignment's configuration file -- Hash codes of files submitted by the user, each in a separate frame +- An URL of the archive that contains the submitted files and isoeval + configuration +- An URL where the worker should store the result of the evaluation If the broker is capable of routing the request to a worker, it responds with `accept`. Otherwise (for example when the requirements specified by the headers @@ -148,7 +155,8 @@ Whenever the broker receives an assignment suitable for the worker, it just forwards the evaluation request message it originally received from the frontend. The worker has to: -- Download the assignment configuration file +- Download the archive containing the submission and an isoeval configuration + file - Download any supplementary files based on the configuration file, such as test inputs or helper programs (This can be done on demand, using a `fetch` command in the assignment configuration)