From 9575286792942581f1a905be83db84c8385c2f2d Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Thu, 28 Jul 2016 15:09:49 +0200 Subject: [PATCH] worker part modified accordingly --- Architecture.md | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/Architecture.md b/Architecture.md index da17da6..3108e66 100644 --- a/Architecture.md +++ b/Architecture.md @@ -1,7 +1,7 @@ # Overall Architecture ![Overall Architecture](https://github.com/ReCodEx/GlobalWiki/blob/master/images/Overall_Architecture.png) -**ReCodEx** is designed to be very modular. **WebApp** is considered to be frontend part of whole system while all other parts/services are considered to be backend part. **WebApp** contains almost all logic of the app including _user management and authentication_, _storing and versioning files_ (through connection with **Fileserver**), _counting and assigning points_ to users etc. **WebApp** is connected with **Broker**, **Fileserver** and partially with **Monitor**. **Broker** is essential part of whole architecture and can be marked single point of failure. +**ReCodEx** is designed to be very modular. **WebApp** is considered to be frontend part of whole system while all other parts/services are considered to be backend part. **WebApp** contains almost all logic of the app including _user management and authentication_, _storing and versioning files_ (through connection with **Fileserver**), _counting and assigning points_ to users etc. **WebApp** is connected with **Broker**, **Fileserver** and partially with **Monitor**. **Broker** is essential part of whole architecture and can be marked as single point of failure. Almost whole communication goes through **Broker** and `ZeroMQ` messaging middleware. When **WebApp** wants to execute submission then all datas are handed over to **Worker** through **Broker**, similar situation is with progress state which start in **Worker** goes through **Broker** then pass **Monitor** and end up in **WebApp**. Only part of communication, which does not include **Broker**, is communication with **Fileserver** which is realized through `HTTP` commmunication. This communication can be initiated by **WebApp** or by **Worker**, other services have no access to **Fileserver**. Summarize of above stated can be found in overall architecture image (green arrows = `HTTP` communication, red arrows = communication through `ZeroMQ`). @@ -16,27 +16,23 @@ Almost whole communication goes through **Broker** and `ZeroMQ` messaging middle ## Worker ![Worker Architecture](https://github.com/ReCodEx/GlobalWiki/blob/master/images/Worker_Architecture.png) -**Worker's** main role is securely _compile_, _run_ and _evaluate_ given submit against model solutions provided by author of each task. It is logicaly divided into three objects: -- **Message Frontend** communicates with **WebApp** using messaging queue [ZeroMQ](http://zeromq.org/). It receives new submits, operates the evaluation through **Work API** and reports progress back. -- **Worker Core** can do all evaluating steps and is responsible for security of them. Sandbox [Isolate](https://github.com/ioi/isolate) is used. -- **File Server Frontend** provides access to files on **File Server** via - **File API**, where testing inputs and corresponding outputs for each task and - other required files are stored. It's possible to upload files, too. +**Worker's** main role is securely execute given submission and possibly _evaluate_ results against model solutions provided by submitter. **Worker** is logicaly divided into two parts: +- **Listener** - listens and communicates with **Broker** through [ZeroMQ](http://zeromq.org/). It receives new jobs, communicates with **Evaluator** part and sends back results or progress. +- **Evaluator** - gets jobs to evaluate from **Listener** part, evaluate them (possibly in sandbox) and get to know to other part that evaluation ended. This part also communicates with **Fileserver**, downloads needed files and uploads detailed results. -The worker has to: +**Worker** after getting evaluation request has to: -- Download the archive containing the submission and an isoeval configuration - file +- Download the archive containing submitted source files and configuration file - Download any supplementary files based on the configuration file, such as test - inputs or helper programs (This can be done on demand, using a `fetch` command + inputs or helper programs (This is done on demand, using a `fetch` command in the assignment configuration) -- Download the source codes of the student's submission -- Evaluate the submission according to the assignment's configuration -- Upload the results of the evaluation to the file server -- Notify the broker that the evaluation is finished +- Evaluate the submission accordingly to job configuration +- During evaluation progress states can be sent back to **Broker** +- Upload the results of the evaluation to the **Fileserver** +- Notify **Broker** that the evaluation finished ### Internal Worker architecture -Picture below is overall internal architecture of worker, which shows its defined classes with private variables and public functions. Vector version of this picture is available [here](https://github.com/ReCodEx/GlobalWiki/raw/master/images/Worker_Internal_Architecture.pdf). +Picture below is overall internal architecture of worker which shows its defined classes with private variables and public functions. Vector version of this picture is available [here](https://github.com/ReCodEx/GlobalWiki/raw/master/images/Worker_Internal_Architecture.pdf). ![Internal Worker architecture](https://github.com/ReCodEx/GlobalWiki/blob/master/images/Worker_Internal_Architecture.png) ## File Server