From dd6e491767d945dc2b0df41099007d271f9e5b77 Mon Sep 17 00:00:00 2001 From: Jan Buchar Date: Fri, 18 Dec 2015 08:56:37 +0100 Subject: [PATCH] edits --- Architecture.md | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/Architecture.md b/Architecture.md index 8f4c150..99808c5 100644 --- a/Architecture.md +++ b/Architecture.md @@ -50,7 +50,10 @@ sandboxes-wrap-limits: ## File Server ![File Server Infrastructure](https://github.com/ReCodEx/GlobalWiki/blob/master/images/File_Server.png) -**File Server** stores data, which is better keep out of **WebApp's** database. It should meet following requirements: +**File Server** stores data, that should be kept outside of **WebApp's** +database (both because storing files in a database is inefficient and because +the workers need to access the files in the simplest possible way). It should +meet following requirements: - store files without duplicates - keep consistent state with main database - serve files to workers on demand @@ -76,8 +79,14 @@ To meet these requirements, **Storage** and **Database** must be set as bellow. └── temp └── 1795184136b8bdddabe50453cc2cc2d46f0f7c5e ``` -- **submits** keep information about all files submited by users to ReCodEx. There are subdirectoris _user_id_ and _advanced_dot_net_1_ which divides submits by users and class they took part of. This structure is easy to maintain for new and deleted users. -- **tasks** contains all files for all existing task in ReCodEx. To avoid too much files in one directory, files are separated to subfolders by first character of their name. +- **submits** keep information about all files submited by users to ReCodEx. + There are subdirectories _user_id_ and _advanced_dot_net_1_ which groups + submits by users and courses the submits are for. This structure is easy to + maintain for new and deleted users. + +- **tasks** contains all files for all existing task in ReCodEx. To avoid too + many files in one directory, files are separated to subfolders by first + character of their name. - **temp** directory is dedicated to temporary storing outputs of programs on teachers' demand. This directory will be erased by cron job on daily basis. ### Database @@ -87,7 +96,13 @@ For user friendly access and modifying tasks following information should be sto - for every hash name one human readable filename ### Conclusion -Files are internally stored by their `sha1sum` hashes, so it's easy to implement versioning and get rid of files with duplicity content. **Worker** also uses files by their hashes, which is great for local caching without worries about actual version number of given file. On the other hand, **Database** keeps info about human readable names, so to users (teachers) in **WebApp** are files presented in a friendly way. +Files are internally stored by their `sha1sum` hashes, so it's easy to implement +versioning and get rid of files with duplicate content (multiple files can have +the same content, which is only stored once). **Worker** also uses files by +their hashes, which is great for local caching without worries about actual +version number of given file. On the other hand, **Database** stores information +about human readable names, so that the files are presented in a friendly way to +users (teachers) in **WebApp**. ## Frontend - broker communication @@ -108,6 +123,11 @@ If the broker is capable of routing the request to a worker, it responds with `ack`. Otherwise (for example when the requirements specified by the headers cannot be met), it responds with `nack`. +Note that we will need to store the job ID and the assignment configuration +somewhere close to the submitted files so it's possible to check how a +submission was evaluated. The job ID will likely be a part of the submission's +path. The configuration could be linked there under some well-known name. + ### Notifying the frontend about evaluation progress The script that requested the evaluation will have exited by the time a worker