@ -33,18 +33,34 @@ Broker gets information about new submission from web application. At this point
Worker gets request from broker to evaluate particular submission. Next step is to evaluate given submission and results upload to fileserver. After this worker only send broker that submission was evaluated. More detailed description follows:
- "listening" thread gets multipart message from broker with command "eval"
- "listening" thread hand over whole message through inproc socket to "execution" thread
- "listening" thread gets multipart message from `broker` with command "eval"
- "listening" thread hand over whole message through `inproc` socket to "execution" thread
- "execution" thread now has to prepare all things and get ready for execution
- temporary folders are initated (but not created) this includes folder with source files, folder with downloaded submission, temporary directory for all possible types of files and folder which will contain results from execution
- temporary folders names are initated (but not created) this includes folder with source files, folder with downloaded submission, temporary directory for all possible types of files and folder which will contain results from execution
- if some of the above stated folders is already existing, then it's deleted
- after successfull initiation submission archive is downloaded to newly created folder
- after successfull initiation submission archive is downloaded to created folder
- submission archive is decompressed into submission files folder
- all files from decompressed archive are copied into evaluation directory which can be used for execution in sandboxes
- all other folders which were not created are created just now
- it's time to build job from configuration
- job configuration file is located in evaluation directory if exists
- ...
- it's time to build `job` from configuration
- job configuration file is located in evaluation directory if exists and is loaded using `yaml-cpp` library
- loaded configuration is now parsed into `job_metadata` structure which is handed over to `job` execution class itself
- `job` execution class will now initialize and construct particular `tasks` from `job_metadata` into task tree
- if there is some item which can use variables (e.g. binary path, cmd arguments, bound directories) it is done at this point
- all tasks from configuration are created and divided into external or internal tasks
- external tasks have to load limits from configuration for this workers hwgroup which was loaded from worker configuration
- if limits were not defined default worker limits are loaded
- internal tasks are just created without any further processing like external tasks
- after successfull creation of all tasks, they are connected into graph structure using dependencies
- next and last step of building `job` structure is to execute topological sorting and get queue of tasks which will be executed in order
- topological sorting take into account priority of tasks and sequential order from job configuration
- running all tasks in order follows
- after that results have to be obtained from all executed tasks and given into corresponding yaml file
- result yaml file alongside with content of result folder is sent back to fileserver in compressed form
- of course there has to be cleaning after whole evaluation which will deinitialize all needed variables and also delete all used temporary folders
- all of previous was in "execution" thread which now have to tell "listening" thread that execution is done
- this is done through multipart message "done" with packed job identification addressed to "listening" thread
- action of "listening" is now pretty straightforward "done" message is resend to `broker`
Martin Polanka
8 years ago