recodex-wiki/Submission-flow.md

# Submission flow

This article will describe in execution flow of submission from the point of submission into web application to the point of evaluation of results from execution. Note that following text is simplified and should only show concepts which are used during execution.

## Web Application

First thing user has to submit his/hers solution to web application which provides interface to upload multiple files and then submit them. More detailed description follows:

1. user upload file by file into prepared submit form
2. after uploading all files connected to assignment user clicks on submit button
3. Web Application send request to Web API that user wants to evaluate assignment with provided files

## Web API

After user submits solution then web application has to hand over all needed information about submission to broker. Submit endpoint is extended with support of "cross-submitting" which means that user with high privileges (supervisor, administrator) can submit solution for user with lower privileges (student). More detailed description follows:

1. deadlines and count of submissions are checked
2. programming language of uploaded files is automatically detected
3. based on programming language proper job configuration file is obtained
4. submitted files from user and job configuration are uploaded to fileserver as archive
5. send message to broker that there is new submission
6. several information about new submission including websocket channel are sent back to user

## Broker

Broker gets information about new submission from Web API. At this point broker has to find suitable worker for execution of this particular submission. When worker is found and is jobless, then broker send detailed submission to worker to evaluation. More detailed description follows:

1. broker gets message from Web API about new submission
2. based on information from Web API, broker has to choose suitable worker which matches all submission criteria
    1. if suitable worker is not found then broker notices Web API about this information and all execution stops
    2. suitable worker is found
        1. worker has no jobs at the moment, broker immediatelly sends job evaluation request to it
        2. otherwise evaluation request is queued to this particular worker and waits until all previous jobs are done
3. broker sends prepared evaluation request to worker with all information which was obtained from Web API

## Worker

Worker gets request from broker to evaluate particular submission. Next step is to evaluate given submission and upload results to fileserver. After this worker only send broker that submission was evaluated. More detailed description follows:

1. worker gets evaluation request from broker
2. worker now has to do some initialization of directories and some internal structures for new incoming evaluation
3. users submission archive from given address from evaluation request is downloaded and decompressed
4. job configuration file is located and parsed into internal job structure
5. tasks which are present in configuration are loaded into tree like structure with dependencies and are divided into internal or external ones
    1. internal tasks are tasks which has no defined limits and contains some internal action
    2. external tasks have defined limits in configuration and are executed in sandbox
6. last step of initializing of job is to topologically sort tasks and prepare queue with right execution order
7. after that topologically sorted execution queue is processed
    1. if execution of `inner` task fails worker will immediatelly stop execution and send back to broker 'internal error during execution' message
    2. execution of `execution` or `evaluation` task fails then worker stops and send back to broker 'execution failed' message
8. when execution successfully ends results are collected from all executed tasks and written into yaml file
9. results yaml file alongside with content of job result folder is sent back to fileserver in compressed form
10. of course there has to be some cleaning after whole evaluation which will mostly delete content of temporarily created folders
11. last step of worker is to send back to broker message that execution successfully ended

## Broker

Broker gets done message from worker and basically only mark submission as done in its internal structures. After that broker has to tell Web API that execution of particular job ended. More detailed description follows:

1. broker gets message from worker that execution of job ended
2. job execution on worker was successful
    1. Web API is notified that job ended successfully
    2. if there are some waiting jobs for worker than the first one is sent for execution
3. job execution on worker ended with internal error
    1. if job ends with internal error than it is possible to reassign it to another worker
    2. broker keeps track of job reassignments and if number of them reach some predefined constant, job is declared as failed
    3. suitable worker different that the original one is picked and evaluation request is sent to it
4. job execution on worker failed
    1. Web API is notified that execution of this particular job failed
    2. again if there is waiting job for execution on worker than it is sent for execution

## Web API

Web API is notified about job status through simple http calls originated from broker. After that API is deciding if evaluated submission will be evaluated immediatelly or on demand. More detailed description follows:

- job ended with status "OK"
    - if submission was evaluation of reference solution
        - results of evaluation are loaded from fileserver and unzipped
        - actual time and memmory consumption and other suitable results are retrieved
        - solution evaluation database entity is created and persisted
        - if evaluation of results failed report to administrator is sent
    - if submission was normal student evaluation
        - if user submitted solution by him/herself then nothing is done here
        - administrator or supervisor submitted exercise as particular user
            - results of submission are processed as described further in "On demand loading" chapter
- job ended with status "FAILED"
    - job failure is saved into database alongside other reported errors
    - email is prepared and sent to ReCodEx administrator

### On demand loading

Realised as calling of evaluate API endpoint. Results of submission are loaded from fileserver and then processed. More detailed description follows:

- results of submission are retrived from fileserver
- with job config used for submission results are loaded into internal API structures
- typed results are used to compute overall score of solution
- above detected score with any other suitable information are stored into database as solution evaluation entity

## Web Application

Web Application has only a simple work to do. If results is obtained on demand then proper API call is executed and results are obtained and shown to user. More detailed description follows:

- if results of submission are not prepared yet evaluate API endpoint is called
- after this results should be present and can be shown to user