|
|
|
@ -2671,16 +2671,28 @@ The responsibilites of broker are:
|
|
|
|
|
either via monitor or REST API
|
|
|
|
|
- notifying the frontend of errors in the backend
|
|
|
|
|
|
|
|
|
|
### Architecture
|
|
|
|
|
### Internal Structure
|
|
|
|
|
|
|
|
|
|
The broker uses our ZeroMQ _reactor_ to bind events on sockets to handler
|
|
|
|
|
classes. There are currently two handlers -- one that handles the main
|
|
|
|
|
classes. There are currently two handlers -- one that handles the main
|
|
|
|
|
functionality and another one that sends status reports to the REST API
|
|
|
|
|
asynchronously so that the broker does not have to wait for HTTP requests which
|
|
|
|
|
can take a lot of time, especially when some kind of error happens on the
|
|
|
|
|
server.
|
|
|
|
|
|
|
|
|
|
#### Worker registry
|
|
|
|
|
Main handler takes care of requests from workers and API servers:
|
|
|
|
|
|
|
|
|
|
- *init* -- initial connection from worker to broker
|
|
|
|
|
- *done* -- currently processed job on worker was executed and is done
|
|
|
|
|
- *ping* -- worker prooving that it is still alive
|
|
|
|
|
- *progress* -- job progress state from worker which is immediatelly forwarded
|
|
|
|
|
to monitor
|
|
|
|
|
- *eval* -- request from API server to execute given job
|
|
|
|
|
|
|
|
|
|
Second handler is asynchronous status notifier which is able to execute HTTP
|
|
|
|
|
requests. This notifier is used on error reporting from backend to frontend API.
|
|
|
|
|
|
|
|
|
|
#### Worker Registry
|
|
|
|
|
|
|
|
|
|
The `worker_registry` class is used to store information about workers, their
|
|
|
|
|
status and the jobs in their queue. It can look up a worker using the headers
|
|
|
|
@ -2699,7 +2711,14 @@ end, which makes it less likely to receive another job soon.
|
|
|
|
|
When a worker is assigned a job, it will not be assigned another one until we
|
|
|
|
|
receive a `done` message from it.
|
|
|
|
|
|
|
|
|
|
#### Error handling
|
|
|
|
|
#### Error Reporting
|
|
|
|
|
|
|
|
|
|
Broker is the only backend component which is able to report errors to frontend
|
|
|
|
|
API. For this purpose HTTP protocol is used through *libcurl* library. To
|
|
|
|
|
address security concerns there is *HTTP Basic Auth* configured on particular
|
|
|
|
|
endpoints which is simply enough to use within *libcurl*.
|
|
|
|
|
|
|
|
|
|
Following types of failures are distinguished:
|
|
|
|
|
|
|
|
|
|
**Job failure** -- we recognize two ways a job can fail -- an internally and
|
|
|
|
|
externally. An internal failure is the fault of worker -- for example when it
|
|
|
|
@ -2725,10 +2744,6 @@ External failures are reported to the frontend immediately.
|
|
|
|
|
internally cannot be reassigned, because the "new" broker does not know their
|
|
|
|
|
headers -- they are reported as failed immediately.
|
|
|
|
|
|
|
|
|
|
@todo: what to mention:
|
|
|
|
|
- API notification using curl, authentication using HTTP Basic Auth
|
|
|
|
|
- asynchronous resending progress messages
|
|
|
|
|
|
|
|
|
|
### Additional Libraries
|
|
|
|
|
|
|
|
|
|
Broker implementation depends on several open-source C and C++ libraries.
|
|
|
|
|