Broker impl

master
Martin Polanka 8 years ago
parent abe471833b
commit c30aced4b1

@ -2671,7 +2671,7 @@ The responsibilites of broker are:
either via monitor or REST API either via monitor or REST API
- notifying the frontend of errors in the backend - notifying the frontend of errors in the backend
### Architecture ### Internal Structure
The broker uses our ZeroMQ _reactor_ to bind events on sockets to handler The broker uses our ZeroMQ _reactor_ to bind events on sockets to handler
classes. There are currently two handlers -- one that handles the main classes. There are currently two handlers -- one that handles the main
@ -2680,7 +2680,19 @@ asynchronously so that the broker does not have to wait for HTTP requests which
can take a lot of time, especially when some kind of error happens on the can take a lot of time, especially when some kind of error happens on the
server. server.
#### Worker registry Main handler takes care of requests from workers and API servers:
- *init* -- initial connection from worker to broker
- *done* -- currently processed job on worker was executed and is done
- *ping* -- worker prooving that it is still alive
- *progress* -- job progress state from worker which is immediatelly forwarded
to monitor
- *eval* -- request from API server to execute given job
Second handler is asynchronous status notifier which is able to execute HTTP
requests. This notifier is used on error reporting from backend to frontend API.
#### Worker Registry
The `worker_registry` class is used to store information about workers, their The `worker_registry` class is used to store information about workers, their
status and the jobs in their queue. It can look up a worker using the headers status and the jobs in their queue. It can look up a worker using the headers
@ -2699,7 +2711,14 @@ end, which makes it less likely to receive another job soon.
When a worker is assigned a job, it will not be assigned another one until we When a worker is assigned a job, it will not be assigned another one until we
receive a `done` message from it. receive a `done` message from it.
#### Error handling #### Error Reporting
Broker is the only backend component which is able to report errors to frontend
API. For this purpose HTTP protocol is used through *libcurl* library. To
address security concerns there is *HTTP Basic Auth* configured on particular
endpoints which is simply enough to use within *libcurl*.
Following types of failures are distinguished:
**Job failure** -- we recognize two ways a job can fail -- an internally and **Job failure** -- we recognize two ways a job can fail -- an internally and
externally. An internal failure is the fault of worker -- for example when it externally. An internal failure is the fault of worker -- for example when it
@ -2725,10 +2744,6 @@ External failures are reported to the frontend immediately.
internally cannot be reassigned, because the "new" broker does not know their internally cannot be reassigned, because the "new" broker does not know their
headers -- they are reported as failed immediately. headers -- they are reported as failed immediately.
@todo: what to mention:
- API notification using curl, authentication using HTTP Basic Auth
- asynchronous resending progress messages
### Additional Libraries ### Additional Libraries
Broker implementation depends on several open-source C and C++ libraries. Broker implementation depends on several open-source C and C++ libraries.

Loading…
Cancel
Save