# Monitor

## Description

Monitor is part of ReCodEx solution for reporting progress of job evaluation back to user in the real time. It gets progress notifications from broker and sends them through WebSockets to clients' browsers. For now, it's meant as an optional part of whole solution, but for full experince it's recommended to use one.

Monitor is needed one per broker, that is one per separate ReCodEx instance. Also, monitor has to be publicly visible (has to have public IP address or be behind public proxy server) and also needs a connection to the broker. If the web application is using HTTPS, it's required to use a proxy for monitor to provide encryption over WebSockets. If this is not done, browsers of the users will block unencrypted connection and won't show the progress to the users.


## Architecture

Monitor is written in Python, tested versions are 3.4 and 3.5. This language was chosen because it's already in project requirements (fileserver) and there are great libraries for ZeroMQ, WebSockets and asynchronous operations. This library saves system resources and provides us great amount of processed messages. Also, coding in Python was pretty simple and saves us time for improving the other parts of ReCodEx.

For monitor functionality there are some required packages. All of them are listed in _requirements.txt_ file in the repository and can be installed by `pip` package manager as
```
$ pip install -r requirements.txt
```

**Description of dependencies:**

- zmq -- binding to ZeroMQ framework
- websockets -- framework for communication over WebSockets
- asyncio -- library for fast asynchronous operations
- pyyaml -- parsing YAML configuration files
- argparse -- parsing command line arguments

### Message flow

![Message flow inside montior](https://raw.githubusercontent.com/ReCodEx/wiki/master/images/Monitor_arch.png)

Monitor runs in 2 threads. _Thread 1_ is the main thread, which initializes all components (logger for example), starts the other thread and runs the ZeroMQ part of the application. This thread receives and parses incomming messages from broker and forwards them to _thread 2_ sending logic.

_Thread 2_ is responsible for managing all of WebSocket connections asynchronously. Whole thread is one big _asyncio_ event loop through which all actions are processed. None of custom data types in Python are thread-safe, so all events from other threads (actually only `send_message` method invocation) must be called within the event loop (via `asyncio.loop.call_soon_threadsafe` function). Please note, that most of the Python interpreters use [Global Interpreter Lock](https://wiki.python.org/moin/GlobalInterpreterLock), so there is actualy no parallelism in the performance point of view, but proper synchronization is still required!

### Handling of incomming messages

Incomming ZeroMQ progress message is received and parsed to JSON format (same as our WebSocket communication format). JSON string is then passed to _thread 2_ for asynchronous sending. Each message has an identifier of channel where to send it to.

There can be multiple receivers to one channel id. Each one has separate _asyncio.Queue_ instance where new messages are added. In addition to that, there is one list of all messages per channel. If a client connects a bit later than the point when monitor starts to receive messages, it'll receive all messages from the beginning. Messages are stored 5 minutes after last progress command (normally FINISHED) is received, then are permanently deleted.

Messages from client's queue are sent through corresponding WebSocket connection via main event loop as soon as possible. This approach with separate queue per connection is easy to implement and guarantees reliability and order of message delivery.


## Installation

Installation will provide you following files:

- `/usr/bin/recodex-monitor` -- simple startup script located in PATH
- `/etc/recodex/monitor/config.yml` -- configuration file
- `/etc/systemd/system/recodex-monitor.service` -- systemd startup script
- code files will be installed in location depending on your system settings, mostly into `/usr/lib/python3.5/site-packages/monitor/` or similar

Systemd script runs monitor binary as specific _recodex_ user, so in `postinst` script user and group of this name are created. Also, ownership of configuration file will be granted to that user.

- RPM distributions can make and install binary package. This can be done like this:
	- run command
	```
	$ python3 setup.py bdist_rpm --post-install ./install/postints
	```
	to generate binary `.rpm` package or download precompiled one from releases tab of monitor GitHub repository (it's architecture independent package)
	- install package using
	```
	# yum install ./dist/recodex-monitor-<version>-1.noarch.rpm
	```
- Other Linux distributions can install cleaner straight
	```
	$ python3 setup.py install --install-scripts /usr/bin
	# ./install/postinst
	```

## Configuration and usage

### Configuration
Configuration file is located in subdirectory `monitor` of standard ReCodEx configuration folder `/etc/recodex/`. It's in YAML format as all of the other configurations. Format is very similar to configurations of broker or workers.

### Configuration items

Description of configurable items, bold ones are required, italics ones are optional.

- _websocket_uri_ -- URI where is the endpoint of websocket connection. Must be visible to the clients (directly or through public proxy)
    - string representation of IP address or a hostname
    - port number
- _zeromq_uri_ -- URI where is the endpoint of zeromq connection from broker. Could be hidden from public internet.
    - string representation of IP address or a hostname
    - port number
- _logger_ -- settings of logging
    - _file_ -- path with name of log file. Defaults to `/var/log/recodex/monitor.log`
    - _level_ -- logging level, one of "debug", "info", "warning", "error" and "critical"
    - _max-size_ -- maximum size of log file before rotation in bytes
    - _rotations_ -- number of rotations kept

### Example configuration file

```{.yml}
---
websocket_uri:
    - "127.0.0.1"
    - 4567
zeromq_uri:
    - "127.0.0.1"
    - 7894
logger:
    file: "/var/log/recodex/monitor.log"
    level: "debug"
    max-size: 1048576  # 1 MB
    rotations: 3
...
```

### Usage

Preferred way to start monitor as a service is via systemd as the other parts of ReCodEx solution.

- Running monitor is fairly simple:
```
# systemctl start recodex-monitor.service
```
- Current state can be obtained by
```
# systemctl status recodex-monitor.service
```
You should see green **Active (running)**.
- Setting up monitor to be started on system startup:
```
# systemctl enable recodex-monitor.service
```

Alternatively monitor can be started directly from command line. 
Note that this command won't start monitor as a daemon.
```
$ recodex-monitor -c /etc/recodex/monitor/config.yml
```