# Monitor

## Description

Monitor is part of ReCodEx solution for reporting progress of job evaluation back to user in the real time. It gets progress notifications from broker and sends them through WebSockets to clients' browsers. For now, it's meant as an optional part of whole solution, but for full experince it's recommended to use one.

Monitor is one per broker, that is one per separate ReCodEx instance. Also, monitor has to be publicly visible (has to have public IP address or be behind public proxy server) and also needs a connection to the broker.


## Architecture

Monitor is written in Python, tested versions are 3.4 and 3.5. For it's functionality following packages are required:

- zmq - binding to ZeroMQ framework
- websockets - framework for communication over WebSockets
- asyncio - library for fast asynchronous operations
- pyyaml - parsing YAML configuration files
- argparse - parsing command line arguments

**Monitor's architecture and message flow describes following diagram:**

![Monitor message flow](https://raw.githubusercontent.com/ReCodEx/wiki/master/images/Monitor_arch.png)

Monitor runs in 2 threads - _Thread 1_ is main thread, which initializes all components (logger, ...), starts the other thread and runs the ZeroMQ part of the application - receives and parses incomming messages from broker and forwards them to _thread 2_ sending logic. _Thread 2_ is responsible for managing all of WebSocket connections asynchronously. Whole thread is one big _asyncio_ event loop through which are processed all actions. None of custom data types is thread-safe, so all events from other threads (actually only `send_message` method) must be called within the event loop (via `asyncio.loop.call_soon_threadsafe` function). Please note, that most of the Python interpreter use GIL ([Global Interpreter Lock](https://wiki.python.org/moin/GlobalInterpreterLock)), so there is actualy no parallelism in the performance point of view, but proper synchronization is still required!

**Handling of incomming messages:**

Incomming ZeroMQ message is received and parsed to JSON format (same as our WebSocket communication format). JSON string is then passed to _thread 2_ for sending. Each message has an identifier of channel where to send it to.

There can be multiple receivers to one channel id. Each one has separate _asyncio.Queue_ instance where new messages are added. In addition to that, there is one list of all messages per channel. If a client connects a bit later, it'll receive all messages from the beginning. Messages are stored 5 minutes after last progress command (FINISHED), then are permanently deleted.

Messages from client's queue are sent through corresponding WebSocket connection via main event loop as soon as possible. This approach with separate queue per connection is easy to implement and guarantees reliability and order of message delivery.


## Installation

Installation will provide you with following files:

- `/usr/bin/recodex-monitor` - simple startup script located in PATH
- `/etc/recodex/monitor/config.yml` - configuration file
- `/etc/systemd/system/recodex-monitor.service` - systemd startup script
- code files will be installed in location depending on your settings, mostly into `/usr/lib/python3.5/site-packages/monitor/` or similar

Systemd script runs monitor binary as specific _recodex_ user, so in `postinst` script user and group of this name are created. Also, ownership of configuration file will be granted to that user.

Make sure to allow TCP connection to WebSocket address and port specified in configuration in your firewall. Otherwise, monitor won't work.


**Fedora (and other RPM distributions):**

- run `python3 setup.py bdist_rpm --post-install ./install/postints` to generate binary `.rpm` package or download it from releases tab of monitor GitHub repository (it's architecture independent package)
- install package using `sudo dnf install ./dist/recodex-monitor-0.1.0-1.noarch.rpm` (number depends on actual version)

**Other Linux systems:**

- run installation as `python3 setup.py install --install-scripts /usr/bin`
- run postinst script as root - `sudo ./install/postinst`

## Configuration and usage

### Configuration
Configuration file is located in subdirectory `monitor` of standard ReCodEx configuration folder `/etc/recodex/`. It's in YAML format as all of the other configurations. Format is very similar to configurations of broker or workers. Example configuration file is here:

```{.yml}
---
websocket_uri:
    - "127.0.0.1"
    - 4567
zeromq_uri:
    - "127.0.0.1"
    - 7894
logger:
    file: "/var/log/recodex/monitor.log"
    level: "debug"
    max-size: 1048576  # 1 MB
    rotations: 3
...
```

### Configuration items

- **websocket_uri** - URI where is the endpoint of websocket connection. Must be visible to the clients (directly or through public proxy)!
    - string representation of IP address or a hostname
    - port number
- **zeromq_uri** - URI where is the endpoint of zeromq connection from broker. Could be hidden from public internet.
    - string representation of IP address or a hostname
    - port number
- **logger** - settings of logging
    - **file** - path with name of log file. Defaults to `/var/log/recodex/monitor.log`
    - **level** - logging level, one of "debug", "info", "warning", "error" and "critical"
    - **max-size** - maximum size of log file before rotation in bytes
    - **rotations** - number of rotations kept

### Usage
Preferred way to start monitor as a service is via systemd as `sudo systemctl start recodex-monitor.service`. After that, check if application lauch was successful using `sudo systemctl status recodex-monitor.service`. Now you should see green **Active (running)**.

Alternative starting command is `recodex-monitor -c /etc/recodex/monitor/config.yml` from command line. But note that it won't start monitor as a daemon.