recodex-wiki/Monitor.md

Monitor

Description

Monitor is part of ReCodEx solution for reporting progress of job evaluation back to user in the real time. It gets progress notifications from broker and sends them through WebSockets to clients' browsers. For now, it's meant as an optional part of whole solution, but for full experince it's recommended to use one.

Monitor is needed one per broker, that is one per separate ReCodEx instance. Also, monitor has to be publicly visible (has to have public IP address or be behind public proxy server) and also needs a connection to the broker. If the web application is using HTTPS, it's required to use a proxy for monitor to provide encryption over WebSockets. If this is not done, browsers of the users will block unencrypted connection and won't show the progress to the users.

Architecture

Monitor is written in Python, tested versions are 3.4 and 3.5. This language was chosen because it's already in project requirements (fileserver) and there are great libraries for ZeroMQ, WebSockets and asynchronous operations. This library saves system resources and provides us great amount of processed messages. Also, coding in Python was pretty simple and saves us time for improving the other parts of ReCodEx.

For monitor functionality there are some required packages. All of them are listed in requirements.txt file in the repository and can be installed by pip package manager as

$ pip install -r requirements.txt

Description of dependencies:

• zmq -- binding to ZeroMQ framework
• websockets -- framework for communication over WebSockets
• asyncio -- library for fast asynchronous operations
• pyyaml -- parsing YAML configuration files
• argparse -- parsing command line arguments

Message flow

Monitor runs in 2 threads. Thread 1 is the main thread, which initializes all components (logger for example), starts the other thread and runs the ZeroMQ part of the application. This thread receives and parses incomming messages from broker and forwards them to thread 2 sending logic.

Thread 2 is responsible for managing all of WebSocket connections asynchronously. Whole thread is one big asyncio event loop through which all actions are processed. None of custom data types in Python are thread-safe, so all events from other threads (actually only send_message method invocation) must be called within the event loop (via asyncio.loop.call_soon_threadsafe function). Please note, that most of the Python interpreters use Global Interpreter Lock, so there is actualy no parallelism in the performance point of view, but proper synchronization is still required!

Handling of incomming messages

Incomming ZeroMQ progress message is received and parsed to JSON format (same as our WebSocket communication format). JSON string is then passed to thread 2 for asynchronous sending. Each message has an identifier of channel where to send it to.

There can be multiple receivers to one channel id. Each one has separate asyncio.Queue instance where new messages are added. In addition to that, there is one list of all messages per channel. If a client connects a bit later than the point when monitor starts to receive messages, it'll receive all messages from the beginning. Messages are stored 5 minutes after last progress command (normally FINISHED) is received, then are permanently deleted.

Messages from client's queue are sent through corresponding WebSocket connection via main event loop as soon as possible. This approach with separate queue per connection is easy to implement and guarantees reliability and order of message delivery.

Installation

Installation will provide you following files:

• /usr/bin/recodex-monitor -- simple startup script located in PATH
• /etc/recodex/monitor/config.yml -- configuration file
• /etc/systemd/system/recodex-monitor.service -- systemd startup script
• code files will be installed in location depending on your system settings, mostly into /usr/lib/python3.5/site-packages/monitor/ or similar

Systemd script runs monitor binary as specific recodex user, so in postinst script user and group of this name are created. Also, ownership of configuration file will be granted to that user.

• RPM distributions can make and install binary package. This can be done like this:
  • run command

  $ python3 setup.py bdist_rpm --post-install ./install/postints
  

  to generate binary .rpm package or download precompiled one from releases tab of monitor GitHub repository (it's architecture independent package)
  • install package using

  # yum install ./dist/recodex-monitor-<version>-1.noarch.rpm
  

• Other Linux distributions can install cleaner straight

  $ python3 setup.py install --install-scripts /usr/bin
  # ./install/postinst
  

Configuration and usage

Configuration

Configuration file is located in subdirectory monitor of standard ReCodEx configuration folder /etc/recodex/. It's in YAML format as all of the other configurations. Format is very similar to configurations of broker or workers.

Configuration items

Description of configurable items, bold ones are required, italics ones are optional.

• websocket_uri -- URI where is the endpoint of websocket connection. Must be visible to the clients (directly or through public proxy)
  • string representation of IP address or a hostname
  • port number
• zeromq_uri -- URI where is the endpoint of zeromq connection from broker. Could be hidden from public internet.
  • string representation of IP address or a hostname
  • port number
• logger -- settings of logging
  • file -- path with name of log file. Defaults to /var/log/recodex/monitor.log
  • level -- logging level, one of "debug", "info", "warning", "error" and "critical"
  • max-size -- maximum size of log file before rotation in bytes
  • rotations -- number of rotations kept

Example configuration file

---
websocket_uri:
    - "127.0.0.1"
    - 4567
zeromq_uri:
    - "127.0.0.1"
    - 7894
logger:
    file: "/var/log/recodex/monitor.log"
    level: "debug"
    max-size: 1048576  # 1 MB
    rotations: 3
...

Usage

Preferred way to start monitor as a service is via systemd as the other parts of ReCodEx solution.

• Running monitor is fairly simple:

# systemctl start recodex-monitor.service

• Current state can be obtained by

# systemctl status recodex-monitor.service

You should see green Active (running).

• Setting up monitor to be started on system startup:

# systemctl enable recodex-monitor.service

Alternatively monitor can be started directly from command line. Note that this command won't start monitor as a daemon.

$ recodex-monitor -c /etc/recodex/monitor/config.yml