recodex-wiki/Worker.md

Worker

Description

Worker's main role is securely execute given submission and possibly evaluate results against model solutions provided by submitter. Worker is logicaly divided into two parts:

• Listener - listens and communicates with Broker through ZeroMQ. It receives new jobs, communicates with Evaluator part and sends back results or progress.
• Evaluator - gets jobs to evaluate from Listener part, evaluate them (possibly in sandbox) and get to know to other part that evaluation ended. This part also communicates with Fileserver, downloads needed files and uploads detailed results.

Worker after getting evaluation request has to:

• Download the archive containing submitted source files and configuration file
• Download any supplementary files based on the configuration file, such as test inputs or helper programs (This is done on demand, using a fetch command in the assignment configuration)
• Evaluate the submission accordingly to job configuration
• During evaluation progress states can be sent back to Broker
• Upload the results of the evaluation to the Fileserver
• Notify Broker that the evaluation finished

Architecture

Picture below is overall internal architecture of worker which shows its defined classes with private variables and public functions. Vector version of this picture is available here.

Installation

Dependencies

Worker specific requirements are written in this section. Some parts of this guide may be different for each type of worker, for example Compiler principles worker is not fully covered here.

Install libarchive library (optional)

Installing this package will only speed up build process, otherwise libarchive is built from source.

• Debian package is libarchive-dev.
• RedHat packages are libarchive and libarchive-devel. These are probably not available for RHEL.

Install Isolate from source

First, we need to compile sandbox Isolate from source and install. Assume that we keep source code in /opt/src dir. For building man page you need to have package asciidoc installed.

$ cd /opt/src
$ git clone https://github.com/ioi/isolate.git
$ cd isolate
$ make
# make install && make install-doc

For proper work Isolate depends on several advanced features of the Linux kernel. Make sure that your kernel is compiled with CONFIG_PID_NS, CONFIG_IPC_NS, CONFIG_NET_NS, CONFIG_CPUSETS, CONFIG_CGROUP_CPUACCT, CONFIG_MEMCG. If your machine has swap enabled, also check CONFIG_MEMCG_SWAP. Which flags was your kernel compiled with can be found in /boot directory, for example in /boot/config-4.2.6-301.fc23.x86_64 file for kernel version 4.2.6-301. Red Hat distros should have these enabled by default, for Debian you you may want to add the parameters cgroup_enable=memory swapaccount=1 to the kernel command-line, which can be set using GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub.

For better reproducibility of results, some kernel parameters can be tweaked:

• Disable address space randomization. Create file /etc/sysctl.d/10-recodex.conf with content kernel.randomize_va_space=0. Changes will take effect after restart or run sysctl kernel.randomize_va_space=0 command.
• Disable dynamic CPU frequency scaling. This requires setting the cpufreq scaling governor to performance. TODO - do we really need it and is it worth higher power consumption? Red Hat setup, Debian setup

Configuration and usage

Following text describes how to set up and run worker program. It's supposed to have required binaries installed. For instructions see Installation section. Also, using systemd is recommended for best user experience, but it's not required. Almost all modern Linux distributions are using systemd now.

Installation of worker program does following step to your computer:

• create config file /etc/recodex/worker/config-1.yml
• create systemd unit file /etc/systemd/system/recodex-worker@.service
• put main binary to /usr/bin/recodex-worker
• put judges binaries to /usr/bin/recodex-judge-normal, /usr/bin/recodex-judge-shuffle and /usr/bin/recodex-judge-filter
• create system user and group recodex with /sbin/nologin shell (if not existing)
• create log directory /var/log/recodex
• set ownership of config (/etc/recodex) and log (/var/log/recodex) directories to recodex user and group

Default worker configuration

Worker should have some default configuration which is applied to worker itself or may be used in given jobs (implicitly if something is missing, or explicitly with special variables). This configuration should be hardcoded and can be rewritten by explicitly declared configuration file. Format of this configuration is yaml like in the job config.

Configuration items

Mandatory items are bold, optional italic.

• worker-id - unique identification of worker at one server. This id is used by isolate sanbox on linux systems, so make sure to meet isolates requirements (default is number from 1 to 999).
• broker-uri - URI of the broker (hostname, IP address, including port, ...)
• broker-ping-interval - time interval how often to send ping messages to broker. Used units are milliseconds.
• max-broker-liveness - specifies how many pings in a row can broker miss without making the worker dead.
• headers - headers specifies worker's capabilities
  • env - map of enviromental variables
  • threads - information about available threads for this worker
• hwgroup - hardware group of this worker. Hardware group must specify worker hardware and software capabilities and it's main item for broker routing decisions.
• working-directory - where will be stored all needed files. Can be the same for multiple workers on one server.
• file-managers - addresses and credentials to all file managers used (eq. all different frontends using this worker)
  • hostname - URI of file manager
  • username - username for http authentication (if needed)
  • password - password for http authentication (if needed)
• file-cache - configuration of caching feature
  • cache-dir - path to caching directory. Can be the same for mutltiple workers.
• logger - settings of logging capabilities
  • file - path to the logging file with name without suffix. /var/log/recodex/worker item will produce worker.log, worker.1.log, ...
  • level - level of logging, one of off, emerg, alert, critical, err, warn, notice, info and debug
  • max-size - maximal size of log file before rotating
  • rotations - number of rotation kept
• limits - default sandbox limits for this worker. All items are described on assignments page.

Example config file

worker-id: 1
broker-uri: tcp://localhost:9657
broker-ping-interval: 10  # milliseconds
max-broker-liveness: 10
headers:
    env:
        - c
        - python
    threads: 2
hwgroup: "group1"
working-directory: /tmp/recodex
file-managers:
    - hostname: "http://localhost:9999"  # port is optional
      username: ""  # can be ignored in specific modules
      password: ""  # can be ignored in specific modules
file-cache:  # only in case that there is cache module
    cache-dir: "/tmp/recodex/cache"
logger:
    file: "/var/log/recodex/worker"  # w/o suffix - actual names will be worker.log, worker.1.log, ...
    level: "debug"  # level of logging
    max-size: 1048576  # 1 MB; max size of file before log rotation
    rotations: 3  # number of rotations kept
limits:
    time: 5  # in secs
    wall-time: 6  # seconds
    extra-time: 2  # seconds
    stack-size: 0  # normal in KB, but 0 means no special limit
    memory: 50000  # in KB
    parallel: 1  # time and memory limits are merged
    disk-size: 50
    disk-files: 5
    environ-variable:
        ISOLATE_BOX: "/box"
        ISOLATE_TMP: "/tmp"
    bound-directories:
        - src: /tmp/recodex/eval_5
          dst: /evaluate
          mode: RW,NOEXEC

Running worker

For easy and comfortable managing worker is included systemd unit file. It integrates worker nicely into your Linux system and allow you to run worker automaticaly after system startup for example. It's supposed to have more than one worker on every server, so provided unit file is templated. Each instance of worker have unique string identifier, which is used for managing that instance through systemd. By default, only one worker instance is ready to use after installation. It's ID is "1".

• Starting worker with id "1" can be done this way:

# systemctl start recodex-worker@1.service

Check with

# systemctl status recodex-worker@1.service

if the worker is running. You should see "active (running)" message.

• Worker can be stopped or restarted accordigly using systemctl stop and systemctl restart commands.
• If you want to run worker after system startup, run:

# systemctl enable recodex-worker@1.service

For further information about using systemd please refer to systemd documentation.

Adding new worker

To add a new worker you need to do a few steps:

• Get unique string ID. Think of one.
• Copy default configuration file /etc/recodex/worker/config-1.yml to the same directory and name it config-<your_unique_ID>.yml
• Edit that config file to fit your needs. Note that you must at least change worker-id and logger file values to be unique.
• Run new instance using

# systemctl start recodex-worker@<your_unique_ID>.service

Sandboxes

TODO

Cleaner

TODO