recodex-wiki/System-configuration.md

System configuration

Worker

Worker should have some default configuration which is applied to worker itself or may be used in given jobs (implicitly if something is missing, or explicitly with special variables). This configuration should be hardcoded and can be rewritten by explicitly declared configuration file. Format of this configuration is yaml with similar structure to job configuration.

Configuration items

Mandatory items are bold, optional italic.

• worker-id -- unique identification of worker at one server. This id is used by isolate sanbox on linux systems, so make sure to meet isolate's requirements (default is number from 1 to 999).
• worker-description -- human readable description of this worker
• broker-uri -- URI of the broker (hostname, IP address, including port, ...)
• broker-ping-interval -- time interval how often to send ping messages to broker. Used units are milliseconds.
• max-broker-liveness -- specifies how many pings in a row can broker miss without making the worker dead.
• headers -- map of headers specifies worker's capabilities
  • env -- list of enviromental variables which are sent to broker in init command
  • threads -- information about available threads for this worker
• hwgroup -- hardware group of this worker. Hardware group must specify worker hardware and software capabilities and it is main item for broker routing decisions.
• working-directory -- where will be stored all needed files. Can be the same for multiple workers on one server.
• file-managers -- addresses and credentials to all file managers used (eq. all different frontends using this worker)
  • hostname -- URI of file manager
  • username -- username for http authentication (if needed)
  • password -- password for http authentication (if needed)
• file-cache -- configuration of caching feature
  • cache-dir -- path to caching directory. Can be the same for multiple workers.
• logger -- settings of logging capabilities
  • file -- path to the logging file with name without suffix. /var/log/recodex/worker item will produce worker.log, worker.1.log, ...
  • level -- level of logging, one of off, emerg, alert, critical, err, warn, notice, info and debug
  • max-size -- maximal size of log file before rotating
  • rotations -- number of rotation kept
• limits -- default sandbox limits for this worker. All items are described in assignments section in job configuration description. If some limits are not set in job configuration, defaults from worker config will be used. In such case the worker's defaults will be set as the maximum for the job. Also, limits in job configuration cannot exceed limits from worker.

Example config file

worker-id: 1
broker-uri: tcp://localhost:9657
broker-ping-interval: 10  # milliseconds
max-broker-liveness: 10
headers:
    env:
        - c
        - cpp
    threads: 2
hwgroup: "group1"
working-directory: /tmp/recodex
file-managers:
    - hostname: "http://localhost:9999"  # port is optional
      username: ""  # can be ignored in specific modules
      password: ""  # can be ignored in specific modules
file-cache:  # only in case that there is cache module
    cache-dir: "/tmp/recodex/cache"
logger:
    file: "/var/log/recodex/worker" # w/o suffix - actual names will
	                                # be worker.log, worker.1.log,...
    level: "debug"  # level of logging
    max-size: 1048576  # 1 MB; max size of file before log rotation
    rotations: 3  # number of rotations kept
limits:
    time: 5  # in secs
    wall-time: 6  # seconds
    extra-time: 2  # seconds
    stack-size: 0  # normal in KB, but 0 means no special limit
    memory: 50000  # in KB
    parallel: 1
    disk-size: 50
    disk-files: 5
    environ-variable:
        ISOLATE_BOX: "/box"
        ISOLATE_TMP: "/tmp"
    bound-directories:
        - src: /tmp/recodex/eval_5
          dst: /evaluate
          mode: RW,NOEXEC

Isolate sandbox

New feature in version 1.3 is possibility of limit Isolate box to one or more cpu or memory node. This functionality is provided by cpusets kernel mechanism and is now integrated in isolate. It is allowed to set only cpuset.cpus and cpuset.mems which should be just fine for sandbox purposes. As kernel functionality further description can be found in manual page of cpuset or in Linux documentation in section linux/Documentation/cgroups/cpusets.txt. As previously stated this settings can be applied for particular isolate boxes and has to be written in isolate configuration. Standard configuration path should be /usr/local/etc/isolate but it may depend on your installation process. Configuration of cpuset in there is really simple and is described in example below.

box0.cpus = 0  # assign processor with ID 0 to isolate box with ID 0
box0.mems = 0  # assign memory node with ID 0
# if not set, linux by itself will decide where should
# the sandboxed programs run at
box2.cpus = 1-3  # assign range of processors to isolate box 2
box2.mems = 4-7  # assign range of memory nodes 
box3.cpus = 1,2,3  # assign list of processors to isolate box 3

• cpuset.cpus: Cpus limitation will restrict sandboxed program only to processor threads set in configuration. On hyperthreaded processors this means that all virtual threads are assignable, not only the physical ones. Value can be represented by single number, list of numbers separated by commas or range with hyphen delimiter.
• cpuset.mems: This value is particularly handy on NUMA systems which has several memory nodes. On standard desktop computers this value should always be zero because only one independent memory node is present. As stated in cpus limitation there can be single value, list of values separated by comma or range stated with hyphen.

Broker

Configuration items

Description of configurable items in broker's config. Mandatory items are bold, optional italic.

• clients -- specifies address and port to bind for clients (frontend instance)
  • address -- hostname or IP address as string (* for any)
  • port -- desired port
• workers -- specifies address and port to bind for workers
  • address -- hostname or IP address as string (* for any)
  • port -- desired port
  • max_liveness -- maximum amount of pings the worker can fail to send before it is considered disconnected
  • max_request_failures -- maximum number of times a job can fail (due to e.g. worker disconnect or a network error when downloading something from the fileserver) and be assigned again
• monitor -- settings of monitor service connection
  • address -- IP address of running monitor service
  • port -- desired port
• notifier -- details of connection which is used in case of errors and good to know states
  • address -- address where frontend API runs
  • port -- desired port
  • username -- username which can be used for HTTP authentication
  • password -- password which can be used for HTTP authentication
• logger -- settings of logging capabilities
  • file -- path to the logging file with name without suffix. /var/log/recodex/broker item will produce broker.log, broker.1.log, ...
  • level -- level of logging, one of off, emerg, alert, critical, err, warn, notice, info and debug
  • max-size -- maximal size of log file before rotating
  • rotations -- number of rotation kept

Example config file

# Address and port for clients (frontend)
clients:
    address: "*"
    port: 9658
# Address and port for workers
workers:
    address: "*"
    port: 9657
    max_liveness: 10
    max_request_failures: 3
monitor:
    address: "127.0.0.1"
    port: 7894
notifier:
    address: "127.0.0.1"
    port: 8080
    username: ""
    password: ""
logger:
    file: "/var/log/recodex/broker"  # w/o suffix - actual names will be
	                                 # broker.log, broker.1.log, ...
    level: "debug"  # level of logging
    max-size: 1048576  # 1 MB; max size of file before log rotation
    rotations: 3  # number of rotations kept

Monitor

Configuration file is located in subdirectory monitor of standard ReCodEx configuration folder /etc/recodex/. It is in YAML format as all of the other configurations. Format is very similar to configurations of broker or workers.

Configuration items

Description of configurable items, bold ones are required, italics ones are optional.

• websocket_uri -- URI where is the endpoint of websocket connection. Must be visible to the clients (directly or through public proxy)
  • string representation of IP address or a hostname
  • port number
• zeromq_uri -- URI where is the endpoint of zeromq connection from broker. Could be hidden from public internet.
  • string representation of IP address or a hostname
  • port number
• logger -- settings of logging
  • file -- path with name of log file. Defaults to /var/log/recodex/monitor.log
  • level -- logging level, one of "debug", "info", "warning", "error" and "critical"
  • max-size -- maximum size of log file before rotation in bytes
  • rotations -- number of rotations kept

Example configuration file

---
websocket_uri:
    - "127.0.0.1"
    - 4567
zeromq_uri:
    - "127.0.0.1"
    - 7894
logger:
    file: "/var/log/recodex/monitor.log"
    level: "debug"
    max-size: 1048576  # 1 MB
    rotations: 3
...

Cleaner

Configuration items

• cache-dir -- directory which cleaner manages
• file-age -- file age in seconds which are considered outdated and will be deleted

Example configuration

cache-dir: "/tmp"
file-age: "3600"  # in seconds

REST API

The API can be configured in config.neon and config.local.neon files in app/config directory. The first file is predefined by authors and should not be modified. The second one is not present and could be created by copying config.local.neon.example template in the config directory. Local configuration have higher precedence, so it will override default values from config.neon.

Configurable items

Description of configurable items. All timeouts are in milliseconds if not stated otherwise.

• accessManager -- configuration of access token in JWT standard. Do not modify unless you really know what are you doing.
• fileServer -- connection to fileserver
  • address -- URI of fileserver
  • auth -- username and password for HTTP basic authentication
  • timeouts -- connection timeout for establishing new connection and request timeout for completing one request
• broker -- connection to broker
  • address -- URI of broker
  • auth -- username and password for broker callback authentication back to API
  • timeouts -- ack timeout for first response that broker receives the message, send timeout how long try to send new job to the broker and result timeout how long to wait for confirmation if job can be processed or not
• monitor -- connection to monitor
  • address -- URI of monitor
• CAS -- CAS external authentication
  • serviceId -- visible identifier of this service
  • ldapConnection -- parameters for connecting to LDAP, hostname, base_dn, port, security and bindName
  • fields -- names of LDAP keys for informations as email, firstName and lastName
• emails -- common configuration for sending email (addresses and template variables)
  • apiUrl -- base URL of API server including port (for referencing pictures in messages)
  • footerUrl -- link in the message footer
  • siteName -- name of frontend (ReCodEx, or KSP for unique instance for KSP course)
  • githubUrl -- URL to GitHub repository of this project
  • from -- sending email address
• failures -- admin messages on errors
  • emails -- additional info for sending mails, to is admin mail address, from is source address, subjectPrefix is prefix of mail subject
• forgottenPassword -- user messages for changing passwords
  • redirectUrl -- URL of web application where the password can be changed
  • tokenExpiration -- expiration timeout of temporary token (in seconds)
  • emails -- additional info for sending mails, from is source address and subjectPrefix is prefix of mail subject
• mail -- configuration of sending mails
  • smtp -- using SMTP server, have to be "true"
  • host -- address of the server
  • port -- sending port (common values are 25, 465, 587)
  • username -- login to the server
  • password -- password to the server
  • secure -- security, values are empty for no security, "ssl" or "tls"
  • context -- additional parameters, depending on used mail engine. For examle self-signed certificates can be allowed as verify_peer and verify_peer_name to false and allow_self_signed to true under ssl key (see example).

Outside the parameters section of configuration is configuration for Doctrine. It is ORM framework which maps PHP objects (entities) into database tables and rows. The configuration is simple, required items are only user, password and host with dbname, i.e. address of database computer (mostly localhost) with name of ReCodEx database.

Example local configuration file

parameters:
  accessManager:
    leeway: 60
    issuer: https://recodex.projekty.ms.mff.cuni.cz
    audience: https://recodex.projekty.ms.mff.cuni.cz
    expiration: 86400  # 24 hours in seconds
    usedAlgorithm: HS256
    allowedAlgorithms:
      - HS256
    verificationKey: "recodex-123"
  fileServer:
    address: http://127.0.0.1:9999
    auth:
      username: "user"
      password: "pass"
    timeouts:
      connection: 500
  broker:
    address: tcp://127.0.0.1:9658
    auth:
      username: "user"
      password: "pass"
    timeouts:
      ack: 100
      send: 5000
      result: 1000
  monitor:
    address: wss://recodex.projekty.ms.mff.cuni.cz:4443/ws
  CAS:
    serviceId: "cas-uk"
    ldapConnection:
      hostname: "ldap.cuni.cz"
      base_dn: "ou=people,dc=cuni,dc=cz"
      port: 389
      security: SSL
      bindName: "cunipersonalid"
    fields:
      email: "mail"
      firstName: "givenName"
      lastName: "sn"
  emails:
    apiUrl: https://recodex.projekty.ms.mff.cuni.cz:4000
    footerUrl: https://recodex.projekty.ms.mff.cuni.cz
    siteName: "ReCodEx"
    githubUrl: https://github.com/ReCodEx
    from: "ReCodEx <noreply@example.com>"
  failures:
    emails:
      to: "Admin Name <admin@example.com>"
      from: %emails.from%
      subjectPrefix: "ReCodEx Failure Report - "
  forgottenPassword:
    redirectUrl: "https://recodex.projekty.ms.mff.cuni.cz/
                  forgotten-password/change"
    tokenExpiration: 600 # 10 minues
    emails:
      from: %emails.from%
      subjectPrefix: "ReCodEx Forgotten Password Request - "
  mail:
    smtp: true
    host: "smtp.ps.stdin.cz"
    port: 587
    username: "user"
    password: "pass"
    secure: "tls"
    context:
      ssl:
        verify_peer: false
        verify_peer_name: false
        allow_self_signed: true
doctrine:
  user: "user"
  password: "pass"
  host: localhost
  dbname: "recodex-api"

Web application

Configurable items

Description of configurable options. Bold are required values, optional ones are in italics.

• NODE_ENV -- mode of the server
• API_BASE -- base address of API server, including port and API version
• PORT -- port where the app is listening
• WEBPACK_DEV_SERVER_PORT -- port for webpack dev server when running in development mode. Default one is 8081, this option might be useful when this port is necessary for some other service.

Example configuration file

NODE_ENV=production
API_BASE=https://recodex.projekty.ms.mff.cuni.cz:4000/v1
PORT=8080