recodex-wiki/System-configuration.md

# System configuration

## Worker

Worker should have some default configuration which is applied to worker itself
or may be used in given jobs (implicitly if something is missing, or explicitly
with special variables). This configuration should be hardcoded and can be
rewritten by explicitly declared configuration file. Format of this
configuration is yaml with similar structure to job configuration.

### Configuration items

Mandatory items are bold, optional italic.

- **worker-id** -- unique identification of worker at one server. This id is
  used by _isolate_ sanbox on linux systems, so make sure to meet isolate's
  requirements (default is number from 1 to 999).
- _worker-description_ -- human readable description of this worker
- **broker-uri** -- URI of the broker (hostname, IP address, including port,
  ...)
- _broker-ping-interval_ -- time interval how often to send ping messages to
  broker. Used units are milliseconds.
- _max-broker-liveness_ -- specifies how many pings in a row can broker miss
  without making the worker dead.
- _headers_ -- map of headers specifies worker's capabilities
	- _env_ -- list of enviromental variables which are sent to broker in init
	  command
	- _threads_ -- information about available threads for this worker
- **hwgroup** -- hardware group of this worker. Hardware group must specify
  worker hardware and software capabilities and it is main item for broker
  routing decisions.
- _working-directory_ -- where will be stored all needed files. Can be the same
  for multiple workers on one server.
- **file-managers** -- addresses and credentials to all file managers used (eq.
  all different frontends using this worker)
	- **hostname** -- URI of file manager
	- _username_ -- username for http authentication (if needed)
	- _password_ -- password for http authentication (if needed)
- _file-cache_ -- configuration of caching feature
	- _cache-dir_ -- path to caching directory. Can be the same for multiple
	  workers.
- _logger_ -- settings of logging capabilities
	- _file_ -- path to the logging file with name without suffix.
	  `/var/log/recodex/worker` item will produce `worker.log`, `worker.1.log`,
	  ...
	- _level_ -- level of logging, one of `off`, `emerg`, `alert`, `critical`,
	  `err`, `warn`, `notice`, `info` and `debug`
	- _max-size_ -- maximal size of log file before rotating
	- _rotations_ -- number of rotation kept
- _limits_ -- default sandbox limits for this worker. All items are described in
  assignments section in job configuration description. If some limits are not
  set in job configuration, defaults from worker config will be used. In such
  case the worker's defaults will be set as the maximum for the job. Also,
  limits in job configuration cannot exceed limits from worker.

### Example config file

```{.yml}
worker-id: 1
broker-uri: tcp://localhost:9657
broker-ping-interval: 10  # milliseconds
max-broker-liveness: 10
headers:
    env:
        - c
        - cpp
    threads: 2
hwgroup: "group1"
working-directory: /tmp/recodex
file-managers:
    - hostname: "http://localhost:9999"  # port is optional
      username: ""  # can be ignored in specific modules
      password: ""  # can be ignored in specific modules
file-cache:  # only in case that there is cache module
    cache-dir: "/tmp/recodex/cache"
logger:
    file: "/var/log/recodex/worker" # w/o suffix - actual names will
	                                # be worker.log, worker.1.log,...
    level: "debug"  # level of logging
    max-size: 1048576  # 1 MB; max size of file before log rotation
    rotations: 3  # number of rotations kept
limits:
    time: 5  # in secs
    wall-time: 6  # seconds
    extra-time: 2  # seconds
    stack-size: 0  # normal in KB, but 0 means no special limit
    memory: 50000  # in KB
    parallel: 1
    disk-size: 50
    disk-files: 5
    environ-variable:
        ISOLATE_BOX: "/box"
        ISOLATE_TMP: "/tmp"
    bound-directories:
        - src: /tmp/recodex/eval_5
          dst: /evaluate
          mode: RW,NOEXEC
```

### Isolate sandbox

New feature in version 1.3 is possibility of limit Isolate box to one or more
cpu or memory node. This functionality is provided by _cpusets_ kernel mechanism
and is now integrated in isolate. It is allowed to set only `cpuset.cpus` and
`cpuset.mems` which should be just fine for sandbox purposes. As kernel
functionality further description can be found in manual page of _cpuset_ or in
Linux documentation in section `linux/Documentation/cgroups/cpusets.txt`. As
previously stated this settings can be applied for particular isolate boxes and
has to be written in isolate configuration. Standard configuration path should
be `/usr/local/etc/isolate` but it may depend on your installation process.
Configuration of _cpuset_ in there is really simple and is described in example
below.

```
box0.cpus = 0  # assign processor with ID 0 to isolate box with ID 0
box0.mems = 0  # assign memory node with ID 0
# if not set, linux by itself will decide where should
# the sandboxed programs run at
box2.cpus = 1-3  # assign range of processors to isolate box 2
box2.mems = 4-7  # assign range of memory nodes 
box3.cpus = 1,2,3  # assign list of processors to isolate box 3
```

- **cpuset.cpus:** Cpus limitation will restrict sandboxed program only to
  processor threads set in configuration. On hyperthreaded processors this means
  that all virtual threads are assignable, not only the physical ones. Value can
  be represented by single number, list of numbers separated by commas or range
  with hyphen delimiter.
- **cpuset.mems:** This value is particularly handy on NUMA systems which has
  several memory nodes. On standard desktop computers this value should always
  be zero because only one independent memory node is present. As stated in
  `cpus` limitation there can be single value, list of values separated by comma
  or range stated with hyphen.

## Broker

### Configuration items

Description of configurable items in broker's config. Mandatory items are bold,
optional italic.

- _clients_ -- specifies address and port to bind for clients (frontend
  instance)
	- _address_ -- hostname or IP address as string (`*` for any)
	- _port_ -- desired port
- _workers_ -- specifies address and port to bind for workers
	- _address_ -- hostname or IP address as string (`*` for any)
	- _port_ -- desired port
	- _max_liveness_ -- maximum amount of pings the worker can fail to send
	  before it is considered disconnected
	- _max_request_failures_ -- maximum number of times a job can fail (due to
	  e.g. worker disconnect or a network error when downloading something from
	  the fileserver) and be assigned again 
- _monitor_ -- settings of monitor service connection
	- _address_ -- IP address of running monitor service
	- _port_ -- desired port
- _notifier_ -- details of connection which is used in case of errors and good
  to know states
	- _address_ -- address where frontend API runs
	- _port_ -- desired port
	- _username_ -- username which can be used for HTTP authentication
	- _password_ -- password which can be used for HTTP authentication
- _logger_ -- settings of logging capabilities
	- _file_ -- path to the logging file with name without suffix.
	  `/var/log/recodex/broker` item will produce `broker.log`, `broker.1.log`,
	  ...
	- _level_ -- level of logging, one of `off`, `emerg`, `alert`, `critical`,
	  `err`, `warn`, `notice`, `info` and `debug`
	- _max-size_ -- maximal size of log file before rotating
	- _rotations_ -- number of rotation kept

### Example config file

```{.yml}
# Address and port for clients (frontend)
clients:
    address: "*"
    port: 9658
# Address and port for workers
workers:
    address: "*"
    port: 9657
    max_liveness: 10
    max_request_failures: 3
monitor:
    address: "127.0.0.1"
    port: 7894
notifier:
    address: "127.0.0.1"
    port: 8080
    username: ""
    password: ""
logger:
    file: "/var/log/recodex/broker"  # w/o suffix - actual names will be
	                                 # broker.log, broker.1.log, ...
    level: "debug"  # level of logging
    max-size: 1048576  # 1 MB; max size of file before log rotation
    rotations: 3  # number of rotations kept
```

## Monitor

Configuration file is located in subdirectory `monitor` of standard ReCodEx
configuration folder `/etc/recodex/`. It is in YAML format as all of the other
configurations. Format is very similar to configurations of broker or workers.

### Configuration items

Description of configurable items, bold ones are required, italics ones are
optional.

- _websocket_uri_ -- URI where is the endpoint of websocket connection. Must be
  visible to the clients (directly or through public proxy)
	- string representation of IP address or a hostname
	- port number
- _zeromq_uri_ -- URI where is the endpoint of zeromq connection from broker.
  Could be hidden from public internet.
	- string representation of IP address or a hostname
	- port number
- _logger_ -- settings of logging
	- _file_ -- path with name of log file. Defaults to
	  `/var/log/recodex/monitor.log`
	- _level_ -- logging level, one of "debug", "info", "warning", "error" and
	  "critical"
	- _max-size_ -- maximum size of log file before rotation in bytes
	- _rotations_ -- number of rotations kept

### Example configuration file

```{.yml}
---
websocket_uri:
    - "127.0.0.1"
    - 4567
zeromq_uri:
    - "127.0.0.1"
    - 7894
logger:
    file: "/var/log/recodex/monitor.log"
    level: "debug"
    max-size: 1048576  # 1 MB
    rotations: 3
...
```

## Cleaner

### Configuration items
- **cache-dir** -- directory which cleaner manages
- **file-age** -- file age in seconds which are considered outdated and will be deleted

### Example configuration
```{.yml}
cache-dir: "/tmp"
file-age: "3600"  # in seconds
```

## REST API

The API can be configured in `config.neon` and `config.local.neon` files in
`app/config` directory. The first file is predefined by authors and should not
be modified. The second one is not present and could be created by copying
`config.local.neon.example` template in the config directory. Local
configuration have higher precedence, so it will override default values from
`config.neon`.

### Configurable items

Description of configurable items. All timeouts are in milliseconds if not
stated otherwise.

- accessManager -- configuration of access token in [JWT
  standard](https://www.rfc-editor.org/rfc/rfc7519.txt). Do **not** modify
  unless you really know what are you doing.
- fileServer -- connection to fileserver
	- address -- URI of fileserver
	- auth -- _username_ and _password_ for HTTP basic authentication
	- timeouts -- _connection_ timeout for establishing new connection and
	  _request_ timeout for completing one request
- broker -- connection to broker
	- address -- URI of broker
	- auth -- _username_ and _password_ for broker callback authentication back
	  to API
	- timeouts -- _ack_ timeout for first response that broker receives the
	  message, _send_ timeout how long try to send new job to the broker and
	  _result_ timeout how long to wait for confirmation if job can be processed
	  or not
- monitor -- connection to monitor
	- address -- URI of monitor
- CAS -- CAS external authentication
	- serviceId -- visible identifier of this service
	- ldapConnection -- parameters for connecting to LDAP, _hostname_,
	  _base_dn_, _port_, _security_ and _bindName_
	- fields -- names of LDAP keys for informations as _email_, _firstName_ and
	  _lastName_
- emails -- common configuration for sending email (addresses and template
  variables)
	- apiUrl -- base URL of API server including port (for referencing pictures
	  in messages)
	- footerUrl -- link in the message footer
	- siteName -- name of frontend (ReCodEx, or KSP for unique instance for KSP
	  course)
	- githubUrl -- URL to GitHub repository of this project
	- from -- sending email address
- failures -- admin messages on errors
	- emails -- additional info for sending mails, _to_ is admin mail address,
	  _from_ is source address, _subjectPrefix_ is prefix of mail subject
- forgottenPassword -- user messages for changing passwords
	- redirectUrl -- URL of web application where the password can be changed
	- tokenExpiration -- expiration timeout of temporary token (in seconds)
	- emails -- additional info for sending mails, _from_ is source address and
	  _subjectPrefix_ is prefix of mail subject
- mail -- configuration of sending mails
	- smtp -- using SMTP server, have to be "true"
	- host -- address of the server
	- port -- sending port (common values are 25, 465, 587)
	- username -- login to the server
	- password -- password to the server
	- secure -- security, values are empty for no security, "ssl" or "tls"
	- context -- additional parameters, depending on used mail engine. For
	  examle self-signed certificates can be allowed as _verify_peer_ and
	  _verify_peer_name_ to false and _allow_self_signed_ to true under _ssl_
	  key (see example).

Outside the parameters section of configuration is configuration for Doctrine.
It is ORM framework which maps PHP objects (entities) into database tables and
rows. The configuration is simple, required items are only _user_, _password_
and _host_ with _dbname_, i.e. address of database computer (mostly localhost)
with name of ReCodEx database.

### Example local configuration file

```{.yml}
parameters:
  accessManager:
    leeway: 60
    issuer: https://recodex.projekty.ms.mff.cuni.cz
    audience: https://recodex.projekty.ms.mff.cuni.cz
    expiration: 86400  # 24 hours in seconds
    usedAlgorithm: HS256
    allowedAlgorithms:
      - HS256
    verificationKey: "recodex-123"
  fileServer:
    address: http://127.0.0.1:9999
    auth:
      username: "user"
      password: "pass"
    timeouts:
      connection: 500
  broker:
    address: tcp://127.0.0.1:9658
    auth:
      username: "user"
      password: "pass"
    timeouts:
      ack: 100
      send: 5000
      result: 1000
  monitor:
    address: wss://recodex.projekty.ms.mff.cuni.cz:4443/ws
  CAS:
    serviceId: "cas-uk"
    ldapConnection:
      hostname: "ldap.cuni.cz"
      base_dn: "ou=people,dc=cuni,dc=cz"
      port: 389
      security: SSL
      bindName: "cunipersonalid"
    fields:
      email: "mail"
      firstName: "givenName"
      lastName: "sn"
  emails:
    apiUrl: https://recodex.projekty.ms.mff.cuni.cz:4000
    footerUrl: https://recodex.projekty.ms.mff.cuni.cz
    siteName: "ReCodEx"
    githubUrl: https://github.com/ReCodEx
    from: "ReCodEx <noreply@example.com>"
  failures:
    emails:
      to: "Admin Name <admin@example.com>"
      from: %emails.from%
      subjectPrefix: "ReCodEx Failure Report - "
  forgottenPassword:
    redirectUrl: "https://recodex.projekty.ms.mff.cuni.cz/
                  forgotten-password/change"
    tokenExpiration: 600 # 10 minues
    emails:
      from: %emails.from%
      subjectPrefix: "ReCodEx Forgotten Password Request - "
  mail:
    smtp: true
    host: "smtp.ps.stdin.cz"
    port: 587
    username: "user"
    password: "pass"
    secure: "tls"
    context:
      ssl:
        verify_peer: false
        verify_peer_name: false
        allow_self_signed: true
doctrine:
  user: "user"
  password: "pass"
  host: localhost
  dbname: "recodex-api"
```

## Web application

### Configurable items

Description of configurable options. Bold are required values, optional ones are
in italics.

- **NODE_ENV** -- mode of the server
- **API_BASE** -- base address of API server, including port and API version
- **PORT** -- port where the app is listening
- _WEBPACK_DEV_SERVER_PORT_ -- port for webpack dev server when running in
  development mode. Default one is 8081, this option might be useful when this
  port is necessary for some other service.

### Example configuration file

```
NODE_ENV=production
API_BASE=https://recodex.projekty.ms.mff.cuni.cz:4000/v1
PORT=8080
```


<!---
// vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
-->