# System configuration This section describes configuration of ReCodEx components. Bold items in lists describing the values are mandatory, italic ones are optional. ## Worker Worker have a default configuration which is applied to worker itself or is used in given jobs (implicitly if something is missing, or explicitly with special variables). This configuration is hardcoded in worker sources and can be rewritten by explicitly declared configuration file. Format of this configuration is YAML file with similar structure as job configuration. The default location is `/etc/recodex/worker/config-.yml` where `N` is identifier of the particular worker instance. ### Configuration items - **worker-id** -- unique identification of worker at one server. This id is used by _isolate_ sandbox on linux systems, so make sure to meet requirements of the isolate (default is number from 1 to 999). - _worker-description_ -- human readable description of this worker - **broker-uri** -- URI of the broker (hostname, IP address, including port, ...) - _broker-ping-interval_ -- time interval how often to send ping messages to broker. Used units are milliseconds. - _max-broker-liveness_ -- specifies how many pings in a row can broker miss without making the worker dead. - _headers_ -- map of headers specifies worker's capabilities - _env_ -- list of environmental variables which are sent to broker in init command - _threads_ -- information about available threads for this worker - **hwgroup** -- hardware group of this worker. Hardware group must specify worker hardware and software capabilities and it is main item for broker routing decisions. - _working-directory_ -- where will be stored all needed files. Can be the same for multiple workers on one server. - **file-managers** -- addresses and credentials to all file managers used (eq. all different frontends using this worker) - **hostname** -- URI of file manager - _username_ -- username for http authentication (if needed) - _password_ -- password for http authentication (if needed) - _file-cache_ -- configuration of caching feature - _cache-dir_ -- path to caching directory. Can be the same for multiple workers. - _logger_ -- settings of logging capabilities - _file_ -- path to the logging file with name without suffix. `/var/log/recodex/worker` item will produce `worker.log`, `worker.1.log`, ... - _level_ -- level of logging, one of `off`, `emerg`, `alert`, `critical`, `err`, `warn`, `notice`, `info` and `debug` - _max-size_ -- maximal size of log file before rotating in bytes - _rotations_ -- number of rotation kept - _limits_ -- default sandbox limits for this worker. All items are described in assignments section in job configuration description. If some limits are not set in job configuration, defaults from worker config will be used. In such case the worker's defaults will be set as the maximum for the job. Also, limits in job configuration cannot exceed limits from worker. - **max-output-length** -- used for `tasks.{task}.sandbox.output` option, defined in bytes, applied to both stdout and stderr and is not divided, both will get this value - **cleanup-submission** -- if set to true, then files produced during evaluation of submission will be deleted at the end, extra caution is advised because this setup can cause extensive disk usage ### Example config file ```{.yml} worker-id: 1 broker-uri: tcp://localhost:9657 broker-ping-interval: 10 # milliseconds max-broker-liveness: 10 headers: env: - c - cpp threads: 2 hwgroup: "group1" working-directory: /tmp/recodex file-managers: - hostname: "http://localhost:9999" # port is optional username: "" # can be ignored in specific modules password: "" # can be ignored in specific modules file-cache: # only in case that there is cache module cache-dir: "/tmp/recodex/cache" logger: file: "/var/log/recodex/worker" # w/o suffix - actual names will # be worker.log, worker.1.log,... level: "debug" # level of logging max-size: 1048576 # 1 MB; max size of file before log rotation rotations: 3 # number of rotations kept limits: time: 5 # in secs wall-time: 6 # seconds extra-time: 2 # seconds stack-size: 0 # normal in KB, but 0 means no special limit memory: 50000 # in KB parallel: 1 disk-size: 50 disk-files: 5 environ-variable: ISOLATE_BOX: "/box" ISOLATE_TMP: "/tmp" bound-directories: - src: /tmp/recodex/eval_5 dst: /evaluate mode: RW,NOEXEC ``` ### Isolate sandbox New feature of the 1.3 version is a possibility of limit Isolate box to one or more CPU or memory nodes. This functionality is provided by _cpusets_ kernel mechanism and is now integrated into Isolate. It is allowed to set only `cpuset.cpus` and `cpuset.mems` which should be just fine for sandbox purposes. As a kernel functionality further description can be found in manual page of _cpuset_ or in Linux documentation in section `linux/Documentation/cgroups/cpusets.txt`. As previously stated this settings can be applied for particular Isolate boxes and has to be written in Isolate configuration. Standard configuration path should be `/usr/local/etc/isolate` but it may depend on your installation process. Configuration of _cpuset_ in there is described in example below. ``` box0.cpus = 0 # assign processor with ID 0 to isolate box with ID 0 box0.mems = 0 # assign memory node with ID 0 # if not set, linux by itself will decide where should # the sandboxed programs run at box2.cpus = 1-3 # assign range of processors to isolate box 2 box2.mems = 4-7 # assign range of memory nodes box3.cpus = 1,2,3 # assign list of processors to isolate box 3 ``` - _cpuset.cpus:_ Cpus limitation will restrict sandboxed program only to processor threads set in configuration. On hyperthreaded processors this means that all virtual threads are assignable, not only the physical ones. Value can be represented by single number, list of numbers separated by commas or range with hyphen delimiter. - _cpuset.mems:_ This value is particularly handy on NUMA systems which has several memory nodes. On standard desktop computers this value should always be zero because only one independent memory node is present. As stated in `cpus` limitation there can be single value, list of values separated by comma or range stated with hyphen. ## Broker The default location for broker configuration file is `/etc/recodex/broker/config.yml`. ### Configuration items - _clients_ -- specifies address and port to bind for clients (frontend instance) - _address_ -- hostname or IP address as string (`*` for any) - _port_ -- desired port - _workers_ -- specifies address and port to bind for workers - _address_ -- hostname or IP address as string (`*` for any) - _port_ -- desired port - _max_liveness_ -- maximum amount of pings the worker can fail to send before it is considered disconnected - _max_request_failures_ -- maximum number of times a job can fail (due to e.g. worker disconnect or a network error when downloading something from the fileserver) and be assigned again - _monitor_ -- settings of monitor service connection - _address_ -- IP address of running monitor service - _port_ -- desired port - _notifier_ -- details of connection which is used in case of errors and good to know states - _address_ -- address where frontend API runs - _port_ -- desired port - _username_ -- username which can be used for HTTP authentication - _password_ -- password which can be used for HTTP authentication - _logger_ -- settings of logging capabilities - _file_ -- path to the logging file with name without suffix. `/var/log/recodex/broker` item will produce `broker.log`, `broker.1.log`, ... - _level_ -- level of logging, one of `off`, `emerg`, `alert`, `critical`, `err`, `warn`, `notice`, `info` and `debug` - _max-size_ -- maximal size of log file before rotating - _rotations_ -- number of rotation kept ### Example config file ```{.yml} # Address and port for clients (frontend) clients: address: "*" port: 9658 # Address and port for workers workers: address: "*" port: 9657 max_liveness: 10 max_request_failures: 3 monitor: address: "127.0.0.1" port: 7894 notifier: address: "127.0.0.1" port: 8080 username: "" password: "" logger: file: "/var/log/recodex/broker" # w/o suffix - actual names will be # broker.log, broker.1.log, ... level: "debug" # level of logging max-size: 1048576 # 1 MB; max size of file before log rotation rotations: 3 # number of rotations kept ``` ## Monitor Configuration file is located in directory `/etc/recodex/monitor/` by default. It is in YAML format as all of the other configurations. ### Configuration items Description of configurable items, bold ones are required, italics ones are optional. - _websocket_uri_ -- URI where is the endpoint of WebSocket connection. Must be visible to the clients (directly or through public proxy) - string representation of IP address or a hostname - port number - _zeromq_uri_ -- URI where is the endpoint of ZeroMQ connection from broker. Could be hidden from public internet. - string representation of IP address or a hostname - port number - _logger_ -- settings of logging - _file_ -- path with name of log file. Defaults to `/var/log/recodex/monitor.log` - _level_ -- logging level, one of "debug", "info", "warning", "error" and "critical" - _max-size_ -- maximum size of log file before rotation in bytes - _rotations_ -- number of rotations kept ### Example configuration file ```{.yml} --- websocket_uri: - "127.0.0.1" - 4567 zeromq_uri: - "127.0.0.1" - 7894 logger: file: "/var/log/recodex/monitor.log" level: "debug" max-size: 1048576 # 1 MB rotations: 3 ... ``` ## Cleaner The default location for cleaner configuration file is `/etc/recodex/cleaner/config.yml`. ### Configuration items - **cache-dir** -- directory which cleaner manages - **file-age** -- file age in seconds which is considered as outdated and will be marked for deletion ### Example configuration ```{.yml} cache-dir: "/tmp" file-age: "3600" # in seconds ``` ## REST API The API can be configured in `config.neon` and `config.local.neon` files in `app/config` directory of the API project source tree. The first file is predefined by authors and should not be modified. The second one is not present and could be created by copying `config.local.neon.example` template in the config directory. Local configuration have higher precedence, so it will override default values from `config.neon`. ### Configurable items Description of configurable items. All timeouts are in milliseconds if not stated otherwise. - accessManager -- configuration of access token in [JWT standard](https://www.rfc-editor.org/rfc/rfc7519.txt). Do **not** modify unless you really know what are you doing. - fileServer -- connection to fileserver - address -- URI of fileserver - auth -- _username_ and _password_ for HTTP basic authentication - timeouts -- _connection_ timeout for establishing new connection and _request_ timeout for completing one request - broker -- connection to broker - address -- URI of broker - auth -- _username_ and _password_ for broker callback authentication back to API - timeouts -- _ack_ timeout for first response that broker receives the message, _send_ timeout how long try to send new job to the broker and _result_ timeout how long to wait for confirmation if job can be processed or not - monitor -- connection to monitor - address -- URI of monitor - CAS -- CAS external authentication - serviceId -- visible identifier of this service - ldapConnection -- parameters for connecting to LDAP, _hostname_, _base_dn_, _port_, _security_ and _bindName_ - fields -- names of LDAP keys for informations as _email_, _firstName_ and _lastName_ - emails -- common configuration for sending email (addresses and template variables) - apiUrl -- base URL of API server including port (for referencing pictures in messages) - footerUrl -- link in the message footer - siteName -- name of frontend (ReCodEx, or KSP for unique instance for KSP course) - githubUrl -- URL to GitHub repository of this project - from -- sending email address - failures -- admin messages on errors - emails -- additional info for sending mails, _to_ is admin mail address, _from_ is source address, _subjectPrefix_ is prefix of mail subject - forgottenPassword -- user messages for changing passwords - redirectUrl -- URL of web application where the password can be changed - tokenExpiration -- expiration timeout of temporary token (in seconds) - emails -- additional info for sending mails, _from_ is source address and _subjectPrefix_ is prefix of mail subject - mail -- configuration of sending mails - smtp -- using SMTP server, have to be "true" - host -- address of the server - port -- sending port (common values are 25, 465, 587) - username -- login to the server - password -- password to the server - secure -- security, values are empty for no security, "ssl" or "tls" - context -- additional parameters, depending on used mail engine. For examle self-signed certificates can be allowed as _verify_peer_ and _verify_peer_name_ to false and _allow_self_signed_ to true under _ssl_ key (see example). Outside the parameters section of configuration is configuration for Doctrine. It is ORM framework which maps PHP objects (entities) into database tables and rows. The configuration is simple, required items are only _user_, _password_ and _host_ with _dbname_, i.e. address of database computer (mostly localhost) with name of ReCodEx database. ### Example local configuration file ```{.yml} parameters: accessManager: leeway: 60 issuer: https://recodex.projekty.ms.mff.cuni.cz audience: https://recodex.projekty.ms.mff.cuni.cz expiration: 86400 # 24 hours in seconds usedAlgorithm: HS256 allowedAlgorithms: - HS256 verificationKey: "recodex-123" fileServer: address: http://127.0.0.1:9999 auth: username: "user" password: "pass" timeouts: connection: 500 broker: address: tcp://127.0.0.1:9658 auth: username: "user" password: "pass" timeouts: ack: 100 send: 5000 result: 1000 monitor: address: wss://recodex.projekty.ms.mff.cuni.cz:4443/ws CAS: serviceId: "cas-uk" ldapConnection: hostname: "ldap.cuni.cz" base_dn: "ou=people,dc=cuni,dc=cz" port: 389 security: SSL bindName: "cunipersonalid" fields: email: "mail" firstName: "givenName" lastName: "sn" emails: apiUrl: https://recodex.projekty.ms.mff.cuni.cz:4000 footerUrl: https://recodex.projekty.ms.mff.cuni.cz siteName: "ReCodEx" githubUrl: https://github.com/ReCodEx from: "ReCodEx " failures: emails: to: "Admin Name " from: %emails.from% subjectPrefix: "ReCodEx Failure Report - " forgottenPassword: redirectUrl: "https://recodex.projekty.ms.mff.cuni.cz/ forgotten-password/change" tokenExpiration: 600 # 10 minues emails: from: %emails.from% subjectPrefix: "ReCodEx Forgotten Password Request - " mail: smtp: true host: "smtp.ps.stdin.cz" port: 587 username: "user" password: "pass" secure: "tls" context: ssl: verify_peer: false verify_peer_name: false allow_self_signed: true doctrine: user: "user" password: "pass" host: localhost dbname: "recodex-api" ``` ## Web application The location for configuration of the web application is in root of the project source tree. The name have to be `.env` and can be created by copying template `.env-example` file. ### Configurable items Description of configurable options. Bold are required values, optional ones are in italics. - **NODE_ENV** -- mode of the server - **API_BASE** -- base address of API server, including port and API version - **PORT** -- port where the app is listening - _WEBPACK_DEV_SERVER_PORT_ -- port for webpack dev server when running in development mode. Default one is 8081, this option might be useful when this port is necessary for some other service. ### Example configuration file ```{.ini} NODE_ENV=production API_BASE=https://recodex.projekty.ms.mff.cuni.cz:4000/v1 PORT=8080 ```