From e3c13ad35e9eadf905ba43a862a7fb0f6a8c40fd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Martin=20Kruli=C5=A1?= Date: Thu, 27 Aug 2020 16:13:53 +0200 Subject: [PATCH] Updated Installation (markdown) --- Installation.md | 291 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 284 insertions(+), 7 deletions(-) diff --git a/Installation.md b/Installation.md index e2f6571..ac32a01 100644 --- a/Installation.md +++ b/Installation.md @@ -306,21 +306,298 @@ The important commands are once a week (e.g., during the night between Saturday and Monday). The frequency of cleanup commands depend on your system utilization. In -intensively utilized instances, it might be prudent to call cleanups at least -once a week. In case of mostly idle instances, cleanup once per month or even -once per year may be sufficient. +intensively used instances, it might be prudent to call cleanups as often as +once a week. In case of mostly idle instances, a cleanup per month or even +per year may be sufficient. One option is to create a script (or multiple scripts) and schedule their execution in crontab. Do not forget to run these commands as `recodex` user. For example, adding the following line in `/etc/crontab` will execute your -cleanup script every day at 3 AM under `recodex` user. +cleanup script every day at 3 AM. ``` 0 3 * * * recodex /path/to/your/cleanup/script.sh ``` ---- -## Set-up Workers and Environments -To be documented yet... +## Setup Workers and Environments + +### Worker configuration + +Worker is ready to be executed in multiple instances. Each instance has config +file `/etc/recodex/worker/config-%i.yml`, where %i is the numeric ID of the worker +(first one has ID = 1). Make sure that you have a config file for each worker +you want to start; however, it might be good idea to configure one worker (make +sure it is running properly) and then use the first config as template for others. +If you are managing many workers, some macro-preprocessing tool may be useful +to manage their configurations. Each instance needs to be enabled and started +(after config file is ready) as (replace `1` with other IDs for other workers): + +``` +# systemctl enable recodex-worker@1 +# systemctl start recodex-worker@1 +# systemctl status recodex-worker@1 +``` + +Before starting the worker service, edit the config file first. +The `worker-id` (and optionally) `worker-description` distinguish individual +workers in case you run multiple workers on the same machine. It is highly +recommended that these IDs match the IDs of systemd services (which are also +embedded in config file names). + +The worker needs broker and file server to operate. Update `broker-uri` so it +matches your broker location and port designated to workers. The `file-managers` +structure configures the file server access (`hostname` and HTTP auth +credentials). + +Create worker(s) working directory (e.g., `/var/recodex-worker-wd`) and cache +directory (e.g., `/var/recodex-worker-cache`) and set their paths to +`working-directory` and `file-cache` > `cache-dir` properties respectively. Both +directories must be owned (and writeable) by the user, under which the worker +runs (typically `recodex`). If you run multiple workers on one machine, these +directories may be shared (recommended). On the other hand, multiple workers +should not share log, so update `logger` > `file` so it holds unique file for +each worker. + +The `hwgroup` holds an ID of hardware group -- a group of workers with the same +capabilities (i.e., running on the same hardware with the same system +configuration). This is by default set to the only hwgroup present in +init fixtures used to initialize the database for the first time. Optionally, +you might want to restrict runtime environments the worker supports (`headers` +> `env`). The IDs correspond to IDs in the database and the default config file +holds all specified runtimes in the initial database fill. + +Based on supported runtime environments, it might be necessary to update the +configuration of sandbox (`limits` structure) -- namely the pre-set +environmental variables and mapped directories. For instance, when using `java` +runtime, a mapping for JDK directory needs to be added to `bound-directories` +(`src` refers to directory on the real file systems, `dst` specifies, where the +directory will be mounted in the sandbox), and `JAVA_HOME` variable holding the +path (`dst`) needs to be added to `environ-variable` list. Be careful when +editing `PATH` or `LD_LIBRARY_PATH` environmental variables as they may apply +to multiple runtimes. + + +### Isolate configuration + +The worker uses sandbox [isolate](https://github.com/ioi/isolate). The sandbox +is installed automatically, if you installed worker as RPM package (otherwise +you need to compile it manually). It has configuration in `/etc/isolate/default.cf`. +If you are running multiple workers (or other services) on the hardware, where +the testing will take place, it might be a good idea to configure sandbox CPU +affinity here, so that individual workers will not share CPU cores. For example: + +``` +box1.cpus = 0 +box2.cpus = 1 +box3.cpus = 2-7 +``` + +Will configure `box1` and `box2` (which correspond to workers with IDs `1` and `2`) +as single-core boxes bound to the first and the second CPU and the `box3` will be +a multicore box occupying the following six cores. + +For greater precision, it is better not to utilize the entire CPU (all CPUs). +Furthermore, we recommend turning off hyperthreading or multithreading feature. +The best option is when a sandbox occupies one socket alone, but that +might be a waste if you are using CPU dies with many cores (consider that when +planning the purchase of your hardware). + + +### Cleaner configuration + +TODO + + +### Runtime environments + +Some runtimes require access to `/etc` directory since their compilers or +interprets have their configuration there (e.g., freepascal or PHP). It is not +a good idea to map entire `/etc` in sandbox. One possibility is to prepare +separated config directory (e.g., `/usr/etc`) and place configs visible to sandbox +there (and add it to `bound-directories` list). + +Runtimes `bash` and `data-linux` should work out of the box. Other runtimes will +require additional installations and configurations: + +#### C and C++ (`c-gcc-linux` and `cxx-gcc-linux`) + +Simply install the GCC compiler. +``` +dnf -y install gcc gcc-c++ +``` + +#### C# and .NET Core + +TODO! Currently migrating from Mono to .NET Core. + +``` +https://raw.githubusercontent.com/ReCodEx/utils/master/runners/cs/Reader.cs +https://raw.githubusercontent.com/ReCodEx/utils/master/runners/cs/Wrapper.cs +``` + +#### Free Pascal (`freepascal-linux`) + +First you need to install Free Pascal compiler: +``` +# dnf -y install fpc +``` + +Copy `/etc/fpc.cfg` to `/usr/etc/fpc.cfg` (or your sandbox-only etc directory). +Set `PPC_CONFIG_PATH` environmental variable (`environ-variable` list) to +`/usr/etc` and make sure `/usr/etc` is mapped to sandbox as read-only (as +explained at the beginning of runtime environments section). + +#### Go + +Simply install the Go language package. +``` +# dnf -y install golang +``` + +#### Groovy + +TODO + +#### Haskell + +TODO + +#### Java + +Install latest OpenJDK including java compiler (packages are available in EPEL): +``` +# dnf -y install java-latest-openjdk java-latest-openjdk-devel +``` + +Download the runner source file +``` +https://raw.githubusercontent.com/ReCodEx/utils/master/runners/java/javarun.java +``` + +Compile the source file into `class` file (`javac ./javarun.java`). +Open your running ReCodEx instance in web browser and log in as administrator. +Open `Pipeline` page and do the following with both Java execution pipelines +(*Java execution & evaluation [outfile]* and *Java execution & evaluation [stdout]*): +Click on `Edit` button on the right, scroll down to *Supplementary files* box +and upload the compiled `javarun.class`. + + +#### Kotlin + +TODO + +#### Node.js (JavaScript) + +Install Node.js (if you are running the worker on the same machine as frontend, +you have already done this). + +``` +# dnf -y install curl +# curl -sL https://rpm.nodesource.com/setup_14.x | bash - +# dnf -y install nodejs +``` + +You may check the right version is installed by typing `node -v`, which should +report version of Node.js (starting with 14). + + +#### PHP + +Install PHP (if you are running the worker on the same machine as frontend, +you have already done this). + +``` +# dnf install dnf-utils http://rpms.remirepo.net/enterprise/remi-release-8.rpm +# dnf module enable php:remi-7.3 +# dnf install php-cli +``` + +Copy `php.ini` to `/usr/etc` and update it (**especially make sure that all needed +modules are loaded explicitly**). Also make sure the `/usr/etc` is mapped to +sandbox as suggested at the beginning of this section. + +Optionally, you might want to consider installing User Operations for Zend +(`uopz`) PECL package for PHP. This package could help you creating hooks or +mock existing functions, which is helpful when testing PHP assignments. If you +enable this module, it is important to re-allow the source code control over the +exit opcode as follows: + +Create a PHP script in a folder accessible by sandbox (e.g., `/usr/etc`): +``` +