Note that `rpm` and `deb` packages are build in the same time. You may need to have `rpmbuild` command (usually as `rpmbuild` or `rpm` package) or edit CPACK_GENERATOR variable _CMakeLists.txt_ file in root of source code tree.
- Install generated package through your package manager (`yum`, `dnf`, `dpkg`).
_Note:_ If you do not want to generate binary packages, you can just install the project with `make install` (as root). But installation through your distribution's package manager is preferred way to keep your system clean and manageable in long term horizon.
## Configuration and usage
Following text describes how to set up and run broker program. It is supposed to have required binaries installed. Also, using systemd is recommended for best user experience, but it is not required. Almost all modern Linux distributions are using systemd now.
Installation of broker program does following step to your computer:
- create _systemd_ unit file `/etc/systemd/system/recodex-broker.service`
- put main binary to `/usr/bin/recodex-broker`
- create system user and group `recodex` with nologin shell (if not existing)
- create log directory `/var/log/recodex`
- set ownership of config (`/etc/recodex`) and log (`/var/log/recodex`) directories to `recodex` user and group
### Default broker configuration
#### Configuration items
@ -166,25 +130,3 @@ logger:
rotations: 3 # number of rotations kept
```
### Running broker
Running broker is very similar to the worker setup. There is also provided systemd unit file for convenient usage. There is only one broker per whole ReCodEx solution, so there is no need for systemd templates.
- Running broker can be done by following command:
```
# systemctl start recodex-broker.service
```
Check with
```
# systemctl status recodex-broker.service
```
if the broker is running. You should see "active (running)" message.
- Broker can be stopped or restarted accordigly using `systemctl stop` and `systemctl restart` commands.
- If you want to run broker after system startup, run:
```
# systemctl enable recodex-broker.service
```
For further information about using systemd please refer to systemd documentation.
To install and use the fileserver, it is necessary to have Python3 with `pip` package manager installed. It is needed to install the dependencies. From clonned repository run the following command:
```
$ pip install -r requirements.txt
```
That is it. Fileserver does not need any special installation. It is possible to build and install _rpm_ package or install it without packaging the same way as monitor, but it is only optional. The installation would provide you with script `recodex-fileserver` in you `PATH`. No systemd unit files are provided, because of the configuration and usage of fileserver component is much different to our other Python parts.
## Configuration and usage
There are several ways of running the ReCodEx fileserver. We will cover two
typical use cases.
### Running in development mode
For simple development usage, it is possible to run the fileserver in the command
- **port** -- port where the fileserver should listen
- **working_directory** -- directory where the files should be stored
- **htpasswd** -- path to user file for HTTP Basic Authentication
- **user** -- user under which the server should be run
### Running using uWSGI
Another option is to run fileserver as a standalone app via uWSGI service. Setup is also quite simple, configuration file can be also generated by `mkconfig.py` script.
1. (Optional) Create a user for running the fileserver
2. Make sure that your user can access your clone of the repository
Installation of whole ReCodEx solution is a very complex process. It is recommended to have good unix skills with basic knowledge of project architecture.
Installation of whole ReCodEx solution is a very complex process. It is
recommended to have good unix skills with basic knowledge of project
architecture.
There are a lot of different GNU/Linux distributions with different package management, naming convention and version release policies. So it is impossible to cover all of the possible variants. We picked one distribution, which is fully supported by automatic installation script, for others there are brief information about installation in every project component's own chapter.
There are a lot of different GNU/Linux distributions with different package
management, naming convention and version release policies. So it is impossible
to cover all of the possible variants. We picked one distribution, which is
fully supported by automatic installation script, for others there are brief
information about installation in every project component's own chapter.
Distribution of our choice is CentOS, currently in version 7. It is a well known server distribution, derived from enterprise distrubution from Red Hat, so it is very stable and widely used system with long term support. There are [EPEL](https://fedoraproject.org/wiki/EPEL) additional repositories from Fedora project, which adds newer versions of some packages into CentOS, which allows us to use current environment. Also, _rpm_ packages are much easier to build (for example from Python sources) and maintain.
Distribution of our choice is CentOS, currently in version 7. It is a well known
server distribution, derived from enterprise distrubution from Red Hat, so it is
very stable and widely used system with long term support. There are
[EPEL](https://fedoraproject.org/wiki/EPEL) additional repositories from Fedora
project, which adds newer versions of some packages into CentOS, which allows us
to use current environment. Also, _rpm_ packages are much easier to build (for
example from Python sources) and maintain.
The big rival of CentOS in server distributions field is Debian. We are running one instance of ReCodEx on Debian too. You need to use _testing_ repositories to use some decent package versions. It is easy to mess your system easily, so create file `/etc/apt/apt.conf` with content of `APT::Default-Release "stable";`. After you add testing repos to `/etc/apt/sources.list`, you can install packages from there like `$ sudo apt-get -t testing install gcc`.
The big rival of CentOS in server distributions field is Debian. We are running
one instance of ReCodEx on Debian too. You need to use _testing_ repositories to
use some decent package versions. It is easy to mess your system easily, so
create file `/etc/apt/apt.conf` with content of `APT::Default-Release
"stable";`. After you add testing repos to `/etc/apt/sources.list`, you can
install packages from there like `$ sudo apt-get -t testing install gcc`.
Some components are also capable of running in Windows environment. However setting up Windows OS is a little bit of pain and it is not supposed to run ReCodEx in this way. Only worker component may be needed to run on Windows, so we are providing clickable installer including dependencies. Just for info, all components should be able to run on Windows, only broker was not tested and may require small tweaks to properly work.
Some components are also capable of running in Windows environment. However
setting up Windows OS is a little bit of pain and it is not supposed to run
ReCodEx in this way. Only worker component may be needed to run on Windows, so
we are providing clickable installer including dependencies. Just for info, all
components should be able to run on Windows, only broker was not tested and may
require small tweaks to properly work.
## Ansible installer
For automatic installation is used set of Ansible scripts. Ansible is one of the best known and used tools for automatic server management. It is required only to have SSH access to the server and ansible installed on the client machine. For further reading is supposed basic Ansible knowledge. For more info check their [documentation](http://docs.ansible.com/ansible/intro.html).
For automatic installation is used set of Ansible scripts. Ansible is one of the
best known and used tools for automatic server management. It is required only
to have SSH access to the server and ansible installed on the client machine.
For further reading is supposed basic Ansible knowledge. For more info check
their [documentation](http://docs.ansible.com/ansible/intro.html).
All Ansible scripts are located in _utils_ repository, _installation_ [directory](https://github.com/ReCodEx/utils/tree/master/installation). Ansible files are pretty self-describing, they can be also use as template for installation to different systems. Before installation itself it is required to edit two files -- set addresses of hosts and values of some variables.
All Ansible scripts are located in _utils_ repository, _installation_
files are pretty self-describing, they can be also use as template for
installation to different systems. Before installation itself it is required to
edit two files -- set addresses of hosts and values of some variables.
### Hosts configuration
First, it is needed to set ip addresses of your computers. Common practise is to have multiple files with definitions, one for development, another for production for example. Example configuration is in _development_ file. Each component of ReCodEx project can be installed on different server. Hosts can be specified as hostnames or ip addresses, optionally with port of SSH after colon.
First, it is needed to set IP addresses of your computers. Common practise is to
have multiple files with definitions, one for development, another for
production for example. Example configuration is in _development_ file. Each
component of ReCodEx project can be installed on different server. Hosts can be
specified as hostnames or IP addresses, optionally with port of SSH after colon.
Shorten example of hosts config:
@ -36,69 +70,752 @@ broker
### Variables
Configurable variables are saved in _group_vars/all.yml_ file. Syntax is basic key-value pair per line, separated by colon. Values with brief description:
Configurable variables are saved in _group_vars/all.yml_ file. Syntax is basic
key-value pair per line, separated by colon. Values with brief description:
- _source_dir_ -- Directory, where to store all sources from GitHub. Defaults `/opt/recodex`.
- _mysql_root_password_ -- Password of root user of MySQL database. Will be set after installation and saved to `/root/.my.cnf` file.
- _source_dir_ -- Directory, where to store all sources from GitHub. Defaults
`/opt/recodex`.
- _mysql_root_password_ -- Password of root user of MySQL database. Will be set
after installation and saved to `/root/.my.cnf` file.
- _mysql_recodex_username_ -- MySQL username for ReCodEx API access.
- _mysql_recodex_password_ -- Password for the user above.
- _admin_email_ -- Email of administrator. Used when configuring Apache webserver.
- _recodex_hostname_ -- Hostname where the API and web app will be accessible. For example "recodex.projekty.ms.mff.cuni.cz".
- _webapp_node_addr_ -- IP address of NodeJS server running web app. Defaults to "127.0.0.1" and should not be changed.
- _admin_email_ -- Email of administrator. Used when configuring Apache
webserver.
- _recodex_hostname_ -- Hostname where the API and web app will be accessible.
For example "recodex.projekty.ms.mff.cuni.cz".
- _webapp_node_addr_ -- IP address of NodeJS server running web app. Defaults to
"127.0.0.1" and should not be changed.
- _webapp_node_port_ -- Port to above.
- _webapp_public_addr_ -- Public address, where web server for web app will listen. Defaults to "*".
- _webapp_public_addr_ -- Public address, where web server for web app will
listen. Defaults to "*".
- _webapp_public_port_ -- Port to above.
- _webapp_firewall_ -- Open port for web app in firewall, values "yes" or "no".
- _webapi_public_endpoint_ -- Public URL when the API will be running, for example "https://recodex.projekty.ms.mff.cuni.cz:4000/v1".
- _webapi_public_addr_ -- Public address, where web server for API will listen. Defaults to "*".
- _webapi_public_endpoint_ -- Public URL when the API will be running, for
example "https://recodex.projekty.ms.mff.cuni.cz:4000/v1".
- _webapi_public_addr_ -- Public address, where web server for API will listen.
Defaults to "*".
- _webapi_public_port_ -- Port to above.
- _webapi_firewall_ -- Open port for API in firewall, values "yes" or "no".
- _database_firewall_ -- Open port for database in firewall, values "yes" or "no".
- _broker_to_webapi_addr_ -- Address, where API can reach broker. Private one is recommended.
- _database_firewall_ -- Open port for database in firewall, values "yes" or
"no".
- _broker_to_webapi_addr_ -- Address, where API can reach broker. Private one is
recommended.
- _broker_to_webapi_port_ -- Port to above.
- _broker_firewall_api_ -- Open above port in firewall, "yes" or "no".
- _broker_to_workers_addr_ -- Address, where workers can reach broker. Private one is recommended.
- _broker_to_workers_addr_ -- Address, where workers can reach broker. Private
one is recommended.
- _broker_to_workers_port_ -- Port to above.
- _broker_firewall_workers_ -- Open above port in firewall, "yes" or "no".
- _broker_notifier_address_ -- URL (on API), where broker will send notifications, for example "https://recodex.projekty.ms.mff.cuni.cz/v1/broker-reports".
- _broker_notifier_port_ -- Port to above, should be the same as for API itself (_webapi_public_port_)
- _broker_notifier_address_ -- URL (on API), where broker will send
- _broker_notifier_port_ -- Port to above, should be the same as for API itself
(_webapi_public_port_)
- _broker_notifier_username_ -- Username for HTTP Authentication for reports
- _broker_notifier_password_ -- Password for HTTP Authentication for reporst
- _monitor_websocket_addr_ -- Address, where websocket connection from monitor will be available
- _monitor_websocket_addr_ -- Address, where websocket connection from monitor
will be available
- _monitor_websocket_port_ -- Port to above.
- _monitor_firewall_websocket_ -- Open above port in firewall, "yes" or "no".
- _monitor_zeromq_addr_ -- Address, where monitor will be available on ZeroMQ socket for broker to receive reports.
- _monitor_zeromq_addr_ -- Address, where monitor will be available on ZeroMQ
socket for broker to receive reports.
- _monitor_zeromq_port_ -- Port to above.
- _monitor_firewall_zeromq_ -- Open above port in firewall, "yes" or "no".
- _fileserver_addr_ -- Address, where fileserver will serve files.
- _fileserver_port_ -- Port to above.
- _fileserver_firewall_ -- Open above port in firewall, "yes" or "no".
- _fileserver_username_ -- Username for HTTP Authentication for access the fileserver.
- _fileserver_password_ -- Password for HTTP Authentication for access the fileserver.
- _worker_cache_dir_ -- File cache storage for workers. Defaults to "/tmp/recodex/cache".
- _fileserver_username_ -- Username for HTTP Authentication for access the
fileserver.
- _fileserver_password_ -- Password for HTTP Authentication for access the
fileserver.
- _worker_cache_dir_ -- File cache storage for workers. Defaults to
"/tmp/recodex/cache".
- _worker_cache_age_ -- How long hold fetched files in worker cache, in seconds.
- _isolate_version_ -- Git tag of Isolate version worker depends on.
### Installation itself
With your computers installed with CentOS and configuration modified it is time to run the installation.
With your computers installed with CentOS and configuration modified it is time
to run the installation.
```
$ ansible-playbook -i development recodex.yml
```
This command installs all components of ReCodEx onto machines listed in _development_ file. It is possible to install only specified parts of project, just use component's YAML file instead of _recodex.yml_.
This command installs all components of ReCodEx onto machines listed in
_development_ file. It is possible to install only specified parts of project,
just use component's YAML file instead of _recodex.yml_.
Ansible expects to have password-less access to the remote machines. If you have not such setup, use options `--ask-pass` and `--ask-become-pass`.
Ansible expects to have password-less access to the remote machines. If you have
not such setup, use options `--ask-pass` and `--ask-become-pass`.
## Manual installation
### Worker
#### Dependencies
Worker specific requirements are written in this section. It covers only basic
requirements, additional runtimes or tools may be needed depending on type of
use. The package names are for CentOS if not specified otherwise.
- ZeroMQ in version at least 4.0, packages `zeromq` and `zeromq-devel`
(`libzmq3-dev` on Debian)
- YAML-CPP library, `yaml-cpp` and `yaml-cpp-devel` (`libyaml-cpp0.5v5` and
`libyaml-cpp-dev` on Debian)
- libcurl library `libcurl-devel` (`libcurl4-gnutls-dev` on Debian)
- libarchive library as optional dependency. Installing will speed up build
process, otherwise libarchive is built from source during installation.
Package name is `libarchive` and `libarchive-devel` (`libarchive-dev` on
Debian)
**Install Isolate from source** (only for Linux installations)
First, we need to compile sandbox Isolate from source and install it. Current
worker is tested against version 1.3, so this version needs to be checked out.
Assume that we keep source code in `/opt/src` dir. For building man page you
need to have package `asciidoc` installed.
```
$ cd /opt/src
$ git clone https://github.com/ioi/isolate.git
$ cd isolate
$ git checkout v1.3
$ make
# make install && make install-doc
```
For proper work Isolate depends on several advanced features of the Linux
kernel. Make sure that your kernel is compiled with `CONFIG_PID_NS`,
With API comes some initial values, for example default user roles with proper
permissions. To fill your database with these values there is another command
line command:
```
$ php www/index.php db:fill
```
Check the outputs of both commands for errors. If there are any, try to clean
temporary API cache in `temp/cache/` directory and repeat the action.
#### Webserver configuration
The simplest way to get started is to start the built-in PHP server in the root
directory of your project:
```
$ php -S localhost:4000 -t www
```
Then visit `http://localhost:4000` in your browser to see the welcome page of API project.
For Apache or Nginx, setup a virtual host to point to the `www/` directory of
the project and you should be ready to go. It is **critical** that whole `app/`,
`log/` and `temp/` directories are not accessible directly via a web browser
(see [security warning](https://nette.org/security-warning)). Also it is
**highly recommended** to set up a HTTPS certificate for public access to the
API.
#### Troubleshooting
In case of any issues first remove the Nette cache directory `temp/cache/` and
try again. This solves most of the errors. If it does not help, examine API logs
from `log/` directory of the API source or logs of your webserver.
### Web application
Web application requires [NodeJS](https://nodejs.org/en/) server as its runtime
environment. This runtime is needed for executing JavaScript code on server and
sending the pre-render parts of pages to clients, so the final rendering in
browsers is a lot quicker and the page is accessible to search engines for
indexing.
But some functionality is better in other full fledged web servers like *Apache*
or *Nginx*, so the common practice is to use a tandem of both. *NodeJS* takes
care of basic functionality of the app while the other server (Apache) is set as
reverse proxy and providing additional functionality like SSL encryption, load
balancing or caching of static files. The recommended setup contains both NodeJS
and one of Apache and Nginx web servers for the reasons discussed above.
Stable versions of 4th and 6th series of NodeJS server are sufficient, using at
least 6th series is highly recommended. Please check the most recent version of
the packages in your distribution's repositories, there are often outdated ones.
However, there are some third party repositories for all main Linux
distributions.
The app depends on several libraries and components, all of them are listed in
`package.json` file in source repository. For managing dependencies is used node
package manager (`npm`), which can come with NodeJS installation otherwise can
be installed separately. To fetch and install all dependencies run:
```
$ npm install
```
For easy production usage there is an additional package for managing NodeJS
processes, `pm2`. This tool can run your application as a daemon, monitor
occupied resources, gather logs and provide simple console interface for
managing app's state. To install it globally into your system run:
```
# npm install pm2 -g
```
#### Usage
The application can be run in two modes, development and production. Development
mode uses only client rendering and tracks code changes with rebuilds of the
application in real time. In production mode the compilation (transpile to _ES5_
standard using *Babel* and bundle into single file using *webpack*) has to be
done separately prior to running. The scripts for compilation are provided as
additional `npm` commands.
- Development mode can be use for local testing of the app. This mode uses
webpack dev server, so all code runs on a client, there is no server side
rendering available. Starting is simple command, default address is
http://localhost:8080.
```
$ npm run dev
```
- Production mode is mostly used on the servers. It provides all features such
as server side rendering. This can be run via:
```
$ npm run build
$ npm start
```
Both modes can be configured to use different ports or set base address of used
API server. This can be configured in `.env` file in root of the repository.
There is `.env-sample` file which can be just copied and altered.
The production mode can be run also as a demon controled by `pm2` tool. First
the web application has to be built and then the server javascript file can run
as a daemon.
```
$ npm run build
$ pm2 start bin/server.js
```
The `pm2` tool has several options, most notably _status_, _stop_, _restart_ and
_logs_. Further description is available on project
[website](http://pm2.keymetrics.io).
## Security
One of the most important aspects of ReCodEx instance is security. It is crucial to keep gathered data safe and not to allow unauthorized users modify restricted pieces of information. Here is a small list of recommendations to keep running ReCodEx instance safe.
One of the most important aspects of ReCodEx instance is security. It is crucial
to keep gathered data safe and not to allow unauthorized users modify restricted
pieces of information. Here is a small list of recommendations to keep running
ReCodEx instance safe.
- Secure MySQL installation. The installation script does not do any security actions, so please run at least `mysql_secure_installation` script on database computer.
- Get HTTPS certificate and set it in Apache for web application and API. Monitor should be proxied through the web server too with valid certificate. You can get free DV certificate from [Let's Encrypt](https://letsencrypt.org/). Do not forget to set up automatic renewing!
- Hide broker, workers and fileserver behind firewall, private subnet or IPsec tunnel. They are not required to be reached from public internet, so it is better keep them isolated.
- Keep your server updated and well configured. For automatic installation of security updates on CentOS system refer to `yum-cron` package. Configure SSH and Apache to use only strong ciphers, some recommendations can be found [here](https://bettercrypto.org/static/applied-crypto-hardening.pdf).
- Do not put actually used credentials on web, for example do not commit your passwords (in Ansible variables file) on GitHub.
- Secure MySQL installation. The installation script does not do any security
actions, so please run at least `mysql_secure_installation` script on database
computer.
- Get HTTPS certificate and set it in Apache for web application and API.
Monitor should be proxied through the web server too with valid certificate.
You can get free DV certificate from [Let's
Encrypt](https://letsencrypt.org/). Do not forget to set up automatic
renewing!
- Hide broker, workers and fileserver behind firewall, private subnet or IPsec
tunnel. They are not required to be reached from public internet, so it is
better keep them isolated.
- Keep your server updated and well configured. For automatic installation of
security updates on CentOS system refer to `yum-cron` package. Configure SSH
and Apache to use only strong ciphers, some recommendations can be found
@ -11,18 +11,6 @@ Monitor is needed one per broker, that is one per separate ReCodEx instance. Als
Monitor is written in Python, tested versions are 3.4 and 3.5. This language was chosen because it is already in project requirements (fileserver) and there are great libraries for ZeroMQ, WebSockets and asynchronous operations. This library saves system resources and provides us great amount of processed messages. Also, coding in Python was pretty simple and saves us time for improving the other parts of ReCodEx.
For monitor functionality there are some required packages. All of them are listed in _requirements.txt_ file in the repository and can be installed by `pip` package manager as
```
$ pip install -r requirements.txt
```
**Description of dependencies:**
- zmq -- binding to ZeroMQ framework
- websockets -- framework for communication over WebSockets
- asyncio -- library for fast asynchronous operations
- pyyaml -- parsing YAML configuration files
- argparse -- parsing command line arguments
### Message flow
@ -41,33 +29,6 @@ There can be multiple receivers to one channel id. Each one has separate _asynci
Messages from client's queue are sent through corresponding WebSocket connection via main event loop as soon as possible. This approach with separate queue per connection is easy to implement and guarantees reliability and order of message delivery.
## Installation
Installation will provide you following files:
- `/usr/bin/recodex-monitor` -- simple startup script located in PATH
- code files will be installed in location depending on your system settings, mostly into `/usr/lib/python3.5/site-packages/monitor/` or similar
Systemd script runs monitor binary as specific _recodex_ user, so in `postinst` script user and group of this name are created. Also, ownership of configuration file will be granted to that user.
- RPM distributions can make and install binary package. This can be done like this:
@ -88,14 +88,6 @@ both our internal login service and CAS.
An advantage of this approach is being able control the authentication process
completely instead of just receiving session data through a global variable.
## Installation
The web API requires a PHP runtime version at least 7. Which one depends on actual configuration, there is a choice between _mod_php_ inside Apache, _php-fpm_ with Apache or Nginx proxy or running it as standalone uWSGI script. It is common that there are some PHP extensions, that have to be installed on the system. Namely ZeroMQ binding (`php-zmq` package or similar), MySQL module (`php-mysqlnd` package) and ldap extension module for CAS authentication (`php-ldap` package). Make sure that the extensions are loaded in your `php.ini` file (`/etc/php.ini` or files in `/etc/php.d/`).
The API depends on some other projects and libraries. For managing them [Composer](https://getcomposer.org/) is used. It can be installed from system repositories or downloaded from the website, where detailed instructions are as well. Composer reads `composer.json` file in the project root and installs dependencies to the `vendor/` subdirectory. To do that, run:
```
$ composer install
```
## Configuration and usage
@ -223,36 +215,3 @@ doctrine:
dbname: "recodex-api"
```
### Database preparation
When the API is installed and configured (_doctrine_ section is sufficient here) the database schema can be generated. There is a prepared command to do that from command line:
With API comes some initial values, for example default user roles with proper permissions. To fill your database with these values there is another command line command:
```
$ php www/index.php db:fill
```
Check the outputs of both commands for errors. If there are any, try to clean temporary API cache in `temp/cache/` directory and repeat the action.
### Webserver configuration
The simplest way to get started is to start the built-in PHP server in the root directory of your project:
```
$ php -S localhost:4000 -t www
```
Then visit `http://localhost:4000` in your browser to see the welcome page of API project.
For Apache or Nginx, setup a virtual host to point to the `www/` directory of the project and you should be ready to go. It is **critical** that whole `app/`, `log/` and `temp/` directories are not accessible directly via a web browser (see [security warning](https://nette.org/security-warning)). Also it is **highly recommended** to set up a HTTPS certificate for public access to the API.
### Troubleshooting
In case of any issues first remove the Nette cache directory `temp/cache/` and try again. This solves most of the errors. If it does not help, examine API logs from `log/` directory of the API source or logs of your webserver.
This will create *JSON* files with the exported strings for the *'en'* and *'cs'* locale. If you want to export strings for more languages, you must edit the `/manageTranslations.js` script. The exported strings are placed in the `/src/locales` directory.
## Installation
Web application requires [NodeJS](https://nodejs.org/en/) server as its runtime environment. This runtime is needed for executing JavaScript code on server and sending the pre-render parts of pages to clients, so the final rendering in browsers is a lot quicker and the page is accessible to search engines for indexing.
But some functionality is better in other full fledged web servers like *Apache* or *Nginx*, so the common practice is to use a tandem of both. *NodeJS* takes care of basic functionality of the app while the other server (Apache) is set as reverse proxy and providing additional functionality like SSL encryption, load balancing or caching of static files. The recommended setup contains both NodeJS and one of Apache and Nginx web servers for the reasons discussed above.
Stable versions of 4th and 6th series of NodeJS server are sufficient, using at least 6th series is highly recommended. Please check the most recent version of the packages in your distribution's repositories, there are often outdated ones. However, there are some third party repositories for all main Linux distributions.
The app depends on several libraries and components, all of them are listed in `package.json` file in source repository. For managing dependencies is used node package manager (`npm`), which can come with NodeJS installation otherwise can be installed separately. To fetch and install all dependencies run:
```
$ npm install
```
For easy production usage there is an additional package for managing NodeJS processes, `pm2`. This tool can run your application as a daemon, monitor occupied resources, gather logs and provide simple console interface for managing app's state. To install it globally into your system run:
```
# npm install pm2 -g
```
## Configuration and usage
The application can be run in two modes, development and production. Development mode uses only client rendering and tracks code changes with rebuilds of the application in real time. In production mode the compilation (transpile to _ES5_ standard using *Babel* and bundle into single file using *webpack*) has to be done separately prior to running. The scripts for compilation are provided as additional `npm` commands.
- Development mode can be use for local testing of the app. This mode uses webpack dev server, so all code runs on a client, there is no server side rendering available. Starting is simple command, default address is http://localhost:8080.
```
$ npm run dev
```
- Production mode is mostly used on the servers. It provides all features such as server side rendering. This can be run via:
```
$ npm run build
$ npm start
```
Both modes can be configured to use different ports or set base address of used API server. This can be configured in `.env` file in root of the repository. There is `.env-sample` file which can be just copied and altered.
The production mode can be run also as a demon controled by `pm2` tool. First the web application has to be built and then the server javascript file can run as a daemon.
```
$ npm run build
$ pm2 start bin/server.js
```
The `pm2` tool has several options, most notably _status_, _stop_, _restart_ and _logs_. Further description is available on project [website](http://pm2.keymetrics.io).
#### Configurable items
## Configurable items
Description of configurable options. Bold are required values, optional ones are in italics.
@ -82,92 +82,6 @@ Isolate is executed in separate Linux process created by `fork` and `exec` syste
Sandbox in general has to be command line application taking parameters with arguments, standard input or file. Outputs should be written to file or standard output. There are no other requirements, worker design is very versatile and can be adapted to different needs.
## Installation
### Dependencies
Worker specific requirements are written in this section. It covers only basic requirements, additional runtimes or tools may be needed depending on type of use. The package names are for CentOS if not specified otherwise.
- ZeroMQ in version at least 4.0, packages `zeromq` and `zeromq-devel` (`libzmq3-dev` on Debian)
- YAML-CPP library, `yaml-cpp` and `yaml-cpp-devel` (`libyaml-cpp0.5v5` and `libyaml-cpp-dev` on Debian)
- libcurl library `libcurl-devel` (`libcurl4-gnutls-dev` on Debian)
- libarchive library as optional dependency. Installing will speed up build process, otherwise libarchive is built from source during installation. Package name is `libarchive` and `libarchive-devel` (`libarchive-dev` on Debian)
**Install Isolate from source**
First, we need to compile sandbox Isolate from source and install it. Current worker is tested against version 1.3, so this version needs to be checked out. Assume that we keep source code in `/opt/src` dir. For building man page you need to have package `asciidoc` installed.
```
$ cd /opt/src
$ git clone https://github.com/ioi/isolate.git
$ cd isolate
$ git checkout v1.3
$ make
# make install && make install-doc
```
For proper work Isolate depends on several advanced features of the Linux kernel. Make sure that your kernel is compiled with `CONFIG_PID_NS`, `CONFIG_IPC_NS`, `CONFIG_NET_NS`, `CONFIG_CPUSETS`, `CONFIG_CGROUP_CPUACCT`, `CONFIG_MEMCG`. If your machine has swap enabled, also check `CONFIG_MEMCG_SWAP`. With which flags was your kernel compiled with can be found in `/boot` directory, file `config-` and version of your kernel. Red Hat based distributions should have these enabled by default, for Debian you you may want to add the parameters `cgroup_enable=memory swapaccount=1` to the kernel command-line, which can be set by adding value `GRUB_CMDLINE_LINUX_DEFAULT` to `/etc/default/grub` file.
For better reproducibility of results, some kernel parameters can be tweaked:
- Disable address space randomization. Create file `/etc/sysctl.d/10-recodex.conf` with content `kernel.randomize_va_space=0`. Changes will take effect after restart or run `sysctl kernel.randomize_va_space=0` command.
- Disable dynamic CPU frequency scaling. This requires setting the cpufreq scaling governor to _performance_.
### Clone worker source code repository
```
$ git clone https://github.com/ReCodEx/worker.git
$ git submodule update --init
```
### Install worker on Linux
It is supposed that your current working directory is that one with clonned worker source codes.
- Prepare environment running `mkdir build && cd build`
Note that `rpm` and `deb` packages are build in the same time. You may need to have `rpmbuild` command (usually as `rpmbuild` or `rpm` package) or edit CPACK_GENERATOR variable in _CMakeLists.txt_ file in root of source code tree.
- Install generated package through your package manager (`yum`, `dnf`, `dpkg`).
The worker installation process is composed of following steps:
- create systemd unit file `/etc/systemd/system/recodex-worker@.service`
- put main binary to `/usr/bin/recodex-worker`
- put judges binaries to `/usr/bin/` directory
- create system user and group `recodex` with `/sbin/nologin` shell (if not already existing)
- create log directory `/var/log/recodex`
- set ownership of config (`/etc/recodex`) and log (`/var/log/recodex`) directories to `recodex` user and group
_Note:_ If you do not want to generate binary packages, you can just install the project with `make install` (as root). But installation through your distribution's package manager is preferred way to keep your system clean and manageable in long term horizon.
### Install worker on Windows
From beginning we are determined to support Windows operating system on which some of the workers may run (especially for projects in C# programming language). Support for Windows is quite hard and time consuming and there were several problems during the development. To ensure capability of compilation on Windows we set up CI for Windows named [Appveyor](http://www.appveyor.com/). However installation should be easy due to provided installation script.
There are only two additional dependencies needed, **Windows 7 and higher** and **Visual Studio 2015+**. Provided simple installation batch script should do all the work on Windows machine. Officially only VS2015 and 32-bit compilation is supported, because of hardcoded compile options in installation script. If different VS or different platform is needed, the script should be changed to appropriate values, which is simple and straightforward.
Mentioned script is placed in *install* directory alongside supportive scripts for UNIX systems and is named *win-build.cmd*. Provided script will do almost all the work connected with building and dependency resolving (using **NuGet** package manager and `msbuild` building system). Script should be run under 32-bit version of _Developer Command Prompt for VS2015_ and from *install* directory.
Building and installing of worker is then quite simple, script has command line parameters which can be used to specify what will be done:
- *-build* -- It is the default options if none specified. Builds worker and its tests, all is saved in *build* folder and subfolders.
- *-clean* -- Cleanup of downloaded NuGet packages and built application/libraries.
- *-test* -- Build worker and run tests on compiled test cases.
- *-package* -- Generation of clickable installation using cpack and [NSIS](http://nsis.sourceforge.net/) (has to be installed on machine to get this to work).
```
install> win-build.cmd # same as: win-build.cmd -build
install> win-build.cmd -clean
install> win-build.cmd -test
install> win-build.cmd -package
```
All build binaries and cmake temporary files can be found in *build* folder,
classically there will be subfolder *Release* which will contain compiled
application with all needed dlls. Once if clickable installation binary is
created, it can be found in *build* folder under name
*recodex-worker-VERSION-win32.exe*. Sample screenshot can be found on following picture.
Following text describes how to set up and run **worker** program. It is supposed to have required binaries installed. Also, using systemd is recommended for best user experience, but it is not required. Almost all modern Linux distributions are using systemd nowadays.
@ -247,44 +161,6 @@ limits:
mode: RW,NOEXEC
```
### Running the worker
A systemd unit file is distributed with the worker to simplify its launch. It
integrates worker nicely into your Linux system and allows you to run it
automatically on system startup. It is possible to have more than one worker on
every server, so the provided unit file is templated. Each instance of the
worker unit has a unique string identifier, which is used for managing that
instance through systemd. By default, only one worker instance is ready to use
after installation and its ID is "1".
- Starting worker with id "1" can be done this way:
```
# systemctl start recodex-worker@1.service
```
Check with
```
# systemctl status recodex-worker@1.service
```
if the worker is running. You should see "active (running)" message.
- Worker can be stopped or restarted accordigly using `systemctl stop` and `systemctl restart` commands.
- If you want to run worker after system startup, run:
```
# systemctl enable recodex-worker@1.service
```
For further information about using systemd please refer to systemd documentation.
### Adding new worker
To add a new worker you need to do a few steps:
- Make up an unique string ID.
- Copy default configuration file `/etc/recodex/worker/config-1.yml` to the same directory and name it `config-<your_unique_ID>.yml`
- Edit that config file to fit your needs. Note that you must at least change _worker-id_ and _logger file_ values to be unique.
@ -335,38 +211,6 @@ There is a bit of catch with cleaner service, to work properly, server filesyste
Another possibility seems to be to update last modified timestamp when accessing the file. This timestamp is used in most major filesystems, so there are less issues with compatibility than last access timestamp. The modified timestamp then must be updated by workers at each access, for example using `touch` command or similar. Final decision on better of these ways will be made after practical experience of running production system.
### Installation
To install and use the cleaner, it is necessary to have Python3 with package manager `pip` installed.
- Dependencies of cleaner has to be installed:
```
$ pip install -r requirements.txt
```
- RPM distributions can make and install binary package. This can be done like this:
where path specified with `--install-scripts` can be changed
- copy configuration file alongside with installed executable using
```
> copy install\config.yml \
"C:\Program Files\ReCodEx\cleaner\config.yml"
```
### Configuration and usage
#### Configuration items
@ -379,25 +223,3 @@ cache-dir: "/tmp"
file-age: "3600" # in seconds
```
#### Usage
As stated before cleaner should be cronned, on linux systems this can be done by built in `cron` service or if there is `systemd` present cleaner itself provides `*.timer` file which can be used for cronning from `systemd`. On Windows systems internal scheduler should be used.
- Running cleaner from command line is fairly simple:
```
$ recodex-cleaner -c /etc/recodex/cleaner
```
- Enable cleaner service using systemd:
```
$ systemctl start recodex-cleaner.timer
```
- Add cleaner to linux cron service using following configuration line:
Petr Stefan
8 years ago