You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
173 lines
9.4 KiB
Markdown
173 lines
9.4 KiB
Markdown
# Build and Deployment
|
|
This page contains steps how to set up a computer and how to successfully build and run **ReCodEx**. Most steps are listed in two variants, for Red Hat based distributions (like RHEL, Centos or Fedora) and Debian based distibutions. Before starting, make sure you have completed basic OS installation and set up, including users and logins, SSH, Git, firewall, etc.
|
|
|
|
## Table of Content
|
|
* [Common](#common)
|
|
* [Worker](#worker)
|
|
* [Windows Worker](#windows-worker)
|
|
* [WebApp](#webapp)
|
|
|
|
## Common
|
|
### Repositories
|
|
|
|
**Add _testing_ repositories to Debian OS**
|
|
- Create file `/etc/apt/apt.conf` with content
|
|
```
|
|
APT::Default-Release "stable";
|
|
```
|
|
- Add testing repos to `/etc/apt/sources.list`
|
|
```
|
|
deb http://ftp.cz.debian.org/debian/ testing main contrib non-free
|
|
deb-src http://ftp.cz.debian.org/debian/ testing contrib non-free
|
|
```
|
|
- Install packages with `-t testing` option. For example
|
|
```
|
|
$ sudo apt-get -t testing install gcc
|
|
```
|
|
**Add EPEL repository to RHEL**
|
|
- See [this](https://fedoraproject.org/wiki/EPEL) for instructions.
|
|
|
|
|
|
### Installation of dependencies
|
|
|
|
Install as new version of each package as possible, so mostly Debian packages are from testing repositories and RHEL packages are the newest ones from EPEL repositories.
|
|
|
|
**Basic development tools**
|
|
- `g++`
|
|
- `cmake`
|
|
- `make`
|
|
- `git`
|
|
|
|
**Install ZeroMQ** in version at least 4.0
|
|
- Debian package is `libzmq3-dev`.
|
|
- RedHat packages are `zeromq` and `zeromq-devel`.
|
|
|
|
**Install YAML-CPP library**
|
|
- Debian packages: `libyaml-cpp0.5v5` and `libyaml-cpp-dev`.
|
|
- RedHat packages are `yaml-cpp` and `yaml-cpp-devel`.
|
|
|
|
**Install libcurl library**
|
|
- Debian package is `libcurl4-gnutls-dev`.
|
|
- RedHat package is `libcurl-devel`.
|
|
|
|
### Worker specific dependencies
|
|
|
|
Worker requirements are written in this section. Some parts of this guide may be different for each type of worker, for example Compiler principles worker is not fully covered here.
|
|
|
|
**Install libarchive library** (optional)
|
|
|
|
Installing this package will only speed up build process, otherwise libarchive is built from source.
|
|
- Debian package is `libarchive-dev`.
|
|
- RedHat packages are `libarchive` and `libarchive-devel`. These are probably not available for RHEL.
|
|
|
|
**Install Isolate from source**
|
|
|
|
First, we need to compile sandbox Isolate from source and install. Assume that we keep source code in `/opt/src` dir. For building man page you need to have package `asciidoc` installed.
|
|
```
|
|
$ cd /opt/src
|
|
$ git clone https://github.com/ioi/isolate.git
|
|
$ cd isolate
|
|
$ make
|
|
# make install && make install-doc
|
|
```
|
|
For proper work Isolate depends on several advanced features of the Linux kernel. Make sure that your kernel is compiled with `CONFIG_PID_NS`, `CONFIG_IPC_NS`, `CONFIG_NET_NS`, `CONFIG_CPUSETS`, `CONFIG_CGROUP_CPUACCT`, `CONFIG_MEMCG`. If your machine has swap enabled, also check `CONFIG_MEMCG_SWAP`. Which flags was your kernel compiled with can be found in `/boot` directory, for example in `/boot/config-4.2.6-301.fc23.x86_64` file for kernel version 4.2.6-301. Red Hat distros should have these enabled by default, for Debian you you may want to add the parameters `cgroup_enable=memory swapaccount=1` to the kernel command-line, which can be set using `GRUB_CMDLINE_LINUX_DEFAULT` in `/etc/default/grub`.
|
|
|
|
For better reproducibility of results, some kernel parameters can be tweaked:
|
|
- Disable address space randomization. Create file `/etc/sysctl.d/10-recodex.conf` with content `kernel.randomize_va_space=0`. Changes will take effect after restart or run `sysctl kernel.randomize_va_space=0` command.
|
|
- Disable dynamic CPU frequency scaling. This requires setting the cpufreq scaling governor to _performance_.
|
|
_**TODO** - do we really need it and is it worth higher power consumption? [Red Hat setup](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Power_Management_Guide/cpufreq_governors.html), [Debian setup](https://wiki.debian.org/HowTo/CpuFrequencyScaling)_
|
|
|
|
|
|
### Clone ReCodEx source code repositories
|
|
- Worker:
|
|
```
|
|
$ git clone https://github.com/ReCodEx/worker.git
|
|
$ git submodule update --init
|
|
```
|
|
- Broker
|
|
```
|
|
$ git clone https://github.com/ReCodEx/broker.git
|
|
$ git submodule update --init
|
|
```
|
|
|
|
### Install **worker** and **broker** binaries
|
|
|
|
Installation steps are equal for both worker and broker. Following guide is for worker. It's supposed that your current working directory is that one with clonned worker source codes.
|
|
|
|
- Prepare environment running `mkdir build && cd build`
|
|
- Build sources by `cmake ..` following by `make -j#` where '#' symbol refers to number of your CPU threads.
|
|
- Build binary package by `make package` (may require root permissions).
|
|
Note that `rpm` and `deb` packages are build in the same time. You may need to have `rpmbuild` command (usually as `rpmbuild` or `rpm` package) or edit CPACK_GENERATOR variable _CMakeLists.txt_ file in root of source code tree.
|
|
- Install generated package through your package manager (`yum`, `dnf`, `dpkg`).
|
|
|
|
_Note:_ If you don't want to generate binary packages, you can just install the project with `make install` (as root). But installation through your distribution's package manager is preferred way to keep your system clean and manageable in long term horizon.
|
|
|
|
|
|
### Compilers
|
|
|
|
For evaluating jobs you have to install tools that fit your needs. Here are some useful tips of different compillers to install.
|
|
|
|
**C/C++**
|
|
|
|
For compiling C and C++ programs is used `GCC` compiler. Maybe you could install install most of the staff by executing following command, but it's not needed.
|
|
```
|
|
# yum group install "Development Tools"
|
|
```
|
|
To install the compiler separately, you could install it from the distribution's repositories.
|
|
```
|
|
# yum install gcc gcc-c++ make
|
|
```
|
|
To get reasonably new version, you may consider installing [Red Hat Developer Toolset 4](https://access.redhat.com/documentation/en-US/Red_Hat_Developer_Toolset/4/html-single/4.0_Release_Notes/index.html), or install these from Fedora repo. In Debian, _testing_ repo can be used.
|
|
|
|
**C\#**
|
|
|
|
For new versions of **Mono**, we'll use [Xamarin repositories](http://www.mono-project.com/download/#download-lin). For Red Hat based OS:
|
|
```
|
|
# rpm --import "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF"
|
|
# yum-config-manager --add-repo http://download.mono-project.com/repo/centos/
|
|
# yum upgrade
|
|
# yum install mono-complete
|
|
```
|
|
For Debian based OS:
|
|
```
|
|
$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
|
|
$ echo "deb http://download.mono-project.com/repo/debian wheezy main" | sudo tee /etc/apt/sources.list.d/mono-xamarin.list
|
|
$ sudo apt-get update
|
|
$ sudo apt-get install mono-complete
|
|
```
|
|
|
|
**Free Pascal**
|
|
|
|
Free Pascal compiler 3.0.0 can be downloaded as `rpm` packages from [official website](http://www.freepascal.org/down/x86_64/linux-austria.var). In Debian, this version is [expetimental](https://packages.debian.org/experimental/fpc-3.0.0), but seems to be stable enough.
|
|
|
|
**Java**
|
|
_**TODO**_
|
|
|
|
## Windows Worker
|
|
From beginning we are determined to support Windows operating system on which some of the workers may run (especially for projects in C# programming language). Support for Windows is quite hard and time consuming and there were several problems during this. To ensure capability of compilation on Windows we set up CI for Windows named [Appveyor](http://www.appveyor.com/). However installation should be easy due to provided installation script.
|
|
|
|
There are only two additional dependencies needed, **Windows 7 and higher** and **Visual Studio 2015+**. Provided simple installation batch script should do all the work on Windows machine. Officially only VS2015 and 32-bit compilation is supported, because of hardcoded compile options in installation script. If different VS is needed or different platform, script should be changed to appropriate values, it should be quite simple and straightforward.
|
|
|
|
Mentioned script is placed in *install* directory alongside supportive scripts for UNIX systems and is named *win-build.cmd*. Provided script will do almost all the work connected with building and dependency resolving (using **NuGet** package manager and **msbuild** building system). Script should be run under 32-bit version of **Developer Command Prompt for VS2015** and from *install* directory.
|
|
|
|
Building and installing of worker is then quite simple, script has command line parameters which can be used to specify what will be done:
|
|
* *-build* - Don't have to be specified. Build worker and all its tests, all is saved in *build* folder and subfolders.
|
|
* *-clean* - Cleanup of downloaded NuGet packages and built application/libraries.
|
|
* *-test* - Build worker and run tests on compiled test cases.
|
|
* *-package* - Clickable installation generation using cpack and NSIS. [NSIS](http://nsis.sourceforge.net/) have to be installed on machine to get this to work.
|
|
|
|
```
|
|
install> win-build.cmd # same as: win-build.cmd -build
|
|
install> win-build.cmd -clean
|
|
install> win-build.cmd -test
|
|
install> win-build.cmd -package
|
|
```
|
|
|
|
All build binaries and cmake temporary files can be found in *build* folder, classically there will be subfolder *Release* which will contain compiled application with all needed dlls. Once if clickable installation binary is created, it can be found in *build* folder named something like *recodex-worker-${VERSION}-win32.exe*.
|
|
|
|
**Clickable installation feature:**
|
|
|
|
|
|
|
|
## WebApp
|