Note that `rpm` and `deb` packages are build in the same time. You may need to have `rpmbuild` command (usually as `rpmbuild` or `rpm` package) or edit CPACK_GENERATOR variable _CMakeLists.txt_ file in root of source code tree.
- Install generated package through your package manager (`yum`, `dnf`, `dpkg`).
@ -108,7 +108,7 @@ Installation of broker program does following step to your computer:
Description of configurable items in broker's config. Mandatory items are bold, optional italic.
- _clients_ -- specifies address and port to bind for clients (eq. frontends)
- _clients_ -- specifies address and port to bind for clients (frontend instance)
- _address_ -- hostname or IP address as string (`*` for any)
- _port_ -- desired port
- _workers_ -- specifies address and port to bind for workers
also communicates with fileserver, downloads supplementary files and
uploads detailed results.
These parts run in separate threads of the same process and communicate through a ZeroMQ in-process
socket. This design allows the worker to keep sending `ping` messages even when
it is processing a job.
These parts run in separate threads of the same process and communicate through a ZeroMQ in-process socket. Alternative approach would be using shared memory region with unique access, but messaging is generally accepted to be safer. Shared memory has to be used very carefully because of race condition issues when reading and writing concurently. Also, messages inside worker are small, so there is no huge overhead copying data between threads. This two threaded design allows the worker to keep sending `ping` messages even when it is processing a job.
After receiving an evaluation request, worker has to:
@ -92,7 +90,7 @@ $ git submodule update --init
It is supposed that your current working directory is that one with clonned worker source codes.
- Prepare environment running `mkdir build && cd build`
- Build sources by `cmake ..` following by `make -j#` where '#' symbol refers to number of your CPU threads.
Note that `rpm` and `deb` packages are build in the same time. You may need to have `rpmbuild` command (usually as `rpmbuild` or `rpm` package) or edit CPACK_GENERATOR variable in _CMakeLists.txt_ file in root of source code tree.
- Install generated package through your package manager (`yum`, `dnf`, `dpkg`).
@ -155,8 +153,8 @@ Mandatory items are bold, optional italic.
- **broker-uri** - URI of the broker (hostname, IP address, including port, ...)
- _broker-ping-interval_ - time interval how often to send ping messages to broker. Used units are milliseconds.
- _max-broker-liveness_ - specifies how many pings in a row can broker miss without making the worker dead.
- _headers_ - map of headers specifies worker's capabilities
- _env_ - list of enviromental variables which are sent to broker in init command
- _threads_ - information about available threads for this worker
- **hwgroup** - hardware group of this worker. Hardware group must specify worker hardware and software capabilities and it is main item for broker routing decisions.
- _working-directory_ - where will be stored all needed files. Can be the same for multiple workers on one server.
@ -263,7 +261,7 @@ To add a new worker you need to do a few steps:
Isolate is used as one and only sandbox for linux-based operating systems. Headquarters of this project can be found at [GitHub](https://github.com/ioi/isolate) and more of its installation and setup can be found in [installation](#installation) section. Isolate uses linux kernel features for sandboxing and thus its security depends on them, namely _kernel namespaces_ and _cgroups_ are used. Similar functionality can now be partially achieved with systemd.
From the very beginning of ReCodEx project there was only one thing sure: Isolate will be used. Almost everything else changed but Isolate persist, this of course has some implications: main operating system of whole backend should be linux-based and worker will be designed to interact well with Isolate. This precondition was fulfilled and worker has fully integrated Isolate with almost all possible functionality which Isolate provides. This also means that job configuration was heavily affected and reflects Isolate's capabilities.
From the very beginning of ReCodEx project there was sure that Isolate sandbox for Linux environment will be used. There is no suitable general purpose sandbox on Windows platform, so main operation system of whole backend should be linux-based. Set of supported operations in Isolate seems reasonable for every sandbox, so most of its functionality is accessible from job configuration. As there is no other sandbox, naming often reflects Isolate's names. However worker is prepared to run on Windows too, so integrating with other sandboxes (as libraries or commandline tools) is possible.
Isolate as sandbox provides wide scale of functionality which can be used to limit resources or even cut off particular resources from sandboxed program. There is of course basics like limiting cpu-time and memory consumption, but there can be found also wall-time (human perception of time) or extra-time which is extra limit added to other time limits to increase chance of successful exiting of sandboxed program. From other features there is limiting stack-size, redirection of stdin, stdout or stderr from/to a file. Worth of mentioning is also defining number of processes/threads which can be created or defining environment variables which are passed to sandboxed program.
@ -303,6 +301,8 @@ Cleaner is written in Python programming language and is used as simple script w
There is a bit of catch with cleaner service, to work properly, server filesystem has to have enabled last access timestamp. Cleaner checks these stamps and based on them it decides if file will be deleted or not, simple write timestamp or created at timestamp are not enough to reflect real usage and need of particular file. Last access timestamp feature is a bit controversial (more on this subject can be found [here](https://en.wikipedia.org/wiki/Stat_%28system_call%29#Criticism_of_atime)) and it is not by default enabled on conventional filesystems. In linux this can be solved by adding `strictatime` option to `fstab` file. On Windows following command has to be executed (as administrator) `fsutil behavior set disablelastaccess 0`.
Another possibility seems to be to update last modified timestamp when accessing the file. This timestamp is used in most major filesystems, so there are less issues with compatibility than last access timestamp. The modified timestamp then must be updated by workers at each access, for example using `touch` command or similar. Final decision on better of these ways will be made after practical experience of running production system.
### Installation
To install and use the cleaner, it is necessary to have Python3 with package manager `pip` installed.
Petr Stefan
8 years ago