From 9a0f353025085184c8596c4ac471231882d0f190 Mon Sep 17 00:00:00 2001 From: Petr Stefan Date: Tue, 25 Oct 2016 21:30:09 +0200 Subject: [PATCH] Cleaner polishing --- Worker.md | 81 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 54 insertions(+), 27 deletions(-) diff --git a/Worker.md b/Worker.md index b5d4e15..b2538dd 100644 --- a/Worker.md +++ b/Worker.md @@ -285,49 +285,61 @@ box3.cpus = 1,2,3 # assign list of processors to isolate box 3 ### WrapSharp -WrapSharp is sandbox for programs in C# written also in C#. We have written it as a proof of concept sandbox for using in Windows environment. However, it's not properly tested and integrated to the worker yet. With just a little bit of effort there can be a running sandbox for C# programs on Windows system. +WrapSharp is sandbox for programs in C# written also in C#. We have written it as a proof of concept sandbox for using in Windows environment. However, it's not properly tested and integrated to the worker yet. Security audit should be done before using in production. After that, with just a little bit of effort integrating into worker there can be a running sandbox for C# programs on Windows system. ## Cleaner ### Description -Cleaner is integral part of **worker** which manages its cache folder, mainly deletes outdated files. Every cleaner maintains its one and only cache folder, which can be used by multiple workers. This means on one server there can be numerous instances of workers with the same cache folder, but there can be (and should be) only one cleaner. -Cleaner is written in **Python** and is used as simple script which just does its job and ends and therefore has to be cronned. For proper function of cleaner some suitable cronning interval has to be used. Its recommended to use 24 hour interval which should be sufficient enough. +Cleaner is integral part of worker which manages its cache folder, mainly deletes outdated files. Every cleaner instance maintains one cache folder, which can be used by multiple workers. This means on one server there can be numerous instances of workers with the same cache folder, but there should be only one cleaner. + +Cleaner is written in Python programming language and is used as simple script which just does its job and ends, so has to be cronned. For proper function of cleaner some suitable cronning interval has to be used. Its recommended to use 24 hour interval which should be sufficient enough. #### Last access timestamp + There is a bit of catch with cleaner service, to work properly, server filesystem has to have enabled last access timestamp. Cleaner checks these stamps and based on them it decides if file will be deleted or not, simple write timestamp or created at timestamp are not enough to reflect real usage and need of particular file. Last access timestamp feature is a bit controversial (more on this subject can be found [here](https://en.wikipedia.org/wiki/Stat_%28system_call%29#Criticism_of_atime)) and its not by default enabled on conventional filesystems. In linux this can be solved by adding `strictatime` option to `fstab` file. On Windows following command has to be executed (as administrator) `fsutil behavior set disablelastaccess 0`. ### Installation -To install and use the cleaner, it's necessary to have **Python** and **Pip** in version 3 installed. -* Firstly dependencies of cleaner has to be installed: +To install and use the cleaner, it's necessary to have Python3 with package manager `pip` installed. + +- Dependencies of cleaner has to be installed: ``` -pip install -r requirements.txt +$ pip install -r requirements.txt ``` - -#### Fedora (and other RPM distributions) -- run `python setup.py bdist_rpm --post-install ./cleaner/install/postinst` to generate binary `.rpm` package -- install package using `sudo dnf install ./dist/recodex-cleaner-0.1.0-1.noarch.rpm` (depends on actual version) - -#### Other Linux systems -- run installation as `python setup.py install --install-scripts /usr/bin` -- run postinst script as root with `sudo ./cleaner/install/postinst` - -#### Windows -- start `cmd` with administrator permissions -- decide in which folder cleaner should be installed, `C:\Program Files\ReCodEx\cleaner` is assumed -- run installation with `python setup.py install --install-scripts "C:\Program Files\ReCodEx\cleaner"` where path specified with `--install-scripts` can be changed -- copy configuration file alongside with installed executable using `copy install\config.yml "C:\Program Files\ReCodEx\cleaner\config.yml"` +- RPM distributions can make and install binary package. This can be done like this: +``` +$ python setup.py bdist_rpm --post-install ./cleaner/install/postinst +# yum install ./dist/recodex-cleaner--1.noarch.rpm +``` +- Other Linux distributions can install cleaner straight +``` +$ python setup.py install --install-scripts /usr/bin +# ./cleaner/install/postinst +``` +- For Windows installation do following: + - start `cmd` with administrator permissions + - run installation with + ``` + > python setup.py install --install-scripts \ + "C:\Program Files\ReCodEx\cleaner" + ``` + where path specified with `--install-scripts` can be changed + - copy configuration file alongside with installed executable using + ``` + > copy install\config.yml \ + "C:\Program Files\ReCodEx\cleaner\config.yml" + ``` ### Configuration and usage #### Configuration items -- **cache-dir** - directory which cleaner manages -- **file-age** - file age in seconds which are considered outdated and will be deleted +- **cache-dir** -- directory which cleaner manages +- **file-age** -- file age in seconds which are considered outdated and will be deleted #### Example configuration -``` +```{.yml} cache-dir: "/tmp" file-age: "3600" # in seconds ``` @@ -335,7 +347,22 @@ file-age: "3600" # in seconds #### Usage As stated before cleaner should be cronned, on linux systems this can be done by built in `cron` service or if there is `systemd` present cleaner itself provides `*.timer` file which can be used for cronning from `systemd`. On Windows systems internal scheduler should be used. -- Running cleaner from command line is fairly simple: `recodex-cleaner -c /etc/recodex/cleaner` -- Enable cleaner service using systemd: `systemctl start recodex-cleaner.timer` -- Add cleaner to linux cron service using following configuration line: `0 0 * * * /usr/bin/recodex-cleaner -c /etc/recodex/cleaner/config.yml` -- Add cleaner to Windows cheduler service with following command: `schtasks /create /sc daily /tn "ReCodEx Cleaner" /tr "\"C:\Program Files\ReCodEx\cleaner\recodex-cleaner.exe\" -c \"C:\Program Files\ReCodEx\cleaner\config.yml\""` +- Running cleaner from command line is fairly simple: +``` +$ recodex-cleaner -c /etc/recodex/cleaner +``` +- Enable cleaner service using systemd: +``` +$ systemctl start recodex-cleaner.timer +``` +- Add cleaner to linux cron service using following configuration line: +``` +0 0 * * * /usr/bin/recodex-cleaner -c /etc/recodex/cleaner/config.yml +``` +- Add cleaner to Windows cheduler service with following command: +``` +> schtasks /create /sc daily /tn "ReCodEx Cleaner" /tr \ + "\"C:\Program Files\ReCodEx\cleaner\recodex-cleaner.exe\" \ + -c \"C:\Program Files\ReCodEx\cleaner\config.yml\"" +``` +