Updated Installation (markdown)

Installation.md

@@ -12,10 +12,11 @@ For more details about the individual modules, please see their readme pages.
 * [Core and REST API](https://github.com/ReCodEx/api)
 * [Broker](https://github.com/ReCodEx/broker)
 * [Monitor](https://github.com/ReCodEx/monitor)
-* [File Server](https://github.com/ReCodEx/fileserver)
 * [Worker](https://github.com/ReCodEx/worker)
 * [Cleaner](https://github.com/ReCodEx/cleaner)
 
+Btw. there used to be a [File Server](https://github.com/ReCodEx/fileserver)
+module as well, but we got the file management integrated in Core module.
 
 ## Prerequisites
 
@@ -76,7 +77,7 @@ on bare metal so they can provide more accurate measurements).
 The main part of ReCodEx is installed as follows:
 ```
 # dnf copr enable semai/ReCodEx
-# dnf install recodex-core recodex-web recodex-broker recodex-monitor recodex-fileserver
+# dnf install recodex-core recodex-web recodex-broker recodex-monitor
 ```
 
 The worker (and its utility cleaner) is installed thusly: 
@@ -84,15 +85,10 @@ The worker (and its utility cleaner) is installed thusly:
 # dnf install recodex-worker recodex-cleaner
 ```
 
-**Please note, that if you install worker on another server, it is strongly
-recommended to secure the connection between these two servers (by VPN or IPSec
-tunnel).**
-
 If successful, the following systemd services are now available:
 * `recodex-web`
 * `recodex-broker`
 * `recodex-monitor`
-* `recodex-fileserver`
 
 The core API runs under web server (does not need a custom service) and workers
 will be covered separately later. 
@@ -111,20 +107,6 @@ The last command should show status of the service, which should be running.
 
 ## Configure The Installation
 
-### Fileserver
-
-The fileserver runs under `mod_wsgi` in Apache, so its configuration is in
-`/etc/httpd/conf.d/010-fileserver.conf`. There should be no need to edit this
-config file. You need only to se the HTTP authentication credentials and match
-them with credentials in core module config (`fileServer` > `auth` structure).
-
-The credentials are in `/etc/httpd/recodex_htpasswd`. You may set them using
-`htpasswd`, an Apache CLI tool for generating auth config files. Do not forget
-to restart your web server after you are done:
-```
-#> systemctl restart httpd
-```
-
 ### Broker
 
 Broker configuration is in `/etc/recodex/broker/config.yml`. The most important
@@ -185,8 +167,13 @@ Btw. if you need to invalidate all ReCodEx security tokens at once, just modify
 this string (that will effectively sing everybody off, so everyone will need to
 go through login process again).
 
-3. Configure `fileServer` connection. Under normal circumstances, you just need
+3. Configure `fileStorage` paths. The storages are basically directories managed
-to fill in the credentials you have stored in `/etc/httpd/recodex_htpasswd`.
+by the core component. It is divided into two parts -- hash storage that uses
+hashes as file names for data deduplication and local storage for regular files.
+Note that both directories **must be created manually** and you need to make them
+both readable and writeable by `apache` user and `recodex` user (use fs ACLs).
+Recommended place to store the data would be under `/var/recodex-filestorage`
+where you can put `hash` and `local` subdirs.
 
 4. In the `broker` structure, `auth` must hold credentials that match those set
 in broker configuration (`notifier` structure) and the `address` must provide
@@ -198,7 +185,11 @@ configuration, the address should be something like
 `wss://your.recodex.domain:443/ws`. The 443 port makes sure the initial
 handshake is done in HTTPS manner by Apache. 
 
-6. Setup generated URLs in notification `emails`. The `footerUrl` should be the
+6. Let workers access the API to exchange files. The `workerFiles` modules needs
+to be enabled and you need to set the secret `auth` credentials which are also
+set in worker configuration.
+
+7. Setup generated URLs in notification `emails`. The `footerUrl` should be the
 base URL of the web application. The `from` parameter configures the `From:`
 field set in all notification mails. The `defaultAdminTo` should be a string or
 an array of strings with email addresses where the error notifications will be
@@ -207,12 +198,12 @@ to send them to actual administrators of ReCodEx only. On the other hand, it is
 a good idea to have more than one administrator to reduce the chance of
 overlooking these failures.
 
-7. Set your SMTP configuration in the `mail` structure. SMTP is necessary so
+8. Set your SMTP configuration in the `mail` structure. SMTP is necessary so
 the API can send notification emails. You may temporary use ReCodEx without
 emails (setting `emails` > `debugMode` to `true`), but emails are required for
 key features like resetting forgotten password.
 
-8. Although this is the last step, it is perhaps the most important one. Fill in
+9. Although this is the last step, it is perhaps the most important one. Fill in
 your database credentials of the `recodex` user (which you were supposed to
 create at the very beginning) into `doctrine` configuration (Doctrine framework
 is responsible for database interface in the core module). 
@@ -347,8 +338,9 @@ embedded in config file names).
 
 The worker needs broker and file server to operate. Update `broker-uri` so it
 matches your broker location and port designated to workers. The `file-managers`
-structure configures the file server access (`hostname` and HTTP auth
+structure configures the file server access provided by core API module
-credentials).  
+(`hostname` has to be set to https URL pointing to API and HTTP auth credentials
+must match credentials set in `workerFiles` section of core configuratuion).
 
 Create worker(s) working directory (e.g., `/var/recodex-worker-wd`) and cache
 directory (e.g., `/var/recodex-worker-cache`) and set their paths to
@@ -456,6 +448,10 @@ Install .NET core SDK:
 # dnf -y install dotnet-sdk-3.1
 ```
 
+DOTNET_ROOT: /usr/lib64/dotnet
+DOTNET_BUNDLE_EXTRACT_BASE_DIR: /box/.dotnet-bundle_extract
+
+
 Download the following files:
 ```
 https://raw.githubusercontent.com/ReCodEx/utils/master/runners/cs/Reader.cs
@@ -727,23 +723,22 @@ Not quite yet. There are at least two things you should consider.
 
 ### Backup
 
-The data are stored at two places -- in the database and in the file server.
+The data are stored at two places -- in the database and in the file storage.
 The database can be easily dumped thusly:
 ```
 mysqldump --default-character-set=utf8mb4 -uroot -p recodex > /path/to/backup/file.sql
 ```
 Where `recodex` is the database name.
 
-Furthermore, you need to backup the fileserver main directory
+To backup 
-(`/var/recodex-fileserver` by default), preferably using tools like `rsync`.
+Furthermore, you need to backup the filestorage `hash` and `local` directories
-In addition, it is a good idea to backup the `/opt/recodex-code/uploaded_data`
+(which you may have located at `/var/recodex-fileserver` as suggested),
-directory. Important uploaded files will be eventually all transferred to
+preferably using tools like `rsync`.
-fileserver, but at present, some of them are still kept here.
 
 It might be a good idea to perform a backup every night and to keep several last
-copies. For instance, in our setup, we keep last 7 daily backups, all backups made
+copies. For instance, in our pilot setup, we keep last 7 daily backups, all
-on the 1st of every month for the last year, and all backups made on January 1st
+backups made on the 1st of every month for the last year, and all backups made
-of every year.
+on January 1st of every year.
 
 
 ### Monitoring
@@ -755,3 +750,8 @@ or wget to verify the ReCodEx is running, or you can use more sophisticated tool
 For our main instance, we use [Prometheus](https://prometheus.io/) with
 `node_exporter` and `mysqld_exporter` to gather performance statistics and
 [Grafana](https://grafana.com/) to visualize them.
+
+
+## Are we done now?!
+
+Yes, we are. Enjoy and handle with care.