Fileserver chapter updated

master
Petr Stefan 8 years ago
parent 09fe924e12
commit f09104d165

@ -2,43 +2,47 @@
The fileserver is a simple frontend to a disk storage space that contains The fileserver is a simple frontend to a disk storage space that contains
auxiliary files for assignments, files submitted by users and evaluation auxiliary files for assignments, files submitted by users and evaluation
results. For a description of the communication protocol used by the frontend results.
and workers, see the [Communication](#communication) chapter
For a description of the communication protocol used by the frontend
and workers, see the [Communication](#communication) chapter.
## Description ## Description
The storage is implemented in Python, using the Flask web framework. This The storage is implemented in Python, using the Flask web framework. This
particular implementation evolved from a simple mock fileserver we used in early particular implementation evolved from a simple mock fileserver we used in early
stages of development. stages of development. It prooved to be very reliable, so we decided to keep fileserver
as separate component instead of integrating this functionality into main API.
### Internal storage structure ### Internal storage structure
Fileserver stores its data in a configurable filesystem folder. This folder has Fileserver stores its data in a configurable filesystem folder. This folder has
the following subfolders: the following subfolders:
- `./submissions/<id>` - folders that contain files submitted by users - `./submissions/<id>` -- folders that contain files submitted by users
(student's solutions to assignments). `<id>` is an identifier received from (student's solutions to assignments). `<id>` is an identifier received from
the ReCodEx API. the ReCodEx API.
- `./submission_archives/<id>.zip` - ZIP archives of all submissions. These are - `./submission_archives/<id>.zip` -- ZIP archives of all submissions. These are
created automatically when a submission is uploaded. `<id>` is an identifier created automatically when a submission is uploaded. `<id>` is an identifier
of the corresponding submission. of the corresponding submission.
- `./tasks/<subkey>/<key>` - supplementary task files (e.g. test inputs and - `./tasks/<subkey>/<key>` -- supplementary task files (e.g. test inputs and
outputs). `<key>` is a hash of the file content (sha-1 is used) and `<subkey>` outputs). `<key>` is a hash of the file content (sha-1 is used) and `<subkey>`
is its first letter (this is an attempt to prevent creating a flat directory is its first letter (this is an attempt to prevent creating a flat directory
structure). structure).
## Installation
To install and use the fileserver, it's necessary to have Python and Pip ## Installation
installed.
1. Clone this repository To install and use the fileserver, it's necessary to have Python3 with `pip` package manager installed. It's needed to install the dependencies. From clonned repository run the following command:
2. Install dependencies using `pip`
``` ```
pip install -r requirements.txt $ pip install -r requirements.txt
``` ```
That's it. Fileserver doesn't need any special installation. It's possible to build and install _rpm_ package or install it without packaging the same way as monitor, but it's only optional. The installation would provide you with script `recodex-fileserver` in you `PATH`. No systemd unit files are provided, because of the configuration and usage of fileserver component is much different to our other Python parts.
## Configuration and usage ## Configuration and usage
There are several ways of running the ReCodEx fileserver. We'll cover two There are several ways of running the ReCodEx fileserver. We'll cover two
@ -47,32 +51,62 @@ typical use cases.
### Running in development mode ### Running in development mode
For simple development usage, it's possible to run the fileserver in the command For simple development usage, it's possible to run the fileserver in the command
line: line. Allowed options are described below.
```
usage: fileserver.py [--directory WORKING_DIRECTORY]
{runserver,shell} ...
```
- **runserver** argument starts the Flask development server (i.e. `app.run()`). As additional argument can be given a port number.
- **shell** argument instructs Flask to run a Python shell inside application context.
~~~~~~~~~~~~~~~ Simple development server on port 9999 can be run as
python3 fileserver.py runserver
~~~~~~~~~~~~~~~
When run like this, the fileserver creates a temporary directory where it stores ```
all the files and which is deleted when it exits. It's possible to configure it $ python3 fileserver.py runserver 9999
to use any other directory. Use the `--help` flag for a complete list of options ```
(such as the port where the server listens).
When run like this command, the fileserver creates a temporary directory where it stores all the files and which is deleted when it exits.
### Running with uWSGI and a web server
If you need features such as HTTP authentication or efficient serving of static ### Running as WSGI script in a web server
If you need features such as HTTP authentication (recommended) or efficient serving of static
files, it is recommended to run the app in a full-fledged web server (such as files, it is recommended to run the app in a full-fledged web server (such as
Apache or Nginx) using WSGI. Following examples are simplified, use the `--help` Apache or Nginx) using WSGI. Apache configuration can be generated by `mkconfig.py` script from the repository.
flag of the `mkconfig.py` script to see all available options.
```
usage: mkconfig.py apache [-h] [--port PORT] --working-directory
WORKING_DIRECTORY [--htpasswd HTPASSWD]
[--user USER]
```
- **port** -- port where the fileserver should listen
- **working_directory** -- directory where the files should be stored
- **htpasswd** -- path to user file fot HTTP Basic Authentication
- **user** -- user under which the server should be run
### Running using uWSGI ### Running using uWSGI
Another option is to run fileserver as a standalone app via uWSGI service. Setup is also quite simple, configuration file can be also generated by `mkconfig.py` script.
1. (Optional) Create a user for running the fileserver 1. (Optional) Create a user for running the fileserver
2. Make sure that your user can access your clone of the repository 2. Make sure that your user can access your clone of the repository
3. Run `python mkconfig/mkconfig.py uwsgi --port PORT --user YOUR_USER 3. Run `mkconfig.py` script.
--working-directory DIR` ```
usage: mkconfig.py uwsgi [-h] [--user USER] [--port PORT]
[--socket SOCKET]
--working-directory WORKING_DIRECTORY
```
- **user** -- user under which the server should be run
- **port** -- port where the fileserver should listen
- **socket** -- path to UNIX socket where the fileserver should listen
- **working_directory** -- directory where the files should be stored
4. Save the configuration file generated by the script and run it with uWSGI, 4. Save the configuration file generated by the script and run it with uWSGI,
either directly or using systemd. This depends heavily on your distribution either directly or using systemd. This depends heavily on your distribution.
5. To integrate this with another web server, see the [uWSGI 5. To integrate this with another web server, see the [uWSGI
documentation](http://uwsgi-docs.readthedocs.io/en/latest/WebServers.html) documentation](http://uwsgi-docs.readthedocs.io/en/latest/WebServers.html)
@ -81,3 +115,4 @@ necessary to convert the configuration file to XML and make sure that the
python3 plugin is loaded instead of python. This plugin also uses Python 3.4, python3 plugin is loaded instead of python. This plugin also uses Python 3.4,
even though the rest of the system uses Python 3.5 - make sure to install even though the rest of the system uses Python 3.5 - make sure to install
dependencies for the correct version. dependencies for the correct version.

Loading…
Cancel
Save