LEdoian
/
recodex-wiki
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
fixes
master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'cf867f2906'
${ noResults }
recodex-wiki/Fileserver.md

2.1 KiB
Raw Blame History

Fileserver

The fileserver is a simple frontend to a disk storage space that contains auxiliary files for assignments, archives with job configuration and files submitted by users and evaluation results. These files are the only ones required for backend to run, so dedicated fileserver gives the possibility of testing backend separately. Also, one fileserver instance could be shared among multiple API instances (with the same broker), so common files does not need to be duplicated in each API instance.

One exception is that important files with character of database entry (but not stored in database due to size) are stored directly in filesystem of API server. But this fact does not devaluate benefit of separate fileserver. From security point of view, fileserver should be completely isolated from public internet to keep the data safe while API server must be public from its nature.

For a description of the communication protocol used by the frontend and workers, see the Communication chapter.

Description

The storage is implemented in Python, using the Flask web framework. This particular implementation evolved from a simple mock fileserver we used in early stages of development. It prooved to be very reliable, so we decided to keep fileserver as separate component instead of integrating this functionality into main API.

Internal storage structure

Fileserver stores its data in a configurable filesystem folder. This folder has the following subfolders:

  • ./submissions/<id> -- folders that contain files submitted by users (student's solutions to assignments). <id> is an identifier received from the ReCodEx API.
  • ./submission_archives/<id>.zip -- ZIP archives of all submissions. These are created automatically when a submission is uploaded. <id> is an identifier of the corresponding submission.
  • ./tasks/<subkey>/<key> -- supplementary task files (e.g. test inputs and outputs). <key> is a hash of the file content (sha-1 is used) and <subkey> is its first letter (this is an attempt to prevent creating a flat directory structure).