merge
Simon Rozsival
8 years ago
commit
fcf0c69a64
@ -1,122 +0,0 @@
|
|||||||
# Coding style
|
|
||||||
|
|
||||||
Every project should have some consistent coding style in which all contributors write. Bellow you can find our conventions on which we agreed on and which we try to keep.
|
|
||||||
|
|
||||||
## C++
|
|
||||||
|
|
||||||
**NOTE, that C++ projects have set code linter (`cmake-format`) with custom format. To reformat code run `make format` inside `build` directory of the project (probably not working on Windows).** For quick introduction into our format, see following paragraphs.
|
|
||||||
|
|
||||||
In C++ is written worker and broker. Generally it is used underscore style with all small letters. Inspired by [Google C++ style guide](https://google.github.io/styleguide/cppguide.html). If something is not defined than naming/formatting can be arbitrary, but should be similar to bellow-defined behaviour.
|
|
||||||
|
|
||||||
### Naming convention
|
|
||||||
* For source codes use all lower case with underscores not dashes. Header files should end with `.h` and C++ files with `.cpp`.
|
|
||||||
* Typenames are all in lower case with underscores between words. This is applicable to classes, structs, typedefs, enums and type template parameters.
|
|
||||||
* Variable names can be divided on local variables and class members. Local variables are all lower case with underscores between words. Class members have in addition trailing underscore on the end (struct data members do not have underscore on the end).
|
|
||||||
* Constants are just like any other variables and do not have any specifics.
|
|
||||||
* All function names are again all lower case with underscores between words.
|
|
||||||
* Namespaces if there are ones they should have lower case and underscores.
|
|
||||||
* Macros are classical and should have all capitals and underscores.
|
|
||||||
* Comments can be two types documentational and ordinery ones in code. Documentation should start with `/**` and end with `*/`, convention inside them is javadoc documentation format. Classical comments in code are one liners which starts with `//` and end with the end of the line.
|
|
||||||
|
|
||||||
### Formatting convention
|
|
||||||
* Line length is not explicitly defined, but should be reasonable.
|
|
||||||
* All files should use UTF-8 character set.
|
|
||||||
* For code indentation tabs (`\t`) are used.
|
|
||||||
* Function declaration/definition: return type should be on the same line as the rest of the declaration, if line is too long, than particular parameters are placed on new line. Opening parenthesis of function should be placed on new line bellow declaration. Its possible to write small function which can be on only one line. Between parameter and comma should be one space.
|
|
||||||
```
|
|
||||||
int run(int id, string msg);
|
|
||||||
|
|
||||||
void print_hello_world()
|
|
||||||
{
|
|
||||||
std::cout << "Hello world" << std::endl;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
int get_five() { return 5; }
|
|
||||||
```
|
|
||||||
* Lambda expressions: same formatting as classical functions
|
|
||||||
```
|
|
||||||
auto hello = [](int x) { std::cout << "hello_" << x << std::endl; }
|
|
||||||
```
|
|
||||||
* Function calls: basically same as function header definition.
|
|
||||||
* Condition: after if, or else there always have to be one space in front of opening bracket and again one space after closing condition bracket (and in front of opening parenthesis). If and else always should be on separate lines. Inside condition there should not be any pointless spaces.
|
|
||||||
```
|
|
||||||
if (x == 5) {
|
|
||||||
std::cout << "Exactly five!" << std::endl;
|
|
||||||
} else if (x < 5 && y > 5) {
|
|
||||||
std::cout << "Whoa, that is weird format!" << std::endl;
|
|
||||||
} else {
|
|
||||||
std::cout << "I dont know what is this!" << std::endl;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
* For and while cycles: basically same rules as for if condition.
|
|
||||||
* Try-catch blocks: again same rules as for if conditions. Closing parentheses of try block should be on the same line as catch block.
|
|
||||||
```
|
|
||||||
try {
|
|
||||||
int a = 5 / 0;
|
|
||||||
} catch (...) {
|
|
||||||
std::cout << "Division by zero" << std::endl;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
* Switch: again basics are the same as for if condition. Case statements should not be indented and case body should be intended with 1 tab.
|
|
||||||
```
|
|
||||||
switch (switched) {
|
|
||||||
case 0: // no tab indent
|
|
||||||
... // 1 tab indent
|
|
||||||
break;
|
|
||||||
case 1:
|
|
||||||
...
|
|
||||||
break;
|
|
||||||
default:
|
|
||||||
exit(1);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
* Pointers and references: no spaces between period or arrow in accessing type member. No spaces after asterisk or ampersand. In declaration of pointer or reference format should be that asterisk or ampersand is adjacent to name of the variable not type.
|
|
||||||
```
|
|
||||||
number = *ptr;
|
|
||||||
ptr = &val;
|
|
||||||
number = ptr->number;
|
|
||||||
number = val_ref.number;
|
|
||||||
|
|
||||||
int *i;
|
|
||||||
int &j;
|
|
||||||
|
|
||||||
// bad format bellow
|
|
||||||
int* i;
|
|
||||||
int * i;
|
|
||||||
```
|
|
||||||
* Boolean expression: long boolean expression should be divided into more lines. The division point should always be after logical operators.
|
|
||||||
```
|
|
||||||
if (i > 10 &&
|
|
||||||
j < 10 &&
|
|
||||||
k > 20) {
|
|
||||||
std::cout << "Were here!" << std::endl;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
* Return values should not be generally wrapped with parentheses, only if needed.
|
|
||||||
* Preprocessor directives start with `#` and always should start at the beginning of the line.
|
|
||||||
* Classes: sections aka. public, protected, private should have same indentation as the class start itself. Opening parenthesis of class should be on the same line as class name.
|
|
||||||
```
|
|
||||||
class my_class {
|
|
||||||
public:
|
|
||||||
void class_function();
|
|
||||||
private:
|
|
||||||
int class_member_;
|
|
||||||
};
|
|
||||||
```
|
|
||||||
* Operators: around all binary operators there always should be spaces.
|
|
||||||
```
|
|
||||||
int x = 5;
|
|
||||||
x = x * 5 / 5;
|
|
||||||
x = x + 5 * (10 - 5);
|
|
||||||
```
|
|
||||||
|
|
||||||
## Python
|
|
||||||
|
|
||||||
Python code should correspond to [PEP 8](https://www.python.org/dev/peps/pep-0008/) style.
|
|
||||||
|
|
||||||
## PHP
|
|
||||||
TODO:
|
|
||||||
|
|
||||||
## JavaScript
|
|
||||||
TODO:
|
|
||||||
@ -1,251 +0,0 @@
|
|||||||
# Overall Architecture
|
|
||||||
|
|
||||||
## Description
|
|
||||||
|
|
||||||
**ReCodEx** is designed to be very modular and configurable. One such configuration is sketched in the following picture. There are two separate frontend instances with distinct databases sharing common backend part. This configuration may be suitable for MFF UK -- basic programming course and KSP competition. Note, that connections between components are not fully accurate.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
**Web app** is main part of whole project from user point of view. It provides nice user interface and it is the only part, that interacts with outside world directly. **Web API** contains almost all logic of the app including _user management and authentication_, _storing and versioning files_ (with help of **File server**), _counting and assigning points_ to users etc. Advanced users may connect to the API directly or may create custom frontends. **Broker** is essential part of whole architecture. It maintains list of available **Workers**, receives submissions from the **Web API** and routes them further and reports progress of evaluations back to the **Web app**. **Worker** securely runs each received job and evaluate its results. **Monitor** resends evaluation progress messages to the **Web app** in order to be presented to users.
|
|
||||||
|
|
||||||
|
|
||||||
## Communication
|
|
||||||
|
|
||||||
Detailed communication inside the ReCodEx system is captured in the following
|
|
||||||
image and described in sections below. Red connections are through ZeroMQ
|
|
||||||
sockets, blue are through WebSockets and green are through HTTP(S). All ZeroMQ
|
|
||||||
messages are sent as multipart with one string (command, option) per part, with
|
|
||||||
no empty frames (unles explicitly specified otherwise).
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### Broker - Worker communication
|
|
||||||
|
|
||||||
Broker acts as server when communicating with worker. Listening IP address and port are configurable, protocol family is TCP. Worker socket is of DEALER type, broker one is ROUTER type. Because of that, very first part of every (multipart) message from broker to worker must be target worker's socket identity (which is saved on its **init** command).
|
|
||||||
|
|
||||||
#### Commands from broker to worker:
|
|
||||||
|
|
||||||
- **eval** -- evaluate a job. Requires 3 message frames:
|
|
||||||
- `job_id` -- identifier of the job (in ASCII representation -- we avoid
|
|
||||||
endianness issues and also support alphabetic ids)
|
|
||||||
- `job_url` -- URL of the archive with job configuration and submitted source
|
|
||||||
code
|
|
||||||
- `result_url` -- URL where the results should be stored after evaluation
|
|
||||||
- **intro** -- introduce yourself to the broker (with **init** command) -- this is
|
|
||||||
required when the broker loses track of the worker who sent the command.
|
|
||||||
Possible reasons for such event are e.g. that one of the communicating sides
|
|
||||||
shut down and restarted without the other side noticing.
|
|
||||||
- **pong** -- reply to **ping** command, no arguments
|
|
||||||
|
|
||||||
#### Commands from worker to broker:
|
|
||||||
|
|
||||||
- **init** -- introduce self to the broker. Useful on startup or after reestablishing lost connection. Requires at least 2 arguments:
|
|
||||||
- `hwgroup` -- hardware group of this worker
|
|
||||||
- `header` -- additional header describing worker capabilities. Format must
|
|
||||||
be `header_name=value`, every header shall be in a separate message frame.
|
|
||||||
There is no limit on number of headers.
|
|
||||||
|
|
||||||
There is also an optional third argument -- additional information. If
|
|
||||||
present, it should be separated from the headers with an empty frame. The
|
|
||||||
format is the same as headers. Supported keys for additional information are:
|
|
||||||
- `description` -- a human readable description of the worker for
|
|
||||||
administrators (it will show up in broker logs)
|
|
||||||
- `current_job` -- an identifier of a job the worker is now processing. This
|
|
||||||
is useful when we are reassembling a connection to the broker and need it
|
|
||||||
to know the worker will not accept a new job.
|
|
||||||
- **done** -- notifying of finished job. Contains following message frames:
|
|
||||||
- `job_id` -- identifier of finished job
|
|
||||||
- `result` -- response result, possible values are:
|
|
||||||
- OK -- evaluation finished successfully
|
|
||||||
- FAILED -- job failed and cannot be reassigned to another worker (e.g.
|
|
||||||
due to error in configuration)
|
|
||||||
- INTERNAL_ERROR -- job failed due to internal worker error, but another
|
|
||||||
worker might be able to process it (e.g. downloading a file failed)
|
|
||||||
- `message` -- a human readable error message
|
|
||||||
- **progress** -- notice about current evaluation progress. Contains following message frames:
|
|
||||||
- `job_id` -- identifier of current job
|
|
||||||
- `state` -- what is happening now.
|
|
||||||
- DOWNLOADED -- submission successfuly fetched from fileserver
|
|
||||||
- FAILED -- something bad happened and job was not executed at all
|
|
||||||
- UPLOADED -- results are uploaded to fileserver
|
|
||||||
- STARTED -- evaluation of tasks started
|
|
||||||
- ENDED -- evaluation of tasks is finished
|
|
||||||
- ABORTED -- evaluation of job encountered internal error, job will be rescheduled to another worker
|
|
||||||
- FINISHED -- whole execution is finished and worker ready for another job execution
|
|
||||||
- TASK -- task state changed -- see below
|
|
||||||
- `task_id` -- only present for "TASK" state -- identifier of task in current job
|
|
||||||
- `task_state` -- only present for "TASK" state -- result of task evaluation. One of:
|
|
||||||
- COMPLETED -- task was successfully executed without any error, subsequent task will be executed
|
|
||||||
- FAILED -- task ended up with some error, subsequent task will be skipped
|
|
||||||
- SKIPPED -- some of the previous dependencies failed to execute, so this task will not be executed at all
|
|
||||||
- **ping** -- tell broker I am alive, no arguments
|
|
||||||
|
|
||||||
|
|
||||||
#### Heartbeating
|
|
||||||
|
|
||||||
It is important for the broker and workers to know if the other side is still
|
|
||||||
working (and connected). This is achieved with a simple heartbeating protocol.
|
|
||||||
|
|
||||||
The protocol requires the workers to send a **ping** command regularly (the
|
|
||||||
interval is configurable on both sides -- future releases might let the worker
|
|
||||||
send its ping interval with the **init** command). Upon receiving a **ping**
|
|
||||||
command, the broker responds with **pong**.
|
|
||||||
|
|
||||||
Whenever a heartbeating message doesn't arrive, a counter called _liveness_ is
|
|
||||||
decreased. When this counter drops to zero, the other side is considered
|
|
||||||
disconnected. When a message arrives, the liveness counter is set back to its
|
|
||||||
maximum value, which is configurable for both sides.
|
|
||||||
|
|
||||||
When the broker decides a worker disconnected, it tries to reschedule its jobs
|
|
||||||
to other workers.
|
|
||||||
|
|
||||||
If a worker thinks the broker crashed, it tries to reconnect periodically, with
|
|
||||||
a bounded, exponentially increasing delay.
|
|
||||||
|
|
||||||
This protocol proved great robustness in real world testing. Thus whole backend
|
|
||||||
is reliable and can outlive short term issues with connection without problems.
|
|
||||||
Also, increasing delay of ping messages does not flood the network when there
|
|
||||||
are problems. We experienced no issues since we are using this protocol.
|
|
||||||
|
|
||||||
### Worker - File Server communication
|
|
||||||
|
|
||||||
Worker is communicating with file server only from _execution thread_. Supported
|
|
||||||
protocol is HTTP optionally with SSL encryption (**recommended**). If supported
|
|
||||||
by server and used version of libcurl, HTTP/2 standard is also available. File
|
|
||||||
server should be set up to require basic HTTP authentication and worker is
|
|
||||||
capable to send corresponding credentials with each request.
|
|
||||||
|
|
||||||
#### Worker side
|
|
||||||
|
|
||||||
Workers comunicate with the file server in both directions -- they download
|
|
||||||
student's submissions and then upload evaluation results. Internally, worker is
|
|
||||||
using libcurl C library with very similar setup. In both cases it can verify
|
|
||||||
HTTPS certificate (on Linux against system cert list, on Windows against
|
|
||||||
downloaded one from CURL website during installation), support basic HTTP
|
|
||||||
authentication, offer HTTP/2 with fallback to HTTP/1.1 and fail on error
|
|
||||||
(returned HTTP status code is >=400). Worker have list of credentials to all
|
|
||||||
available file servers in its config file.
|
|
||||||
|
|
||||||
- download file -- standard HTTP GET request to given URL expecting file content as response
|
|
||||||
- upload file -- standard HTTP PUT request to given URL with file data as body -- same as command line tool `curl` with option `--upload-file`
|
|
||||||
|
|
||||||
#### File server side
|
|
||||||
|
|
||||||
File server has its own internal directory structure, where all the files are stored. It provides simple REST API to get them or create new ones. File server does not provide authentication or secured connection by itself, but it is supposed to run file server as WSGI script inside a web server (like Apache) with proper configuration. Relevant commands for communication with workers:
|
|
||||||
|
|
||||||
- **GET /submission_archives/\<id\>.\<ext\>** -- gets an archive with submitted source code and corresponding configuration of this job evaluation
|
|
||||||
- **GET /exercises/\<hash\>** -- gets a file, common usage is for input files or
|
|
||||||
reference result files
|
|
||||||
- **PUT /results/\<id\>.\<ext\>** -- upload archive with evaluation results under specified name (should be same _id_ as name of submission archive). On successful upload returns JSON `{ "result": "OK" }` as body of returned page.
|
|
||||||
|
|
||||||
If not specified otherwise, `zip` format of archives is used. Symbol `/` in API description is root of file server's domain. If the domain is for example `fs.recodex.org` with SSL support, getting input file for one task could look as GET request to `https://fs.recodex.org/tasks/8b31e12787bdae1b5766ebb8534b0adc10a1c34c`.
|
|
||||||
|
|
||||||
|
|
||||||
### Broker - Monitor communication
|
|
||||||
|
|
||||||
Broker communicates with monitor also through ZeroMQ over TCP protocol. Type of
|
|
||||||
socket is same on both sides, ROUTER. Monitor is set to act as server in this
|
|
||||||
communication, its IP address and port are configurable in monitor's config
|
|
||||||
file. ZeroMQ socket ID (set on monitor's side) is "recodex-monitor" and must be
|
|
||||||
sent as first frame of every multipart message -- see ZeroMQ ROUTER socket
|
|
||||||
documentation for more info.
|
|
||||||
|
|
||||||
Note that the monitor is designed so that it can receive data both from the
|
|
||||||
broker and workers. The current architecture prefers the broker to do all the
|
|
||||||
communication so that the workers do not have to know too many network services.
|
|
||||||
|
|
||||||
Monitor is treated as a somewhat optional part of whole solution, so no special
|
|
||||||
effort on communication realibility was made.
|
|
||||||
|
|
||||||
#### Commands from monitor to broker:
|
|
||||||
|
|
||||||
Because there is no need for the monitor to communicate with the broker, there
|
|
||||||
are no commands so far. Any message from monitor to broker is logged and
|
|
||||||
discarded.
|
|
||||||
|
|
||||||
Commands from broker to monitor:
|
|
||||||
|
|
||||||
- **progress** -- notification about progress with job evaluation. See [Progress callback](#progress-callback) section for more info.
|
|
||||||
|
|
||||||
|
|
||||||
### Broker - Web API communication
|
|
||||||
|
|
||||||
Broker communicates with main REST API through ZeroMQ connection over TCP. Socket
|
|
||||||
type on broker side is ROUTER, on frontend part it is DEALER. Broker acts as a
|
|
||||||
server, its IP address and port is configurable in the API.
|
|
||||||
|
|
||||||
#### Commands from API to broker:
|
|
||||||
|
|
||||||
- **eval** -- evaluate a job. Requires at least 4 frames:
|
|
||||||
- `job_id` -- identifier of this job (in ASCII representation -- we avoid endianness issues and also support alphabetic ids)
|
|
||||||
- `header` -- additional header describing worker capabilities. Format must be `header_name=value`, every header shall be in a separate message frame. There is no maximum limit on number of headers. There may be also no headers at all. A worker is considered suitable for the job if and only if it satisfies all of its headers.
|
|
||||||
- empty frame -- frame which contains only empty string and serves only as breakpoint after headers
|
|
||||||
- `job_url` -- URI location of archive with job configuration and submitted source code
|
|
||||||
- `result_url` -- remote URI where results will be pushed to
|
|
||||||
|
|
||||||
#### Commands from broker to API (all are responses to **eval** command):
|
|
||||||
|
|
||||||
- **ack** -- this is first message which is sent back to frontend right after eval command arrives, basically it means "Hi, I am all right and am capable of receiving job requests", after sending this broker will try to find acceptable worker for arrived request
|
|
||||||
- **accept** -- broker is capable of routing request to a worker
|
|
||||||
- **reject** -- broker cannot handle this job (for example when the requirements
|
|
||||||
specified by the headers cannot be met). There are (rare) cases when the
|
|
||||||
broker finds that it cannot handle the job after it was confirmed. In such
|
|
||||||
cases it uses the frontend REST API to mark the job as failed.
|
|
||||||
|
|
||||||
|
|
||||||
#### Asynchronous communication between broker and API
|
|
||||||
|
|
||||||
Only a fraction of the errors that can happen during evaluation can be detected
|
|
||||||
while there is a ZeroMQ connection between the API and broker. To notify the
|
|
||||||
frontend of the rest, we need an asynchronous communication channel that can be
|
|
||||||
used by the broker when the status of a job changes (it's finished, it failed
|
|
||||||
permanently, the only worker capable of processing it disconnected...).
|
|
||||||
|
|
||||||
This functionality is supplied by the `broker-reports/` API endpoint group --
|
|
||||||
see its documentation for more details.
|
|
||||||
|
|
||||||
### File Server - Web API communication
|
|
||||||
|
|
||||||
File server has a REST API for interaction with other parts of ReCodEx. Description of communication with workers is in [File server side](#file-server-side) section. On top of that, there are other commands for interaction with the API:
|
|
||||||
|
|
||||||
- **GET /results/\<id\>.\<ext\>** -- download archive with evaluated results of job _id_
|
|
||||||
- **POST /submissions/\<id\>** -- upload new submission with identifier _id_. Expects that the body of the POST request uses file paths as keys and the content of the files as values. On successful upload returns JSON `{ "archive_path": <archive_url>, "result_path": <result_url> }` in response body. From _archive_path_ the submission can be downloaded (by worker) and corresponding evaluation results should be uploaded to _result_path_.
|
|
||||||
- **POST /tasks** -- upload new files, which will be available by names equal to `sha1sum` of their content. There can be uploaded more files at once. On successful upload returns JSON `{ "result": "OK", "files": <file_list> }` in response body, where _file_list_ is dictionary of original file name as key and new URL with already hashed name as value.
|
|
||||||
|
|
||||||
There are no plans yet to support deleting files from this API. This may change in time.
|
|
||||||
|
|
||||||
Web API calls these fileserver endpoints with standard HTTP requests. There are no special commands involved. There is no communication in opposite direction.
|
|
||||||
|
|
||||||
|
|
||||||
### Monitor - Web app communication
|
|
||||||
|
|
||||||
Monitor interacts with web application through WebSocket connection. Monitor acts as server and browsers are connecting to it. IP address and port are configurable. When client connects to the monitor, it sends a message with string representation of channel id (which messages are interested in, usually id of evaluating job). There can be multiple listeners per channel, even (shortly) delayed connections will receive all messages from the very beginning.
|
|
||||||
|
|
||||||
When monitor receives **progress** message from broker there are two options:
|
|
||||||
|
|
||||||
- there is no WebSocket connection for listed channel (job id) -- message is dropped
|
|
||||||
- there is active WebSocket connection for listed channel -- message is parsed into JSON format (see below) and send as string to that established channel. Messages for active connections are queued, so no messages are discarded even on heavy workload.
|
|
||||||
|
|
||||||
Message JSON format is dictionary (associative array) with keys:
|
|
||||||
|
|
||||||
- **command** -- type of progress, one of:
|
|
||||||
- DOWNLOADED -- submission successfuly fetched from fileserver
|
|
||||||
- FAILED -- something bad happened and job was not executed at all
|
|
||||||
- UPLOADED -- results are uploaded to fileserver
|
|
||||||
- STARTED -- evaluation of tasks started
|
|
||||||
- ENDED -- evaluation of all tasks finished, worker now just have to send results and cleanup after execution
|
|
||||||
- ABORTED -- evaluation of job encountered internal error, job will be rescheduled to another worker
|
|
||||||
- FINISHED -- whole execution finished and worker is ready for another job execution
|
|
||||||
- TASK -- task state changed, further information will be provided -- see below
|
|
||||||
- **task_id** -- id of currently evaluated task. Present only if **command** is "TASK".
|
|
||||||
- **task_state** -- state of task with id **task_id**. Present only if **command** is "TASK". Value is one of "COMPLETED", "FAILED" and "SKIPPED".
|
|
||||||
- COMPLETED -- task was successfully executed without any error, subsequent task will be executed
|
|
||||||
- FAILED -- task ended up with some error, subsequent task will be skipped
|
|
||||||
- SKIPPED -- some of the previous dependencies failed to execute, so this task will not be executed at all
|
|
||||||
|
|
||||||
|
|
||||||
### Web app - Web API communication
|
|
||||||
|
|
||||||
Provided web application runs as javascript client inside user's browser. It communicates with REST API on the server through standard HTTP requests. Documentation of the main REST API is in separate [document](https://recodex.github.io/api/) due to its extensiveness. Results are returned as JSON payload, which is simply parsed in web application and presented to the users.
|
|
||||||
|
|
||||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,437 @@
|
|||||||
|
# System configuration
|
||||||
|
|
||||||
|
## Worker
|
||||||
|
|
||||||
|
Worker should have some default configuration which is applied to worker itself
|
||||||
|
or may be used in given jobs (implicitly if something is missing, or explicitly
|
||||||
|
with special variables). This configuration should be hardcoded and can be
|
||||||
|
rewritten by explicitly declared configuration file. Format of this
|
||||||
|
configuration is yaml with similar structure to job configuration.
|
||||||
|
|
||||||
|
### Configuration items
|
||||||
|
|
||||||
|
Mandatory items are bold, optional italic.
|
||||||
|
|
||||||
|
- **worker-id** -- unique identification of worker at one server. This id is
|
||||||
|
used by _isolate_ sanbox on linux systems, so make sure to meet isolate's
|
||||||
|
requirements (default is number from 1 to 999).
|
||||||
|
- _worker-description_ -- human readable description of this worker
|
||||||
|
- **broker-uri** -- URI of the broker (hostname, IP address, including port,
|
||||||
|
...)
|
||||||
|
- _broker-ping-interval_ -- time interval how often to send ping messages to
|
||||||
|
broker. Used units are milliseconds.
|
||||||
|
- _max-broker-liveness_ -- specifies how many pings in a row can broker miss
|
||||||
|
without making the worker dead.
|
||||||
|
- _headers_ -- map of headers specifies worker's capabilities
|
||||||
|
- _env_ -- list of enviromental variables which are sent to broker in init
|
||||||
|
command
|
||||||
|
- _threads_ -- information about available threads for this worker
|
||||||
|
- **hwgroup** -- hardware group of this worker. Hardware group must specify
|
||||||
|
worker hardware and software capabilities and it is main item for broker
|
||||||
|
routing decisions.
|
||||||
|
- _working-directory_ -- where will be stored all needed files. Can be the same
|
||||||
|
for multiple workers on one server.
|
||||||
|
- **file-managers** -- addresses and credentials to all file managers used (eq.
|
||||||
|
all different frontends using this worker)
|
||||||
|
- **hostname** -- URI of file manager
|
||||||
|
- _username_ -- username for http authentication (if needed)
|
||||||
|
- _password_ -- password for http authentication (if needed)
|
||||||
|
- _file-cache_ -- configuration of caching feature
|
||||||
|
- _cache-dir_ -- path to caching directory. Can be the same for multiple
|
||||||
|
workers.
|
||||||
|
- _logger_ -- settings of logging capabilities
|
||||||
|
- _file_ -- path to the logging file with name without suffix.
|
||||||
|
`/var/log/recodex/worker` item will produce `worker.log`, `worker.1.log`,
|
||||||
|
...
|
||||||
|
- _level_ -- level of logging, one of `off`, `emerg`, `alert`, `critical`,
|
||||||
|
`err`, `warn`, `notice`, `info` and `debug`
|
||||||
|
- _max-size_ -- maximal size of log file before rotating
|
||||||
|
- _rotations_ -- number of rotation kept
|
||||||
|
- _limits_ -- default sandbox limits for this worker. All items are described in
|
||||||
|
assignments section in job configuration description. If some limits are not
|
||||||
|
set in job configuration, defaults from worker config will be used. In such
|
||||||
|
case the worker's defaults will be set as the maximum for the job. Also,
|
||||||
|
limits in job configuration cannot exceed limits from worker.
|
||||||
|
|
||||||
|
### Example config file
|
||||||
|
|
||||||
|
```{.yml}
|
||||||
|
worker-id: 1
|
||||||
|
broker-uri: tcp://localhost:9657
|
||||||
|
broker-ping-interval: 10 # milliseconds
|
||||||
|
max-broker-liveness: 10
|
||||||
|
headers:
|
||||||
|
env:
|
||||||
|
- c
|
||||||
|
- cpp
|
||||||
|
threads: 2
|
||||||
|
hwgroup: "group1"
|
||||||
|
working-directory: /tmp/recodex
|
||||||
|
file-managers:
|
||||||
|
- hostname: "http://localhost:9999" # port is optional
|
||||||
|
username: "" # can be ignored in specific modules
|
||||||
|
password: "" # can be ignored in specific modules
|
||||||
|
file-cache: # only in case that there is cache module
|
||||||
|
cache-dir: "/tmp/recodex/cache"
|
||||||
|
logger:
|
||||||
|
file: "/var/log/recodex/worker" # w/o suffix - actual names will
|
||||||
|
# be worker.log, worker.1.log,...
|
||||||
|
level: "debug" # level of logging
|
||||||
|
max-size: 1048576 # 1 MB; max size of file before log rotation
|
||||||
|
rotations: 3 # number of rotations kept
|
||||||
|
limits:
|
||||||
|
time: 5 # in secs
|
||||||
|
wall-time: 6 # seconds
|
||||||
|
extra-time: 2 # seconds
|
||||||
|
stack-size: 0 # normal in KB, but 0 means no special limit
|
||||||
|
memory: 50000 # in KB
|
||||||
|
parallel: 1
|
||||||
|
disk-size: 50
|
||||||
|
disk-files: 5
|
||||||
|
environ-variable:
|
||||||
|
ISOLATE_BOX: "/box"
|
||||||
|
ISOLATE_TMP: "/tmp"
|
||||||
|
bound-directories:
|
||||||
|
- src: /tmp/recodex/eval_5
|
||||||
|
dst: /evaluate
|
||||||
|
mode: RW,NOEXEC
|
||||||
|
```
|
||||||
|
|
||||||
|
### Isolate sandbox
|
||||||
|
|
||||||
|
New feature in version 1.3 is possibility of limit Isolate box to one or more
|
||||||
|
cpu or memory node. This functionality is provided by _cpusets_ kernel mechanism
|
||||||
|
and is now integrated in isolate. It is allowed to set only `cpuset.cpus` and
|
||||||
|
`cpuset.mems` which should be just fine for sandbox purposes. As kernel
|
||||||
|
functionality further description can be found in manual page of _cpuset_ or in
|
||||||
|
Linux documentation in section `linux/Documentation/cgroups/cpusets.txt`. As
|
||||||
|
previously stated this settings can be applied for particular isolate boxes and
|
||||||
|
has to be written in isolate configuration. Standard configuration path should
|
||||||
|
be `/usr/local/etc/isolate` but it may depend on your installation process.
|
||||||
|
Configuration of _cpuset_ in there is really simple and is described in example
|
||||||
|
below.
|
||||||
|
|
||||||
|
```
|
||||||
|
box0.cpus = 0 # assign processor with ID 0 to isolate box with ID 0
|
||||||
|
box0.mems = 0 # assign memory node with ID 0
|
||||||
|
# if not set, linux by itself will decide where should
|
||||||
|
# the sandboxed programs run at
|
||||||
|
box2.cpus = 1-3 # assign range of processors to isolate box 2
|
||||||
|
box2.mems = 4-7 # assign range of memory nodes
|
||||||
|
box3.cpus = 1,2,3 # assign list of processors to isolate box 3
|
||||||
|
```
|
||||||
|
|
||||||
|
- **cpuset.cpus:** Cpus limitation will restrict sandboxed program only to
|
||||||
|
processor threads set in configuration. On hyperthreaded processors this means
|
||||||
|
that all virtual threads are assignable, not only the physical ones. Value can
|
||||||
|
be represented by single number, list of numbers separated by commas or range
|
||||||
|
with hyphen delimiter.
|
||||||
|
- **cpuset.mems:** This value is particularly handy on NUMA systems which has
|
||||||
|
several memory nodes. On standard desktop computers this value should always
|
||||||
|
be zero because only one independent memory node is present. As stated in
|
||||||
|
`cpus` limitation there can be single value, list of values separated by comma
|
||||||
|
or range stated with hyphen.
|
||||||
|
|
||||||
|
## Broker
|
||||||
|
|
||||||
|
### Configuration items
|
||||||
|
|
||||||
|
Description of configurable items in broker's config. Mandatory items are bold,
|
||||||
|
optional italic.
|
||||||
|
|
||||||
|
- _clients_ -- specifies address and port to bind for clients (frontend
|
||||||
|
instance)
|
||||||
|
- _address_ -- hostname or IP address as string (`*` for any)
|
||||||
|
- _port_ -- desired port
|
||||||
|
- _workers_ -- specifies address and port to bind for workers
|
||||||
|
- _address_ -- hostname or IP address as string (`*` for any)
|
||||||
|
- _port_ -- desired port
|
||||||
|
- _max_liveness_ -- maximum amount of pings the worker can fail to send
|
||||||
|
before it is considered disconnected
|
||||||
|
- _max_request_failures_ -- maximum number of times a job can fail (due to
|
||||||
|
e.g. worker disconnect or a network error when downloading something from
|
||||||
|
the fileserver) and be assigned again
|
||||||
|
- _monitor_ -- settings of monitor service connection
|
||||||
|
- _address_ -- IP address of running monitor service
|
||||||
|
- _port_ -- desired port
|
||||||
|
- _notifier_ -- details of connection which is used in case of errors and good
|
||||||
|
to know states
|
||||||
|
- _address_ -- address where frontend API runs
|
||||||
|
- _port_ -- desired port
|
||||||
|
- _username_ -- username which can be used for HTTP authentication
|
||||||
|
- _password_ -- password which can be used for HTTP authentication
|
||||||
|
- _logger_ -- settings of logging capabilities
|
||||||
|
- _file_ -- path to the logging file with name without suffix.
|
||||||
|
`/var/log/recodex/broker` item will produce `broker.log`, `broker.1.log`,
|
||||||
|
...
|
||||||
|
- _level_ -- level of logging, one of `off`, `emerg`, `alert`, `critical`,
|
||||||
|
`err`, `warn`, `notice`, `info` and `debug`
|
||||||
|
- _max-size_ -- maximal size of log file before rotating
|
||||||
|
- _rotations_ -- number of rotation kept
|
||||||
|
|
||||||
|
### Example config file
|
||||||
|
|
||||||
|
```{.yml}
|
||||||
|
# Address and port for clients (frontend)
|
||||||
|
clients:
|
||||||
|
address: "*"
|
||||||
|
port: 9658
|
||||||
|
# Address and port for workers
|
||||||
|
workers:
|
||||||
|
address: "*"
|
||||||
|
port: 9657
|
||||||
|
max_liveness: 10
|
||||||
|
max_request_failures: 3
|
||||||
|
monitor:
|
||||||
|
address: "127.0.0.1"
|
||||||
|
port: 7894
|
||||||
|
notifier:
|
||||||
|
address: "127.0.0.1"
|
||||||
|
port: 8080
|
||||||
|
username: ""
|
||||||
|
password: ""
|
||||||
|
logger:
|
||||||
|
file: "/var/log/recodex/broker" # w/o suffix - actual names will be
|
||||||
|
# broker.log, broker.1.log, ...
|
||||||
|
level: "debug" # level of logging
|
||||||
|
max-size: 1048576 # 1 MB; max size of file before log rotation
|
||||||
|
rotations: 3 # number of rotations kept
|
||||||
|
```
|
||||||
|
|
||||||
|
## Monitor
|
||||||
|
|
||||||
|
Configuration file is located in subdirectory `monitor` of standard ReCodEx
|
||||||
|
configuration folder `/etc/recodex/`. It is in YAML format as all of the other
|
||||||
|
configurations. Format is very similar to configurations of broker or workers.
|
||||||
|
|
||||||
|
### Configuration items
|
||||||
|
|
||||||
|
Description of configurable items, bold ones are required, italics ones are
|
||||||
|
optional.
|
||||||
|
|
||||||
|
- _websocket_uri_ -- URI where is the endpoint of websocket connection. Must be
|
||||||
|
visible to the clients (directly or through public proxy)
|
||||||
|
- string representation of IP address or a hostname
|
||||||
|
- port number
|
||||||
|
- _zeromq_uri_ -- URI where is the endpoint of zeromq connection from broker.
|
||||||
|
Could be hidden from public internet.
|
||||||
|
- string representation of IP address or a hostname
|
||||||
|
- port number
|
||||||
|
- _logger_ -- settings of logging
|
||||||
|
- _file_ -- path with name of log file. Defaults to
|
||||||
|
`/var/log/recodex/monitor.log`
|
||||||
|
- _level_ -- logging level, one of "debug", "info", "warning", "error" and
|
||||||
|
"critical"
|
||||||
|
- _max-size_ -- maximum size of log file before rotation in bytes
|
||||||
|
- _rotations_ -- number of rotations kept
|
||||||
|
|
||||||
|
### Example configuration file
|
||||||
|
|
||||||
|
```{.yml}
|
||||||
|
---
|
||||||
|
websocket_uri:
|
||||||
|
- "127.0.0.1"
|
||||||
|
- 4567
|
||||||
|
zeromq_uri:
|
||||||
|
- "127.0.0.1"
|
||||||
|
- 7894
|
||||||
|
logger:
|
||||||
|
file: "/var/log/recodex/monitor.log"
|
||||||
|
level: "debug"
|
||||||
|
max-size: 1048576 # 1 MB
|
||||||
|
rotations: 3
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Cleaner
|
||||||
|
|
||||||
|
### Configuration items
|
||||||
|
- **cache-dir** -- directory which cleaner manages
|
||||||
|
- **file-age** -- file age in seconds which are considered outdated and will be deleted
|
||||||
|
|
||||||
|
### Example configuration
|
||||||
|
```{.yml}
|
||||||
|
cache-dir: "/tmp"
|
||||||
|
file-age: "3600" # in seconds
|
||||||
|
```
|
||||||
|
|
||||||
|
## REST API
|
||||||
|
|
||||||
|
The API can be configured in `config.neon` and `config.local.neon` files in
|
||||||
|
`app/config` directory. The first file is predefined by authors and should not
|
||||||
|
be modified. The second one is not present and could be created by copying
|
||||||
|
`config.local.neon.example` template in the config directory. Local
|
||||||
|
configuration have higher precedence, so it will override default values from
|
||||||
|
`config.neon`.
|
||||||
|
|
||||||
|
### Configurable items
|
||||||
|
|
||||||
|
Description of configurable items. All timeouts are in milliseconds if not
|
||||||
|
stated otherwise.
|
||||||
|
|
||||||
|
- accessManager -- configuration of access token in [JWT
|
||||||
|
standard](https://www.rfc-editor.org/rfc/rfc7519.txt). Do **not** modify
|
||||||
|
unless you really know what are you doing.
|
||||||
|
- fileServer -- connection to fileserver
|
||||||
|
- address -- URI of fileserver
|
||||||
|
- auth -- _username_ and _password_ for HTTP basic authentication
|
||||||
|
- timeouts -- _connection_ timeout for establishing new connection and
|
||||||
|
_request_ timeout for completing one request
|
||||||
|
- broker -- connection to broker
|
||||||
|
- address -- URI of broker
|
||||||
|
- auth -- _username_ and _password_ for broker callback authentication back
|
||||||
|
to API
|
||||||
|
- timeouts -- _ack_ timeout for first response that broker receives the
|
||||||
|
message, _send_ timeout how long try to send new job to the broker and
|
||||||
|
_result_ timeout how long to wait for confirmation if job can be processed
|
||||||
|
or not
|
||||||
|
- monitor -- connection to monitor
|
||||||
|
- address -- URI of monitor
|
||||||
|
- CAS -- CAS external authentication
|
||||||
|
- serviceId -- visible identifier of this service
|
||||||
|
- ldapConnection -- parameters for connecting to LDAP, _hostname_,
|
||||||
|
_base_dn_, _port_, _security_ and _bindName_
|
||||||
|
- fields -- names of LDAP keys for informations as _email_, _firstName_ and
|
||||||
|
_lastName_
|
||||||
|
- emails -- common configuration for sending email (addresses and template
|
||||||
|
variables)
|
||||||
|
- apiUrl -- base URL of API server including port (for referencing pictures
|
||||||
|
in messages)
|
||||||
|
- footerUrl -- link in the message footer
|
||||||
|
- siteName -- name of frontend (ReCodEx, or KSP for unique instance for KSP
|
||||||
|
course)
|
||||||
|
- githubUrl -- URL to GitHub repository of this project
|
||||||
|
- from -- sending email address
|
||||||
|
- failures -- admin messages on errors
|
||||||
|
- emails -- additional info for sending mails, _to_ is admin mail address,
|
||||||
|
_from_ is source address, _subjectPrefix_ is prefix of mail subject
|
||||||
|
- forgottenPassword -- user messages for changing passwords
|
||||||
|
- redirectUrl -- URL of web application where the password can be changed
|
||||||
|
- tokenExpiration -- expiration timeout of temporary token (in seconds)
|
||||||
|
- emails -- additional info for sending mails, _from_ is source address and
|
||||||
|
_subjectPrefix_ is prefix of mail subject
|
||||||
|
- mail -- configuration of sending mails
|
||||||
|
- smtp -- using SMTP server, have to be "true"
|
||||||
|
- host -- address of the server
|
||||||
|
- port -- sending port (common values are 25, 465, 587)
|
||||||
|
- username -- login to the server
|
||||||
|
- password -- password to the server
|
||||||
|
- secure -- security, values are empty for no security, "ssl" or "tls"
|
||||||
|
- context -- additional parameters, depending on used mail engine. For
|
||||||
|
examle self-signed certificates can be allowed as _verify_peer_ and
|
||||||
|
_verify_peer_name_ to false and _allow_self_signed_ to true under _ssl_
|
||||||
|
key (see example).
|
||||||
|
|
||||||
|
Outside the parameters section of configuration is configuration for Doctrine.
|
||||||
|
It is ORM framework which maps PHP objects (entities) into database tables and
|
||||||
|
rows. The configuration is simple, required items are only _user_, _password_
|
||||||
|
and _host_ with _dbname_, i.e. address of database computer (mostly localhost)
|
||||||
|
with name of ReCodEx database.
|
||||||
|
|
||||||
|
### Example local configuration file
|
||||||
|
|
||||||
|
```{.yml}
|
||||||
|
parameters:
|
||||||
|
accessManager:
|
||||||
|
leeway: 60
|
||||||
|
issuer: https://recodex.projekty.ms.mff.cuni.cz
|
||||||
|
audience: https://recodex.projekty.ms.mff.cuni.cz
|
||||||
|
expiration: 86400 # 24 hours in seconds
|
||||||
|
usedAlgorithm: HS256
|
||||||
|
allowedAlgorithms:
|
||||||
|
- HS256
|
||||||
|
verificationKey: "recodex-123"
|
||||||
|
fileServer:
|
||||||
|
address: http://127.0.0.1:9999
|
||||||
|
auth:
|
||||||
|
username: "user"
|
||||||
|
password: "pass"
|
||||||
|
timeouts:
|
||||||
|
connection: 500
|
||||||
|
broker:
|
||||||
|
address: tcp://127.0.0.1:9658
|
||||||
|
auth:
|
||||||
|
username: "user"
|
||||||
|
password: "pass"
|
||||||
|
timeouts:
|
||||||
|
ack: 100
|
||||||
|
send: 5000
|
||||||
|
result: 1000
|
||||||
|
monitor:
|
||||||
|
address: wss://recodex.projekty.ms.mff.cuni.cz:4443/ws
|
||||||
|
CAS:
|
||||||
|
serviceId: "cas-uk"
|
||||||
|
ldapConnection:
|
||||||
|
hostname: "ldap.cuni.cz"
|
||||||
|
base_dn: "ou=people,dc=cuni,dc=cz"
|
||||||
|
port: 389
|
||||||
|
security: SSL
|
||||||
|
bindName: "cunipersonalid"
|
||||||
|
fields:
|
||||||
|
email: "mail"
|
||||||
|
firstName: "givenName"
|
||||||
|
lastName: "sn"
|
||||||
|
emails:
|
||||||
|
apiUrl: https://recodex.projekty.ms.mff.cuni.cz:4000
|
||||||
|
footerUrl: https://recodex.projekty.ms.mff.cuni.cz
|
||||||
|
siteName: "ReCodEx"
|
||||||
|
githubUrl: https://github.com/ReCodEx
|
||||||
|
from: "ReCodEx <noreply@example.com>"
|
||||||
|
failures:
|
||||||
|
emails:
|
||||||
|
to: "Admin Name <admin@example.com>"
|
||||||
|
from: %emails.from%
|
||||||
|
subjectPrefix: "ReCodEx Failure Report - "
|
||||||
|
forgottenPassword:
|
||||||
|
redirectUrl: "https://recodex.projekty.ms.mff.cuni.cz/
|
||||||
|
forgotten-password/change"
|
||||||
|
tokenExpiration: 600 # 10 minues
|
||||||
|
emails:
|
||||||
|
from: %emails.from%
|
||||||
|
subjectPrefix: "ReCodEx Forgotten Password Request - "
|
||||||
|
mail:
|
||||||
|
smtp: true
|
||||||
|
host: "smtp.ps.stdin.cz"
|
||||||
|
port: 587
|
||||||
|
username: "user"
|
||||||
|
password: "pass"
|
||||||
|
secure: "tls"
|
||||||
|
context:
|
||||||
|
ssl:
|
||||||
|
verify_peer: false
|
||||||
|
verify_peer_name: false
|
||||||
|
allow_self_signed: true
|
||||||
|
doctrine:
|
||||||
|
user: "user"
|
||||||
|
password: "pass"
|
||||||
|
host: localhost
|
||||||
|
dbname: "recodex-api"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Web application
|
||||||
|
|
||||||
|
### Configurable items
|
||||||
|
|
||||||
|
Description of configurable options. Bold are required values, optional ones are
|
||||||
|
in italics.
|
||||||
|
|
||||||
|
- **NODE_ENV** -- mode of the server
|
||||||
|
- **API_BASE** -- base address of API server, including port and API version
|
||||||
|
- **PORT** -- port where the app is listening
|
||||||
|
- _WEBPACK_DEV_SERVER_PORT_ -- port for webpack dev server when running in
|
||||||
|
development mode. Default one is 8081, this option might be useful when this
|
||||||
|
port is necessary for some other service.
|
||||||
|
|
||||||
|
### Example configuration file
|
||||||
|
|
||||||
|
```
|
||||||
|
NODE_ENV=production
|
||||||
|
API_BASE=https://recodex.projekty.ms.mff.cuni.cz:4000/v1
|
||||||
|
PORT=8080
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
<!---
|
||||||
|
// vim: set formatoptions=tqn flp+=\\\|^\\*\\s* textwidth=80 colorcolumn=+1:
|
||||||
|
-->
|
||||||
|
|
||||||
@ -1,24 +0,0 @@
|
|||||||
### [[Home]]
|
|
||||||
|
|
||||||
### Content
|
|
||||||
* [[Introduction]]
|
|
||||||
* [[User documentation]]
|
|
||||||
* [[Overall architecture]]
|
|
||||||
* [[Assignments]]
|
|
||||||
* [[Submission flow]]
|
|
||||||
* [[Installation]]
|
|
||||||
* [[Worker]]
|
|
||||||
* [[Broker]]
|
|
||||||
* [[Monitor]]
|
|
||||||
* [[Fileserver]]
|
|
||||||
* [[Web API]]
|
|
||||||
* [[Web application]]
|
|
||||||
* [[Database]]
|
|
||||||
* [[Conclusion]]
|
|
||||||
|
|
||||||
### Separated pages
|
|
||||||
* [[FAQ]]
|
|
||||||
* [[Logo]]
|
|
||||||
* [[Coding style]]
|
|
||||||
* [[Database schema]]
|
|
||||||
|
|
||||||
Loading…
Reference in New Issue