Judge implementation improvement

master
Petr Stefan 8 years ago
parent 6e07c308e2
commit 492bbd2ab2

@ -2770,23 +2770,40 @@ directory is configurable and can be the same for multiple worker instances.
### Judges
For future extensibility is critical that judges have some shared interface of
calling and return values.
ReCodEx provides a few initial judges programs. They are mostly adopted from
CodEx system and installed with worker component. Judging programs have to meet
some requirements. Basic ones are inspired by standard `diff` application -- two
mandatory positional parameters which have to be the files for comparison and
exit code reflecting if the results is correct (0) of wrong (1).
This interface lacks support for returning additional data by the judges, for
example similarity of the two files calculated as Levenshtein's edit distance.
To allow passing these additional values an extended judge interface can be
implemented:
- Parameters: There are two mandatory positional parameters which has to be
files for comparision
- Results:
- _comparison OK_
- exitcode: 0
- stdout: there is one line with double value which should be set to 1.0
- stdout: there is one line with a double value which should be set to
1.0
- _comparison BAD_
- exitcode: 1
- stdout: there is one line with double value which should be percentage
of correctness of quality of two given files
- stdout: there is one line with a double value which should be
quality percentage of the two given files
- _error during execution_
- exitcode: 2
- stderr: there should be description of error
The additional double value is saved to the results file and can be used for
score calculation in the frontend. If just the basic judge is used, the values
are 1.0 for exit code 0 and 0.0 for exit code 1.
If more values are needed for score computation, multiple judges can be used in
sequence and the values used together. However, extended judge interface should
comply most of possible use cases.
### Additional libraries
@todo: libcurl, spdlog, boost, yaml-cpp, libarchive, cppzmq

Loading…
Cancel
Save