From 9f719761f159d154152d90aa2ea7d39f1b3a7e35 Mon Sep 17 00:00:00 2001 From: Martin Polanka Date: Sat, 17 Mar 2018 14:00:16 +0100 Subject: [PATCH] custom judges --- FAQ.md | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/FAQ.md b/FAQ.md index 450f0af..8756b6f 100644 --- a/FAQ.md +++ b/FAQ.md @@ -33,4 +33,41 @@ In **C#** language there are none of the handy functions to operate with textual ## Exit Codes -For some particular languages ReCodEx provides mappings of exit codes from internal runners to messages. Full tables of codes can be found on [separate page](https://github.com/ReCodEx/wiki/wiki/Exit-Code-Mappings). \ No newline at end of file +For some particular languages ReCodEx provides mappings of exit codes from internal runners to messages. Full tables of codes can be found on [separate page](https://github.com/ReCodEx/wiki/wiki/Exit-Code-Mappings). + +## How to Write Custom Judges + +ReCodEx provides a few initial judges programs. They are mostly adopted from +CodEx and installed automatically with the worker component. Judging programs +have to meet some requirements. Basic ones are inspired by standard `diff` +application -- two mandatory positional parameters which have to be the files +for comparison and exit code reflecting if the result is correct (0) or wrong +(1). + +This interface lacks support for returning additional data by the judges, for +example similarity of the two files calculated as the Levenshtein edit distance. +To allow passing these additional values an extended judge interface can be +implemented: + +- Parameters: There are two mandatory positional parameters which have to be + files for comparison, first one is expected output, second one result from + tested program +- Results: + - _comparison OK_ + - exitcode: 0 + - stdout: there is a single line with a double value which + should be quality percentage of the judged file + - _comparison BAD_ + - exitcode: 1 + - stdout: can be empty + - _error during execution_ + - exitcode: 2 + - stderr: there should be description of error + +The additional double value is saved to the results file and can be used for +score calculation in the frontend. If just the basic judge is used, the values +are 1.0 for exit code 0 and 0.0 for exit code 1. + +If more values are needed for score computation, multiple judges can be used in +sequence and the values used together. However, extended judge interface should +comply most of possible use cases. \ No newline at end of file