custom judges

FAQ.md

@@ -34,3 +34,40 @@ In **C#** language there are none of the handy functions to operate with textual
 ## Exit Codes
 
 For some particular languages ReCodEx provides mappings of exit codes from internal runners to messages. Full tables of codes can be found on [separate page](https://github.com/ReCodEx/wiki/wiki/Exit-Code-Mappings).
+
+## How to Write Custom Judges
+
+ReCodEx provides a few initial judges programs. They are mostly adopted from
+CodEx and installed automatically with the worker component. Judging programs
+have to meet some requirements. Basic ones are inspired by standard `diff`
+application -- two mandatory positional parameters which have to be the files
+for comparison and exit code reflecting if the result is correct (0) or wrong
+(1).
+
+This interface lacks support for returning additional data by the judges, for
+example similarity of the two files calculated as the Levenshtein edit distance.
+To allow passing these additional values an extended judge interface can be
+implemented:
+
+- Parameters: There are two mandatory positional parameters which have to be
+  files for comparison, first one is expected output, second one result from
+  tested program
+- Results:
+    - _comparison OK_
+		- exitcode: 0
+		- stdout: there is a single line with a double value which
+		  should be quality percentage of the judged file
+    - _comparison BAD_
+		- exitcode: 1
+		- stdout: can be empty
+	- _error during execution_
+		- exitcode: 2
+		- stderr: there should be description of error
+
+The additional double value is saved to the results file and can be used for
+score calculation in the frontend. If just the basic judge is used, the values
+are 1.0 for exit code 0 and 0.0 for exit code 1.
+
+If more values are needed for score computation, multiple judges can be used in
+sequence and the values used together. However, extended judge interface should
+comply most of possible use cases.