You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
69 lines
3.4 KiB
Markdown
69 lines
3.4 KiB
Markdown
Internal term *score calculator* denotes algorithm, which is responsible for computing overall score (correctness) of a submitted solution from the results of individual solution test. Each exercise is given a calculator name and associated calculator configuration (stored in DB text column as Yaml), which is used to initialize/prepare the calculator before it is used for computing the score.
|
|
|
|
We have implemented three calculators so far:
|
|
|
|
## Uniform
|
|
|
|
Trivial algorithm, which computes the overall score as arithmetic average of individual test scores. This calculator has no configuration.
|
|
|
|
## Weighted
|
|
|
|
Computes overall score as weighted average of individual test scores. The calculator expects that the weights are stored in the configuration in the following format:
|
|
|
|
```
|
|
testWeights:
|
|
"Test 01": 200
|
|
"Test 02": 300
|
|
"Test 03": 100
|
|
```
|
|
|
|
The keys are test names (must correspond with the `name` column of the `exercise_test`) and the values has to be integers. When all weights are equal, the weighted calculator behaves exactly as uniform calculator.
|
|
|
|
## Universal
|
|
|
|
The most complex calculator which was added last. It holds an AST (Abstract Syntax Tree) of an expression used to compute the score in the configuration.
|
|
|
|
The AST is encoded directly into Yaml structure -- each node is represented as a collection (object). A node must always have property `type`, which holds a string identifier of the node type (types of nodes are below). Internal nodes also have `children` property, which holds a list (array) of child nodes. Leaf nodes do not have `children` property.
|
|
|
|
Some nodes may have additional properties. Furthermore, a node is allowed to have arbitrary number of extension properties prefixed with `x-`. These properties are not recognized by API/backend, but they may be use by frontend (web app). The only reason they are recognized in the API is to ensure their correct (de)serialization. Other unknown properties are removed in the normalization process.
|
|
|
|
Leaf nodes (without children):
|
|
* `value` - a literal value, which holds one float (the value itself is stored in `value` property)
|
|
* `test-result` - reference to a test score, which is also a float between 0 and 1 (name of the test is in the `test` property)
|
|
|
|
Internal nodes (functions):
|
|
* `sum` - computes sum of all children values
|
|
* `mul` - computes product of all children values
|
|
* `sub` - binary node that computes *a* - *b*
|
|
* `div` - binary node that computes *a* / *b* (it yields 0 if *b* == 0)
|
|
* `neg` - unary node that negates its operand
|
|
* `min` - computes minimum of all children values
|
|
* `max` - computes maximum of all children values
|
|
* `avg` - computes arithmetic average of all children values
|
|
* `clamp` - unary node that clamps its operand value into [0,1] range
|
|
|
|
For practical reasons, we have decided to implement one optimization. Value literals may be also encoded as simple float numbers (instead of objects) in the child list. The `value`-typed objects and float literals are treated as equal; however, root node must be always an object (even when it is a literal).
|
|
|
|
#### Example
|
|
|
|
The following example should yield the same score as the weighted score config presented above.
|
|
|
|
```
|
|
type: div
|
|
children:
|
|
- type: avg
|
|
children:
|
|
- type: mul
|
|
children:
|
|
- 2.0
|
|
- type: test-result
|
|
test: "Test 01"
|
|
- type: mul
|
|
children:
|
|
- 3.0
|
|
- type: test-result
|
|
test: "Test 02"
|
|
- type: test-result
|
|
test: "Test 03"
|
|
- 6.0
|
|
``` |