Improvements

master
Petr Stefan 8 years ago
parent ce7c198529
commit e378fc5aeb

@ -665,33 +665,34 @@ state.
There are lot of things which deserves discussion concerning results of There are lot of things which deserves discussion concerning results of
evaluation, how they should be displayed, what should be visible or not and also evaluation, how they should be displayed, what should be visible or not and also
some what kind of reward for users solutions should be chosen. what kind of reward for users solutions should be chosen.
At first let us focus on all kinds of outputs from executed programs within job. At first let us focus on all kinds of outputs from executed programs within job.
Out of discussion is that supervisors should be able to view almost all outputs Out of discussion is that supervisors should be able to view almost all outputs
from solutions if they choose them to be visible and recorded. This feature is from solutions if they choose them to be visible and recorded. This feature is
critical in debugging either whole exercises or users solutions. But should it critical in debugging either whole exercises or users solutions. But should it
be default behaviour to record every output? Absolutely not, supervisor should be default behaviour to record every output? Absolutely not, supervisor should
have choice to turn it on but defaults has to be not record it. Even without have a choice to turn it on, but discarding the outputs has to be the default
this functionality can be file base around whole ReCodEx system quite large and option. Even without this functionality a file base around whole ReCodEx system
on top of that outputs from executed programs can be sometimes very extensive. can become quite large and on top of that outputs from executed programs can be
Storing this amount of data is purely nonsense to every solution. But if sometimes very extensive. Storing this amount of data is inefficient and
requested by supervisor then this feature should be available. unnecessary to most of the solutions. However, on supervisor request this
feature should be available.
Question what should regular users see from execution of their solution is more
interesting. Simple answer is of course that they should not see anything which More interesting question is what should regular users see from execution of
is partly true. Outputs from their programs can be anything and users can their solution. Simple answer is of course that they should not see anything
somehow analyse inputs or even redirect them to output. So outputs from which is partly true. Outputs from their programs can be anything and users can
somehow analyze inputs or even redirect them to output. So outputs from
execution should not be visible at all or under very special circumstances. But execution should not be visible at all or under very special circumstances. But
that is not so straighforward for compilation or other kinds of initiation. that is not so straightforward for compilation or other kinds of initiation,
Well, this is another story, it really depends on the particular case. But where it really depends on the particular case. Generally it is quite harmless
generally it is quite harmless to display user some kind of compilation error to display user some kind of compilation error which can help a lot during
which can help a lot during troubleshooting. Of course again this kind of troubleshooting. Of course again this kind of functionality should be
functionality should be configurable by supervisors and disabled by default. configurable by supervisors and disabled by default. There is also the last kind
There is also the last kind of tasks which can output some information which is of tasks which can output some information which is evaluation tasks. Output of
evaluation tasks. Output of these tasks is somehow important to whole system and these tasks is somehow important to whole system and again can contain some
again can contain some information about inputs or reference outputs. So outputs information about inputs or reference outputs. So outputs of evaluation tasks
of evaluation tasks should not be visible to regular users too. should not be visible to regular users too.
The overall concept of grading solutions was presented earlier. To briefly The overall concept of grading solutions was presented earlier. To briefly
remind that, backend returns only exact measured values (used time and memory, remind that, backend returns only exact measured values (used time and memory,
@ -702,19 +703,20 @@ implemented and any sort of magic can return the final value.
We found out several computational possibilities. There is basic arithmetic, We found out several computational possibilities. There is basic arithmetic,
weighted arithmetic, geometric and harmonic mean of results of each test (the weighted arithmetic, geometric and harmonic mean of results of each test (the
result is boolean succeeded/failed, optionaly has weight), some kind of result is logical value succeeded/failed, optionally with weight), some kind of
interpolation of used amount of time for each test, the same with used memory interpolation of used amount of time for each test, the same with used memory
amount and surely many others. To keep the project simple, we decided to design amount and surely many others. To keep the project simple, we decided to design
apropriate interface and implement only weighted arithmetic mean computation, appropriate interface and implement only weighted arithmetic mean computation,
which is used in about 90% of all assignments. Of course, diferent scheme can be which is used in about 90% of all assignments. Of course, different scheme can
chosen for every assignment and also configured -- for example test weights can be chosen for every assignment and also can be configured -- for example
be specified for implemented weighted arithmetic mean. Advanced ways of specifying test weights for implemented weighted arithmetic mean. Advanced ways
computation can be implemented on demand when is a real demand for them. of computation can be implemented on demand when there is a real demand for
them.
To avoid assigning points for insufficient solutions (like only printing "File To avoid assigning points for insufficient solutions (like only printing "File
error" which is the valid answer in two tests), a minimal point threshold can be error" which is the valid answer in two tests), a minimal point threshold can be
specified. It he solution is to get less points than specified, it will get zero specified. It the solution is to get less points than specified, it will get
points instead. This functionality can be empedded into grading computation zero points instead. This functionality can be embedded into grading computation
algoritm itself, but it would have to be present in each implementation algoritm itself, but it would have to be present in each implementation
separately, which is a bit ugly. So, this feature is separated from point separately, which is a bit ugly. So, this feature is separated from point
computation. computation.
@ -722,14 +724,14 @@ computation.
Automatic grading cannot reflect all aspects of submitted code. For example, Automatic grading cannot reflect all aspects of submitted code. For example,
structuring the code, number and quality of comments and so on. To allow structuring the code, number and quality of comments and so on. To allow
supervisors bring these manually checked things into grading, there is a concept supervisors bring these manually checked things into grading, there is a concept
of bonus points. They can be positive or negative. Generaly the solution with of bonus points. They can be positive or negative. Generally the solution with
the most assigned points is marked for grading that particular solution. the most assigned points is marked for grading that particular assignment.
However, if supervisor is not satisfied with student solution (really bad code, However, if supervisor is not satisfied with student solution (really bad code,
cheating, ...) he/she assigns the student negative bonus points. But to prevent cheating, ...) he/she assigns the student negative bonus points. To prevent
chosing another solution with more points by the system or even submitting the overriding this decision by system choosing another solution with more points or
same code again which is worth more points by students, supervisor can mark a even student submitting the same code again which evaluates to more points,
particular solution as marked and used for grading instead of solution with the supervisor can mark a particular solution as marked and used for grading instead
most points. of solution with the most points.
### Persistence ### Persistence
@ -752,19 +754,19 @@ databases or formatted plain files (CSV for example) fits best for them.
However, the data often have to support find operation, so they have to be However, the data often have to support find operation, so they have to be
sorted and allow random access for resolving cross references. Also, addition a sorted and allow random access for resolving cross references. Also, addition a
deletion of entries should take reasonable time (at most logaritmic time deletion of entries should take reasonable time (at most logaritmic time
complexity to number of saved values). This practicaly excludes plain files, so complexity to number of saved values). This practically excludes plain files, so
relational database is used instead. relational database is used instead.
On the other hand, there are some data with no such great structure and much On the other hand, there are some data with no such great structure and much
larger size. These can be evaluation logs, sample input files for exercises or larger size. These can be evaluation logs, sample input files for exercises or
submited sources by students. Saving this kind of data into relational database submitted sources by students. Saving this kind of data into relational database
is not suitable, but it is better to keep them as ordinary files or store them is not suitable, but it is better to keep them as ordinary files or store them
into some kind of NoSQL database. Since they are already files and does not need into some kind of NoSQL database. Since they are already files and does not need
to be backed up in multiple copies, it is easier to keep them as ordinary files to be backed up in multiple copies, it is easier to keep them as ordinary files
in filesystem. Also, this solution is more lightweight and does not require in filesystem. Also, this solution is more lightweight and does not require
additional dependencies on third-party software. File can be identified using additional dependencies on third-party software. File can be identified using
its filesystem path or unique index stored as value in relational database. Both its filesystem path or unique index stored as value in relational database. Both
approaches are equaly good, final decission depends on actual case. approaches are equally good, final decision depends on actual case.
## Structure of the project ## Structure of the project
@ -856,10 +858,10 @@ also possible to develop their own frontend with their own user management
system for their specific needs and use the possibilities of the backend without system for their specific needs and use the possibilities of the backend without
any changes, as was mentioned in the previous paragraphs. any changes, as was mentioned in the previous paragraphs.
One possible configuration of ReCodEx system is ilustrated on following picture, One possible configuration of ReCodEx system is illustrated on following
where thete is one shared backend with three workers and two separate instances picture, where there is one shared backend with three workers and two separate
of whole frontend. This configuration may be suitable for MFF UK -- basic instances of whole frontend. This configuration may be suitable for MFF UK --
programming course and KSP competition. But maybe even sharing web API and basic programming course and KSP competition. But maybe even sharing web API and
fileserver with only custom instances of client (web app or own implementation) fileserver with only custom instances of client (web app or own implementation)
is more likely to be used. Note, that connections between components are not is more likely to be used. Note, that connections between components are not
fully accurate. fully accurate.
@ -1068,7 +1070,7 @@ as network communication which is quite handy and solves problems with threads
synchronization and such. synchronization and such.
At this point we have worker with two internal parts listening one and execution At this point we have worker with two internal parts listening one and execution
one. Implementation of first one is quite straighforward and clear. So lets one. Implementation of first one is quite straightforward and clear. So lets
discuss what should be happening in execution subsystem. Jobs as work units can discuss what should be happening in execution subsystem. Jobs as work units can
quite vary and do completely different things, that means configuration and quite vary and do completely different things, that means configuration and
worker has to be prepared for this kind of generality. Configuration and its worker has to be prepared for this kind of generality. Configuration and its
@ -1342,7 +1344,7 @@ development.
Users want to view real time evaluation progress of their solution. It can be Users want to view real time evaluation progress of their solution. It can be
easily done with established double-sided connection stream, but it is hard to easily done with established double-sided connection stream, but it is hard to
achieve with web technologies. HTTP protocol works differently on separate achieve with web technologies. HTTP protocol works differently on separate
requests basis with no longterm connection. However, there is widely used requests basis with no long term connection. However, there is widely used
technology to solve this problem, WebSocket protocol. technology to solve this problem, WebSocket protocol.
Working with WebSocket protocol from the backend is possible, but not ideal from Working with WebSocket protocol from the backend is possible, but not ideal from
@ -1376,7 +1378,7 @@ following picture.
![Message flow inside montior](https://raw.githubusercontent.com/ReCodEx/wiki/master/images/Monitor_arch.png) ![Message flow inside montior](https://raw.githubusercontent.com/ReCodEx/wiki/master/images/Monitor_arch.png)
The message channel inputing the monitor uses ZeroMQ as main message framework The message channel inputing the monitor uses ZeroMQ as main message framework
used by backend. This decision keeps rest of backend avare of used used by backend. This decision keeps rest of backend aware of used
communication protocol and related libraries. Output channel is WebSocket as a communication protocol and related libraries. Output channel is WebSocket as a
protocol for sending messages to web browsers. In Python, there are several protocol for sending messages to web browsers. In Python, there are several
WebSocket libraries. The most popular one is `websockets` in cooperation with WebSocket libraries. The most popular one is `websockets` in cooperation with
@ -1403,8 +1405,8 @@ We considered several technologies which could be used:
the features we need when some additional extensions are installed (to support the features we need when some additional extensions are installed (to support
LDAP or ZeroMQ). LDAP or ZeroMQ).
- ASP.NET (C#), JSP (Java) -- these technologies are very robust and are used to - ASP.NET (C#), JSP (Java) -- these technologies are very robust and are used to
create server technologies in many big enterpises. Both can run on Windows and create server technologies in many big enterprises. Both can run on Windows
Linux servers (ASP.NET using the .NET Core). and Linux servers (ASP.NET using the .NET Core).
- JavaScript (Node.js) -- it is a quite new technology and it is being used to - JavaScript (Node.js) -- it is a quite new technology and it is being used to
create REST APIs lately. Applications running on Node.js are quite performant create REST APIs lately. Applications running on Node.js are quite performant
and the number of open-source libraries available on the Internet is very and the number of open-source libraries available on the Internet is very

Loading…
Cancel
Save