|
|
|
@ -361,152 +361,138 @@ recruiters and also some universities.
|
|
|
|
|
|
|
|
|
|
# Analysis
|
|
|
|
|
|
|
|
|
|
None of the existing projects we came across fulfills all the requested features
|
|
|
|
|
for the new system. There is no grading system which supports arbitrary-length
|
|
|
|
|
evaluation pipeline, so we have to implement this feature ourselves, cautiously
|
|
|
|
|
treading through unexplored fields. Also, no existing solution is extensible
|
|
|
|
|
enough to be used as a base for the new system. After considering all these
|
|
|
|
|
facts, it is clear that a new system has to be written from scratch. This
|
|
|
|
|
None of the existing projects we came across meets all the features requested
|
|
|
|
|
by the new system. There is no grading system which supports an arbitrary-length
|
|
|
|
|
evaluation pipeline, so we have to implement this feature ourselves.
|
|
|
|
|
No existing solution is extensible enough to be used as a base for the new system.
|
|
|
|
|
After considering all these
|
|
|
|
|
facts, a new system has to be written from scratch. This
|
|
|
|
|
implies that only a subset of all the features will be implemented in the first
|
|
|
|
|
version, and more of them will come in the following releases.
|
|
|
|
|
|
|
|
|
|
Gathered features are categorized based on priorities for the whole system. The
|
|
|
|
|
highest priority is the functionality present in current CodEx. It is a base
|
|
|
|
|
line for being useful in production environment. The design of the new solution
|
|
|
|
|
shall allow extending the system easily. Ideas from faculty staff have secondary
|
|
|
|
|
priority, but most of them will be implemented as part of the project. The most
|
|
|
|
|
complicated tasks from this category are advanced low-level evaluation
|
|
|
|
|
configuration format, using modern tools, connecting to a university systems and
|
|
|
|
|
merging separate system instances into a single one. Other tasks are scheduled
|
|
|
|
|
for next releases after the project is successfully defended. Namely, these are
|
|
|
|
|
high-level exercise evaluation configuration with a user-friendly interface for
|
|
|
|
|
common exercise types, SIS integration (when a public API becomes available for
|
|
|
|
|
the system) and a command-line submission tool. Plagiarism detection is not
|
|
|
|
|
The requested features are categorized based on priorities for the whole system. The
|
|
|
|
|
highest priority is the functionality present in the current CodEx. It is a base
|
|
|
|
|
line for being useful in the production environment. The design of the new solution
|
|
|
|
|
should allow that the system will be extended easily. The ideas from faculty staff
|
|
|
|
|
have lower priority, but most of them will be implemented as part of the project.
|
|
|
|
|
The most complicated tasks from this category are an advanced low-level evaluation
|
|
|
|
|
configuration format, use of modern tools, connection to a university system, and
|
|
|
|
|
combining the currently separate instances into one installation of the system.
|
|
|
|
|
|
|
|
|
|
Other tasks are scheduled
|
|
|
|
|
for the next releases after the first version of the project is completed.
|
|
|
|
|
Namely, these are a high-level exercise evaluation configuration with a
|
|
|
|
|
user-friendly UI, SIS integration (when a public API becomes available for
|
|
|
|
|
the system), and a command-line submission tool. Plagiarism detection is not
|
|
|
|
|
likely to be part of any release in near future unless someone else implements a
|
|
|
|
|
sufficiently capable and extendable engine -- this problem is too complex to be
|
|
|
|
|
sufficiently capable and extendable solution -- this problem is too complex to be
|
|
|
|
|
solved as a part of this project.
|
|
|
|
|
|
|
|
|
|
We named the new project **ReCodEx -- ReCodEx Code Examiner**. The name
|
|
|
|
|
should point to the old CodEx, but also reflect the new approach to solve
|
|
|
|
|
issues. **Re** as part of the name means redesigned, rewritten, renewed, or
|
|
|
|
|
restarted.
|
|
|
|
|
should point to the old CodEx, **Re** as part of the name means
|
|
|
|
|
redesigned, rewritten, renewed, or restarted.
|
|
|
|
|
|
|
|
|
|
At this point there is a clear idea how the new system will be used and what are
|
|
|
|
|
the major enhancements for future releases. With this in mind, the overall
|
|
|
|
|
architecture can be sketched. To sum up, here is a list of key features of the
|
|
|
|
|
new system. They come from previous research of the current drawbacks of the system,
|
|
|
|
|
reasonable wishes of university users and our major design choices.
|
|
|
|
|
At this point there is a clear idea of how the new system will be used and what are
|
|
|
|
|
the major enhancements for the future releases. With this in mind, it is possible
|
|
|
|
|
to sketch the overall architecture. To sum this up, here is a list of key features of the
|
|
|
|
|
new system. They come from the previous research of the current drawbacks of the system,
|
|
|
|
|
reasonable wishes of university users, and our major design choices:
|
|
|
|
|
|
|
|
|
|
- modern HTML5 web frontend written in JavaScript using a suitable framework
|
|
|
|
|
- REST API communicating with database, evaluation backend and a file server
|
|
|
|
|
- REST API communicating with a persistent database, evaluation backend, and a file server
|
|
|
|
|
- evaluation backend implemented as a distributed system on top of a messaging
|
|
|
|
|
framework with master-worker architecture
|
|
|
|
|
- multi-platform worker supporting Linux and Windows environment (latter without
|
|
|
|
|
sandbox, no general purpose suitable tool available yet)
|
|
|
|
|
- evaluation procedure configured in a human readable text file, compound of
|
|
|
|
|
small tasks connected into an arbitrary oriented acyclic graph
|
|
|
|
|
|
|
|
|
|
The reasons supporting these decisions are explained in the rest of analysis
|
|
|
|
|
chapter. Also a lot of smaller design choices are mentioned including possible
|
|
|
|
|
options, what is picked to implement and why. But first, we discuss basic
|
|
|
|
|
concepts of the system.
|
|
|
|
|
framework with a master-worker architecture
|
|
|
|
|
- multi-platform worker supporting Linux and Windows environments (latter without
|
|
|
|
|
a sandbox, no suitable general purpose tool available yet)
|
|
|
|
|
- evaluation procedure configured in a human readable text file, consisting of
|
|
|
|
|
small tasks forming an arbitrary oriented acyclic dependency graph
|
|
|
|
|
|
|
|
|
|
## Basic Concepts
|
|
|
|
|
|
|
|
|
|
The system is designed as a web application. The requirements say that the user
|
|
|
|
|
interface must be accessible for students without the need to install additional
|
|
|
|
|
The requirements specify that the user
|
|
|
|
|
interface must be accessible to students without the need to install additional
|
|
|
|
|
software. This immediately implies that users have to be connected to the
|
|
|
|
|
internet, so it is used as communication medium. Today, there are two main ways
|
|
|
|
|
Internet. Nowadays, there are two main ways
|
|
|
|
|
of designing graphical user interfaces -- as a native application or a web page.
|
|
|
|
|
Creating a user-friendly and multi-platform application with graphical interface
|
|
|
|
|
is almost impossible because of the large number of different environments.
|
|
|
|
|
Also, these applications typically require installation or at least downloading
|
|
|
|
|
its files (sources or binaries). On the other hand, distributing a web
|
|
|
|
|
Creating a user-friendly and multi-platform application with graphical UI
|
|
|
|
|
is almost impossible because of the large number of different operating systems.
|
|
|
|
|
These applications typically require installation or at least downloading
|
|
|
|
|
its files (source codes or binaries). On the other hand, distributing a web
|
|
|
|
|
application is easier, because every personal computer has an internet browser
|
|
|
|
|
installed. Also, browsers support a (mostly) unified and standardized
|
|
|
|
|
installed. Browsers support a (mostly) unified and standardized
|
|
|
|
|
environment of HTML5 and JavaScript. CodEx is also a web application and
|
|
|
|
|
everybody seems satisfied with this fact. There are other communicating channels
|
|
|
|
|
everybody seems to be satisfied with this fact. There are other communicating channels
|
|
|
|
|
most programmers use, such as e-mail or git, but they are inappropriate for
|
|
|
|
|
designing user interfaces on top of them.
|
|
|
|
|
|
|
|
|
|
The application interacts with users. From the project assignment it is clear
|
|
|
|
|
that the system has to keep personalized data about users and adapt presented
|
|
|
|
|
content according to this knowledge. User data cannot be publicly visible, which
|
|
|
|
|
implies necessity of user authentication. The application also has to support
|
|
|
|
|
multiple ways of authentication (university authentication systems, a company
|
|
|
|
|
LDAP server, an OAuth server...) and permit adding more security measures in the
|
|
|
|
|
It is clear from the assignment of the project
|
|
|
|
|
that the system has to keep personalized data of the users. User data cannot
|
|
|
|
|
be publicly available, which implies necessity of user authentication.
|
|
|
|
|
The application also has to support
|
|
|
|
|
multiple ways of authentication (e.g., university authentication systems, a company
|
|
|
|
|
LDAP server, an OAuth server), and permit adding more security measures in the
|
|
|
|
|
future, such as two-factor authentication.
|
|
|
|
|
|
|
|
|
|
User data also include a privilege level. From the assignment it is required to
|
|
|
|
|
have at least two roles, _student_ and _supervisor_. However, it is wise to add
|
|
|
|
|
_administrator_ level for users who take care of the system as a whole and is
|
|
|
|
|
responsible for core setup, monitoring, updates and so on. Student role has the
|
|
|
|
|
least power, basically can just view assignments and submit solutions.
|
|
|
|
|
Supervisors have more authority, so they can create exercises and assignments,
|
|
|
|
|
view results of students etc. From the university organization, one possible
|
|
|
|
|
Each user has a specific role in the system. From the assignment it is required to
|
|
|
|
|
have at least two such roles, _student_ and _supervisor_. However, it is advisible
|
|
|
|
|
to add an _administrator_ level for users who take care of the system as a whole and are
|
|
|
|
|
responsible for the setup, monitoring, or updates. The student role has the
|
|
|
|
|
minimum access rights, basically a student can only view assignments and submit solutions.
|
|
|
|
|
Supervisors have more authority, so they can create exercises and assignments, and
|
|
|
|
|
view results of their students. From the organization of the university, one possible
|
|
|
|
|
level could be introduced, _course guarantor_. However, from real experience all
|
|
|
|
|
duties related with lecturing of labs are already associated with supervisors,
|
|
|
|
|
so this role does not seem useful. In addition, no one requested more than three
|
|
|
|
|
so this role does not seem useful. In addition, no one requested more than a three
|
|
|
|
|
level privilege scheme.
|
|
|
|
|
|
|
|
|
|
School labs are lessons for some students lead by supervisors. All students in a
|
|
|
|
|
lab have the same homework and supervisors evaluate their solutions. This
|
|
|
|
|
organization has to be carried over into the new system. Virtual groups are a
|
|
|
|
|
counterpart to real-life labs. This concept was already discussed in the
|
|
|
|
|
previous chapter including the need for a hierarchical structure of groups. Only
|
|
|
|
|
students of the university recorded in the university information system have a
|
|
|
|
|
right to attend labs.
|
|
|
|
|
arrangement has to be transferred into the new system. The groups in the system
|
|
|
|
|
correspond to the real-life labs. This concept was already discussed in the
|
|
|
|
|
previous chapter including the need for a hierarchical structure of the groups.
|
|
|
|
|
|
|
|
|
|
To allow restriction of group members in ReCodEx, there are two types of groups
|
|
|
|
|
-- _public_ and _private_. Public groups are open for all registered users, but
|
|
|
|
|
to become a member of private group, one of its supervisors have to add the
|
|
|
|
|
user. This could be done automatically at the beginning of the term with data
|
|
|
|
|
from information system, but unfortunately there is no API for this yet.
|
|
|
|
|
However, creating this API is now being considered by university staff. Another
|
|
|
|
|
just as good solution for restricting membership of a group is to allow anyone
|
|
|
|
|
join the group with supplementary confirmation of supervisors. It has no
|
|
|
|
|
additional benefits, so approach with public and private groups is implemented.
|
|
|
|
|
|
|
|
|
|
Supervisors using CodEx in their labs usually set minimum amount of points
|
|
|
|
|
to become a member of a private group, one of its supervisors has to add the
|
|
|
|
|
user to the group. This could be done automatically at the beginning of a term with data
|
|
|
|
|
from the information system, but unfortunately there is no API for this yet.
|
|
|
|
|
However, creating this API is now being considered by university staff.
|
|
|
|
|
|
|
|
|
|
Supervisors using CodEx in their labs usually set a minimum amount of points
|
|
|
|
|
required to get a credit. These points can be acquired by solving assigned
|
|
|
|
|
exercises. To show users whether they already have enough points, ReCodEx also
|
|
|
|
|
supports setting this limit for individual groups. There are two equal ways how
|
|
|
|
|
to set a limit -- absolute value or relative value to maximum. The latter way
|
|
|
|
|
seems nicer, so it is implemented. The relative value is set in percents and is
|
|
|
|
|
called threshold.
|
|
|
|
|
supports setting this limit for the groups. There are two equal ways of setting
|
|
|
|
|
a limit -- an absolute number fo points or a percentage of the total possible
|
|
|
|
|
number of points. We decided to implement the latter of these possibilities
|
|
|
|
|
and we call it the threshold.
|
|
|
|
|
|
|
|
|
|
Our university has a few partner grammar schools. There was an idea, that they
|
|
|
|
|
could use CodEx for teaching informatics classes. To make the setup simple for
|
|
|
|
|
Our university has a few partners among grammar schools. There was an idea, that they
|
|
|
|
|
could use CodEx for teaching IT classes. To simplify the setup for
|
|
|
|
|
them, all the software and hardware would be provided by the university as a
|
|
|
|
|
completely ready-to-use remote service. However, CodEx is not prepared for this
|
|
|
|
|
kind of usage and no one has the time to manage a separate instance. With
|
|
|
|
|
ReCodEx it is possible to offer hosted environment as a service to other
|
|
|
|
|
subjects.
|
|
|
|
|
|
|
|
|
|
The concept we came up with is based on user and group separation inside the
|
|
|
|
|
system. The system is divided into multiple separated units called _instances_.
|
|
|
|
|
Each instance has own set of users and groups. Exercises can be optionally
|
|
|
|
|
shared. The rest of the system (API server and evaluation backend) is shared
|
|
|
|
|
between the instances. To keep track of active instances and paying customers,
|
|
|
|
|
each instance must have a valid _licence_ to allow users submit their solutions.
|
|
|
|
|
licence is granted for a definite period of time and can be revoked in advance
|
|
|
|
|
SaaS. However, CodEx is not prepared for this
|
|
|
|
|
kind of usage and no one has the time to manage another separate instance. With
|
|
|
|
|
ReCodEx it is possible to offer a hosted environment as a service to other
|
|
|
|
|
subjects.
|
|
|
|
|
|
|
|
|
|
The system is divided into multiple separate units called _instances_.
|
|
|
|
|
Each instance has its own set of users and groups. Exercises can be optionally
|
|
|
|
|
shared. The rest of the system (the API server and the evaluation backend) is shared
|
|
|
|
|
between the instances. To keep a track of the active instances and allow access
|
|
|
|
|
to the infrastructure to other, paying, customers, each instance must have a
|
|
|
|
|
valid _licence_ to allow its users to submit their solutions.
|
|
|
|
|
Each licence is granted for a specific period of time and can be revoked in advance
|
|
|
|
|
if the subject does not conform with approved terms and conditions.
|
|
|
|
|
|
|
|
|
|
The primary task of the system is to evaluate programming exercises. An
|
|
|
|
|
exercise is quite similar to a homework assignment during school labs. When a
|
|
|
|
|
homework is assigned, two things are important to know for users:
|
|
|
|
|
|
|
|
|
|
- description of the problem
|
|
|
|
|
- metadata -- when and whom to submit solutions, grading scale, penalties, etc.
|
|
|
|
|
|
|
|
|
|
To reflect this idea teachers and students are already familiar with, we decided
|
|
|
|
|
to keep separation between problem itself (_exercise_) and its _assignment_.
|
|
|
|
|
Exercise only describes one problem and provides testing data with description
|
|
|
|
|
of how to evaluate it. In fact, it is a template for assignments. Assignment
|
|
|
|
|
then contains data from its exercise and additional metadata, which can be
|
|
|
|
|
different for every assignment of the same exercise. This separation is natural
|
|
|
|
|
for all users, in CodEx it is implemented in similar way and no other
|
|
|
|
|
considerable solution was found.
|
|
|
|
|
The problems the students solve are broken down into two parts in the system:
|
|
|
|
|
|
|
|
|
|
- the problem itself (an _exercise_),
|
|
|
|
|
- and its _assignment_
|
|
|
|
|
|
|
|
|
|
Exercises only describe the problem and provide testing data with the description
|
|
|
|
|
of how to evaluate them. In fact, these are templates for the assignments. A particular
|
|
|
|
|
assignment then contains data from the exercise and some additional metadata, which can be
|
|
|
|
|
different for every assignment of the same exercise (e.g., the deadline, maximum number
|
|
|
|
|
of points).
|
|
|
|
|
|
|
|
|
|
### Evaluation Unit Executed by ReCodEx
|
|
|
|
|
|
|
|
|
|