correction of the first part of the analysis

master
Simon Rozsival 8 years ago
parent af0c09586d
commit abec266d97

@ -361,152 +361,138 @@ recruiters and also some universities.
# Analysis
None of the existing projects we came across fulfills all the requested features
for the new system. There is no grading system which supports arbitrary-length
evaluation pipeline, so we have to implement this feature ourselves, cautiously
treading through unexplored fields. Also, no existing solution is extensible
enough to be used as a base for the new system. After considering all these
facts, it is clear that a new system has to be written from scratch. This
None of the existing projects we came across meets all the features requested
by the new system. There is no grading system which supports an arbitrary-length
evaluation pipeline, so we have to implement this feature ourselves.
No existing solution is extensible enough to be used as a base for the new system.
After considering all these
facts, a new system has to be written from scratch. This
implies that only a subset of all the features will be implemented in the first
version, and more of them will come in the following releases.
Gathered features are categorized based on priorities for the whole system. The
highest priority is the functionality present in current CodEx. It is a base
line for being useful in production environment. The design of the new solution
shall allow extending the system easily. Ideas from faculty staff have secondary
priority, but most of them will be implemented as part of the project. The most
complicated tasks from this category are advanced low-level evaluation
configuration format, using modern tools, connecting to a university systems and
merging separate system instances into a single one. Other tasks are scheduled
for next releases after the project is successfully defended. Namely, these are
high-level exercise evaluation configuration with a user-friendly interface for
common exercise types, SIS integration (when a public API becomes available for
the system) and a command-line submission tool. Plagiarism detection is not
The requested features are categorized based on priorities for the whole system. The
highest priority is the functionality present in the current CodEx. It is a base
line for being useful in the production environment. The design of the new solution
should allow that the system will be extended easily. The ideas from faculty staff
have lower priority, but most of them will be implemented as part of the project.
The most complicated tasks from this category are an advanced low-level evaluation
configuration format, use of modern tools, connection to a university system, and
combining the currently separate instances into one installation of the system.
Other tasks are scheduled
for the next releases after the first version of the project is completed.
Namely, these are a high-level exercise evaluation configuration with a
user-friendly UI, SIS integration (when a public API becomes available for
the system), and a command-line submission tool. Plagiarism detection is not
likely to be part of any release in near future unless someone else implements a
sufficiently capable and extendable engine -- this problem is too complex to be
sufficiently capable and extendable solution -- this problem is too complex to be
solved as a part of this project.
We named the new project **ReCodEx -- ReCodEx Code Examiner**. The name
should point to the old CodEx, but also reflect the new approach to solve
issues. **Re** as part of the name means redesigned, rewritten, renewed, or
restarted.
should point to the old CodEx, **Re** as part of the name means
redesigned, rewritten, renewed, or restarted.
At this point there is a clear idea how the new system will be used and what are
the major enhancements for future releases. With this in mind, the overall
architecture can be sketched. To sum up, here is a list of key features of the
new system. They come from previous research of the current drawbacks of the system,
reasonable wishes of university users and our major design choices.
At this point there is a clear idea of how the new system will be used and what are
the major enhancements for the future releases. With this in mind, it is possible
to sketch the overall architecture. To sum this up, here is a list of key features of the
new system. They come from the previous research of the current drawbacks of the system,
reasonable wishes of university users, and our major design choices:
- modern HTML5 web frontend written in JavaScript using a suitable framework
- REST API communicating with database, evaluation backend and a file server
- REST API communicating with a persistent database, evaluation backend, and a file server
- evaluation backend implemented as a distributed system on top of a messaging
framework with master-worker architecture
- multi-platform worker supporting Linux and Windows environment (latter without
sandbox, no general purpose suitable tool available yet)
- evaluation procedure configured in a human readable text file, compound of
small tasks connected into an arbitrary oriented acyclic graph
The reasons supporting these decisions are explained in the rest of analysis
chapter. Also a lot of smaller design choices are mentioned including possible
options, what is picked to implement and why. But first, we discuss basic
concepts of the system.
framework with a master-worker architecture
- multi-platform worker supporting Linux and Windows environments (latter without
a sandbox, no suitable general purpose tool available yet)
- evaluation procedure configured in a human readable text file, consisting of
small tasks forming an arbitrary oriented acyclic dependency graph
## Basic Concepts
The system is designed as a web application. The requirements say that the user
interface must be accessible for students without the need to install additional
The requirements specify that the user
interface must be accessible to students without the need to install additional
software. This immediately implies that users have to be connected to the
internet, so it is used as communication medium. Today, there are two main ways
Internet. Nowadays, there are two main ways
of designing graphical user interfaces -- as a native application or a web page.
Creating a user-friendly and multi-platform application with graphical interface
is almost impossible because of the large number of different environments.
Also, these applications typically require installation or at least downloading
its files (sources or binaries). On the other hand, distributing a web
Creating a user-friendly and multi-platform application with graphical UI
is almost impossible because of the large number of different operating systems.
These applications typically require installation or at least downloading
its files (source codes or binaries). On the other hand, distributing a web
application is easier, because every personal computer has an internet browser
installed. Also, browsers support a (mostly) unified and standardized
installed. Browsers support a (mostly) unified and standardized
environment of HTML5 and JavaScript. CodEx is also a web application and
everybody seems satisfied with this fact. There are other communicating channels
everybody seems to be satisfied with this fact. There are other communicating channels
most programmers use, such as e-mail or git, but they are inappropriate for
designing user interfaces on top of them.
The application interacts with users. From the project assignment it is clear
that the system has to keep personalized data about users and adapt presented
content according to this knowledge. User data cannot be publicly visible, which
implies necessity of user authentication. The application also has to support
multiple ways of authentication (university authentication systems, a company
LDAP server, an OAuth server...) and permit adding more security measures in the
It is clear from the assignment of the project
that the system has to keep personalized data of the users. User data cannot
be publicly available, which implies necessity of user authentication.
The application also has to support
multiple ways of authentication (e.g., university authentication systems, a company
LDAP server, an OAuth server), and permit adding more security measures in the
future, such as two-factor authentication.
User data also include a privilege level. From the assignment it is required to
have at least two roles, _student_ and _supervisor_. However, it is wise to add
_administrator_ level for users who take care of the system as a whole and is
responsible for core setup, monitoring, updates and so on. Student role has the
least power, basically can just view assignments and submit solutions.
Supervisors have more authority, so they can create exercises and assignments,
view results of students etc. From the university organization, one possible
Each user has a specific role in the system. From the assignment it is required to
have at least two such roles, _student_ and _supervisor_. However, it is advisible
to add an _administrator_ level for users who take care of the system as a whole and are
responsible for the setup, monitoring, or updates. The student role has the
minimum access rights, basically a student can only view assignments and submit solutions.
Supervisors have more authority, so they can create exercises and assignments, and
view results of their students. From the organization of the university, one possible
level could be introduced, _course guarantor_. However, from real experience all
duties related with lecturing of labs are already associated with supervisors,
so this role does not seem useful. In addition, no one requested more than three
so this role does not seem useful. In addition, no one requested more than a three
level privilege scheme.
School labs are lessons for some students lead by supervisors. All students in a
lab have the same homework and supervisors evaluate their solutions. This
organization has to be carried over into the new system. Virtual groups are a
counterpart to real-life labs. This concept was already discussed in the
previous chapter including the need for a hierarchical structure of groups. Only
students of the university recorded in the university information system have a
right to attend labs.
arrangement has to be transferred into the new system. The groups in the system
correspond to the real-life labs. This concept was already discussed in the
previous chapter including the need for a hierarchical structure of the groups.
To allow restriction of group members in ReCodEx, there are two types of groups
-- _public_ and _private_. Public groups are open for all registered users, but
to become a member of private group, one of its supervisors have to add the
user. This could be done automatically at the beginning of the term with data
from information system, but unfortunately there is no API for this yet.
However, creating this API is now being considered by university staff. Another
just as good solution for restricting membership of a group is to allow anyone
join the group with supplementary confirmation of supervisors. It has no
additional benefits, so approach with public and private groups is implemented.
Supervisors using CodEx in their labs usually set minimum amount of points
to become a member of a private group, one of its supervisors has to add the
user to the group. This could be done automatically at the beginning of a term with data
from the information system, but unfortunately there is no API for this yet.
However, creating this API is now being considered by university staff.
Supervisors using CodEx in their labs usually set a minimum amount of points
required to get a credit. These points can be acquired by solving assigned
exercises. To show users whether they already have enough points, ReCodEx also
supports setting this limit for individual groups. There are two equal ways how
to set a limit -- absolute value or relative value to maximum. The latter way
seems nicer, so it is implemented. The relative value is set in percents and is
called threshold.
supports setting this limit for the groups. There are two equal ways of setting
a limit -- an absolute number fo points or a percentage of the total possible
number of points. We decided to implement the latter of these possibilities
and we call it the threshold.
Our university has a few partner grammar schools. There was an idea, that they
could use CodEx for teaching informatics classes. To make the setup simple for
Our university has a few partners among grammar schools. There was an idea, that they
could use CodEx for teaching IT classes. To simplify the setup for
them, all the software and hardware would be provided by the university as a
completely ready-to-use remote service. However, CodEx is not prepared for this
kind of usage and no one has the time to manage a separate instance. With
ReCodEx it is possible to offer hosted environment as a service to other
subjects.
The concept we came up with is based on user and group separation inside the
system. The system is divided into multiple separated units called _instances_.
Each instance has own set of users and groups. Exercises can be optionally
shared. The rest of the system (API server and evaluation backend) is shared
between the instances. To keep track of active instances and paying customers,
each instance must have a valid _licence_ to allow users submit their solutions.
licence is granted for a definite period of time and can be revoked in advance
SaaS. However, CodEx is not prepared for this
kind of usage and no one has the time to manage another separate instance. With
ReCodEx it is possible to offer a hosted environment as a service to other
subjects.
The system is divided into multiple separate units called _instances_.
Each instance has its own set of users and groups. Exercises can be optionally
shared. The rest of the system (the API server and the evaluation backend) is shared
between the instances. To keep a track of the active instances and allow access
to the infrastructure to other, paying, customers, each instance must have a
valid _licence_ to allow its users to submit their solutions.
Each licence is granted for a specific period of time and can be revoked in advance
if the subject does not conform with approved terms and conditions.
The primary task of the system is to evaluate programming exercises. An
exercise is quite similar to a homework assignment during school labs. When a
homework is assigned, two things are important to know for users:
- description of the problem
- metadata -- when and whom to submit solutions, grading scale, penalties, etc.
To reflect this idea teachers and students are already familiar with, we decided
to keep separation between problem itself (_exercise_) and its _assignment_.
Exercise only describes one problem and provides testing data with description
of how to evaluate it. In fact, it is a template for assignments. Assignment
then contains data from its exercise and additional metadata, which can be
different for every assignment of the same exercise. This separation is natural
for all users, in CodEx it is implemented in similar way and no other
considerable solution was found.
The problems the students solve are broken down into two parts in the system:
- the problem itself (an _exercise_),
- and its _assignment_
Exercises only describe the problem and provide testing data with the description
of how to evaluate them. In fact, these are templates for the assignments. A particular
assignment then contains data from the exercise and some additional metadata, which can be
different for every assignment of the same exercise (e.g., the deadline, maximum number
of points).
### Evaluation Unit Executed by ReCodEx

Loading…
Cancel
Save