From abec266d97d298b4f6d4bb6b8dbd3395cb41408b Mon Sep 17 00:00:00 2001 From: Simon Rozsival Date: Sat, 4 Feb 2017 17:22:55 +0100 Subject: [PATCH] correction of the first part of the analysis --- Rewritten-docs.md | 202 +++++++++++++++++++++------------------------- 1 file changed, 94 insertions(+), 108 deletions(-) diff --git a/Rewritten-docs.md b/Rewritten-docs.md index c26d7cd..7d0eeae 100644 --- a/Rewritten-docs.md +++ b/Rewritten-docs.md @@ -361,152 +361,138 @@ recruiters and also some universities. # Analysis -None of the existing projects we came across fulfills all the requested features -for the new system. There is no grading system which supports arbitrary-length -evaluation pipeline, so we have to implement this feature ourselves, cautiously -treading through unexplored fields. Also, no existing solution is extensible -enough to be used as a base for the new system. After considering all these -facts, it is clear that a new system has to be written from scratch. This +None of the existing projects we came across meets all the features requested +by the new system. There is no grading system which supports an arbitrary-length +evaluation pipeline, so we have to implement this feature ourselves. +No existing solution is extensible enough to be used as a base for the new system. +After considering all these +facts, a new system has to be written from scratch. This implies that only a subset of all the features will be implemented in the first version, and more of them will come in the following releases. -Gathered features are categorized based on priorities for the whole system. The -highest priority is the functionality present in current CodEx. It is a base -line for being useful in production environment. The design of the new solution -shall allow extending the system easily. Ideas from faculty staff have secondary -priority, but most of them will be implemented as part of the project. The most -complicated tasks from this category are advanced low-level evaluation -configuration format, using modern tools, connecting to a university systems and -merging separate system instances into a single one. Other tasks are scheduled -for next releases after the project is successfully defended. Namely, these are -high-level exercise evaluation configuration with a user-friendly interface for -common exercise types, SIS integration (when a public API becomes available for -the system) and a command-line submission tool. Plagiarism detection is not +The requested features are categorized based on priorities for the whole system. The +highest priority is the functionality present in the current CodEx. It is a base +line for being useful in the production environment. The design of the new solution +should allow that the system will be extended easily. The ideas from faculty staff +have lower priority, but most of them will be implemented as part of the project. +The most complicated tasks from this category are an advanced low-level evaluation +configuration format, use of modern tools, connection to a university system, and +combining the currently separate instances into one installation of the system. + +Other tasks are scheduled +for the next releases after the first version of the project is completed. +Namely, these are a high-level exercise evaluation configuration with a +user-friendly UI, SIS integration (when a public API becomes available for +the system), and a command-line submission tool. Plagiarism detection is not likely to be part of any release in near future unless someone else implements a -sufficiently capable and extendable engine -- this problem is too complex to be +sufficiently capable and extendable solution -- this problem is too complex to be solved as a part of this project. We named the new project **ReCodEx -- ReCodEx Code Examiner**. The name -should point to the old CodEx, but also reflect the new approach to solve -issues. **Re** as part of the name means redesigned, rewritten, renewed, or -restarted. +should point to the old CodEx, **Re** as part of the name means +redesigned, rewritten, renewed, or restarted. -At this point there is a clear idea how the new system will be used and what are -the major enhancements for future releases. With this in mind, the overall -architecture can be sketched. To sum up, here is a list of key features of the -new system. They come from previous research of the current drawbacks of the system, -reasonable wishes of university users and our major design choices. +At this point there is a clear idea of how the new system will be used and what are +the major enhancements for the future releases. With this in mind, it is possible +to sketch the overall architecture. To sum this up, here is a list of key features of the +new system. They come from the previous research of the current drawbacks of the system, +reasonable wishes of university users, and our major design choices: - modern HTML5 web frontend written in JavaScript using a suitable framework -- REST API communicating with database, evaluation backend and a file server +- REST API communicating with a persistent database, evaluation backend, and a file server - evaluation backend implemented as a distributed system on top of a messaging - framework with master-worker architecture -- multi-platform worker supporting Linux and Windows environment (latter without - sandbox, no general purpose suitable tool available yet) -- evaluation procedure configured in a human readable text file, compound of - small tasks connected into an arbitrary oriented acyclic graph - -The reasons supporting these decisions are explained in the rest of analysis -chapter. Also a lot of smaller design choices are mentioned including possible -options, what is picked to implement and why. But first, we discuss basic -concepts of the system. + framework with a master-worker architecture +- multi-platform worker supporting Linux and Windows environments (latter without + a sandbox, no suitable general purpose tool available yet) +- evaluation procedure configured in a human readable text file, consisting of + small tasks forming an arbitrary oriented acyclic dependency graph ## Basic Concepts -The system is designed as a web application. The requirements say that the user -interface must be accessible for students without the need to install additional +The requirements specify that the user +interface must be accessible to students without the need to install additional software. This immediately implies that users have to be connected to the -internet, so it is used as communication medium. Today, there are two main ways +Internet. Nowadays, there are two main ways of designing graphical user interfaces -- as a native application or a web page. -Creating a user-friendly and multi-platform application with graphical interface -is almost impossible because of the large number of different environments. -Also, these applications typically require installation or at least downloading -its files (sources or binaries). On the other hand, distributing a web +Creating a user-friendly and multi-platform application with graphical UI +is almost impossible because of the large number of different operating systems. +These applications typically require installation or at least downloading +its files (source codes or binaries). On the other hand, distributing a web application is easier, because every personal computer has an internet browser -installed. Also, browsers support a (mostly) unified and standardized +installed. Browsers support a (mostly) unified and standardized environment of HTML5 and JavaScript. CodEx is also a web application and -everybody seems satisfied with this fact. There are other communicating channels +everybody seems to be satisfied with this fact. There are other communicating channels most programmers use, such as e-mail or git, but they are inappropriate for designing user interfaces on top of them. -The application interacts with users. From the project assignment it is clear -that the system has to keep personalized data about users and adapt presented -content according to this knowledge. User data cannot be publicly visible, which -implies necessity of user authentication. The application also has to support -multiple ways of authentication (university authentication systems, a company -LDAP server, an OAuth server...) and permit adding more security measures in the +It is clear from the assignment of the project +that the system has to keep personalized data of the users. User data cannot +be publicly available, which implies necessity of user authentication. +The application also has to support +multiple ways of authentication (e.g., university authentication systems, a company +LDAP server, an OAuth server), and permit adding more security measures in the future, such as two-factor authentication. -User data also include a privilege level. From the assignment it is required to -have at least two roles, _student_ and _supervisor_. However, it is wise to add -_administrator_ level for users who take care of the system as a whole and is -responsible for core setup, monitoring, updates and so on. Student role has the -least power, basically can just view assignments and submit solutions. -Supervisors have more authority, so they can create exercises and assignments, -view results of students etc. From the university organization, one possible +Each user has a specific role in the system. From the assignment it is required to +have at least two such roles, _student_ and _supervisor_. However, it is advisible +to add an _administrator_ level for users who take care of the system as a whole and are +responsible for the setup, monitoring, or updates. The student role has the +minimum access rights, basically a student can only view assignments and submit solutions. +Supervisors have more authority, so they can create exercises and assignments, and +view results of their students. From the organization of the university, one possible level could be introduced, _course guarantor_. However, from real experience all duties related with lecturing of labs are already associated with supervisors, -so this role does not seem useful. In addition, no one requested more than three +so this role does not seem useful. In addition, no one requested more than a three level privilege scheme. School labs are lessons for some students lead by supervisors. All students in a lab have the same homework and supervisors evaluate their solutions. This -organization has to be carried over into the new system. Virtual groups are a -counterpart to real-life labs. This concept was already discussed in the -previous chapter including the need for a hierarchical structure of groups. Only -students of the university recorded in the university information system have a -right to attend labs. +arrangement has to be transferred into the new system. The groups in the system +correspond to the real-life labs. This concept was already discussed in the +previous chapter including the need for a hierarchical structure of the groups. To allow restriction of group members in ReCodEx, there are two types of groups -- _public_ and _private_. Public groups are open for all registered users, but -to become a member of private group, one of its supervisors have to add the -user. This could be done automatically at the beginning of the term with data -from information system, but unfortunately there is no API for this yet. -However, creating this API is now being considered by university staff. Another -just as good solution for restricting membership of a group is to allow anyone -join the group with supplementary confirmation of supervisors. It has no -additional benefits, so approach with public and private groups is implemented. - -Supervisors using CodEx in their labs usually set minimum amount of points +to become a member of a private group, one of its supervisors has to add the +user to the group. This could be done automatically at the beginning of a term with data +from the information system, but unfortunately there is no API for this yet. +However, creating this API is now being considered by university staff. + +Supervisors using CodEx in their labs usually set a minimum amount of points required to get a credit. These points can be acquired by solving assigned exercises. To show users whether they already have enough points, ReCodEx also -supports setting this limit for individual groups. There are two equal ways how -to set a limit -- absolute value or relative value to maximum. The latter way -seems nicer, so it is implemented. The relative value is set in percents and is -called threshold. +supports setting this limit for the groups. There are two equal ways of setting +a limit -- an absolute number fo points or a percentage of the total possible +number of points. We decided to implement the latter of these possibilities +and we call it the threshold. -Our university has a few partner grammar schools. There was an idea, that they -could use CodEx for teaching informatics classes. To make the setup simple for +Our university has a few partners among grammar schools. There was an idea, that they +could use CodEx for teaching IT classes. To simplify the setup for them, all the software and hardware would be provided by the university as a -completely ready-to-use remote service. However, CodEx is not prepared for this -kind of usage and no one has the time to manage a separate instance. With -ReCodEx it is possible to offer hosted environment as a service to other -subjects. - -The concept we came up with is based on user and group separation inside the -system. The system is divided into multiple separated units called _instances_. -Each instance has own set of users and groups. Exercises can be optionally -shared. The rest of the system (API server and evaluation backend) is shared -between the instances. To keep track of active instances and paying customers, -each instance must have a valid _licence_ to allow users submit their solutions. -licence is granted for a definite period of time and can be revoked in advance +SaaS. However, CodEx is not prepared for this +kind of usage and no one has the time to manage another separate instance. With +ReCodEx it is possible to offer a hosted environment as a service to other +subjects. + +The system is divided into multiple separate units called _instances_. +Each instance has its own set of users and groups. Exercises can be optionally +shared. The rest of the system (the API server and the evaluation backend) is shared +between the instances. To keep a track of the active instances and allow access +to the infrastructure to other, paying, customers, each instance must have a +valid _licence_ to allow its users to submit their solutions. +Each licence is granted for a specific period of time and can be revoked in advance if the subject does not conform with approved terms and conditions. -The primary task of the system is to evaluate programming exercises. An -exercise is quite similar to a homework assignment during school labs. When a -homework is assigned, two things are important to know for users: - -- description of the problem -- metadata -- when and whom to submit solutions, grading scale, penalties, etc. - -To reflect this idea teachers and students are already familiar with, we decided -to keep separation between problem itself (_exercise_) and its _assignment_. -Exercise only describes one problem and provides testing data with description -of how to evaluate it. In fact, it is a template for assignments. Assignment -then contains data from its exercise and additional metadata, which can be -different for every assignment of the same exercise. This separation is natural -for all users, in CodEx it is implemented in similar way and no other -considerable solution was found. +The problems the students solve are broken down into two parts in the system: + +- the problem itself (an _exercise_), +- and its _assignment_ + +Exercises only describe the problem and provide testing data with the description +of how to evaluate them. In fact, these are templates for the assignments. A particular +assignment then contains data from the exercise and some additional metadata, which can be +different for every assignment of the same exercise (e.g., the deadline, maximum number +of points). ### Evaluation Unit Executed by ReCodEx