recodex-wiki/Web-API.md

Web API

Description

The Web API provides a controlled access to the evaluation backend. It also enables the use of different user frontends such as default web application, mobile applications, commandline tools and possibly others. The communication goes as HTTP(S) requests in predefined format, nowadays mostly known as REST format. Results from the API are in plain text in JSON format to be easily parsed in various languages (notably JavaScript).

This component must be publicly visible on the internet, so it is important to care about security and follow our recommendations. Security and user access restriction are among our primary concerns, so proper roles with permission separation are introduced and maintained. Also some additional checks are made directly in the code, so that a user cannot access information which are out of their authorization.

Architecture

Web API is written in PHP using Nette framework. This framework provides useful components like Tracy for logging and showing errors, Tester for productive unit testing or Latte templating engine. Nette is modern, widely used and great performing software with active developers and user community. Nette can help eliminate security holes, simplify debugging and make coding easier with numerous plugins and extensions. Also, it is published under permissive BSD license.

API architecture consists of several parts:

• router -- component handling mapping from URL addresses to methods in presenter classes (called endpoints)
• presenters -- classes containing one method per endpoint responsible for fetching and parsing request arguments and performing desired actions
• entities -- classes persisted using a database with an ORM framework
• repositories -- common operations on entities of one type, mostly finding entity by identifier or persisting changes to the database
• helpers -- set of classes solving more complicated internal logic, used from presenters to keep them reasonably small

Each presenter method has several annotations. They are used for generating REST API documentation in Swagger, specifying request type and its parameters and specifying one level of access restrictions. Also, there is simple description of the endpoint.

For specifying the request type (GET, POST, DELETE) annotations with exactly these names without any parameters are used. To describe request parameters @Param annotation is used with following arguments:

• type -- the type of argument, one of post of query
• name -- name of the argument (the key)
• validation -- validation of the value, see Nette validation rules
• msg -- description for users about the values this parameter can contain
• required -- specifies if this option is mandatory (true, default) or optional (false)
• description -- description for documentation of the API

Another annotation is @LoggedIn which takes no arguments. It can be placed before a whole class or before a method, so requests from unauthorized users are forbidden. Permissions can be granted or prohibitted by @UserIsAllowed annotation. This one is only per method and takes one argument in key="value" format. The value specifies which action (value) of a resource (key) the user needs to be allowed to perform this request. An example of how an annotated endpoint can look like:

/**
 * Create a user account
 * @POST
 * @LoggedIn
 * @UserIsAllowed(users="create")
 * @Param(type="post", name="email", validation="email",
 *    description="An email that will serve as a login name")
 * @Param(type="post", name="name", validation="string:2..",
 *    description="First name")
 * @Param(type="post", name="password", validation="string:1..",
 *    msg="Password cannot be empty.",
 *    description="A password for authentication")
 */
public function actionCreateAccount() {
    ...
}

The Doctrine ORM framework is used as an object persistence layer. It provides simple to use annotations to specify columns of database tables including types, indexes and also it is possible to make mapping between entities. For detailed info refer to official documentation.

The API is capable of sending email messages. They can inform an administrator about errors and users about submission evaluation or a temporary link to change forgotten password. The Nette Mail extension provides nice interface for sending messages through external SMTP server (preferred) or builtin PHP function mail. It is important to set up the mailserver properly to ensure message delivery to the clients. The messages are rendered in HTML format via simple Latte templates.

Authentication

Instead of relying on PHP sessions, we decided to use an authentication flow based on JWT tokens (RFC 7519). On successful login, the user is issued an access token that they have to send with subsequent requests using the HTTP Authorization header (Authorization: Bearer <token>). The token has a limited validity period and has to be renewed periodically using a dedicated API endpoint.

To implement this behavior in Nette framework, a new IUserStorage implementation was created, along with an IIdentity and authenticators for both our internal login service and CAS.

An advantage of this approach is being able control the authentication process completely instead of just receiving session data through a global variable.

Installation

The web API requires a PHP runtime version at least 7. Which one depends on actual configuration, there is a choice between mod_php inside Apache, php-fpm with Apache or Nginx proxy or running it as standalone uWSGI script. It is common that there are some PHP extensions, that have to be installed on the system. Namely ZeroMQ binding (php-zmq package or similar), MySQL module (php-mysqlnd package) and ldap extension module for CAS authentication (php-ldap package). Make sure that the extensions are loaded in your php.ini file (/etc/php.ini or files in /etc/php.d/).

The API depends on some other projects and libraries. For managing them Composer is used. It can be installed from system repositories or downloaded from the website, where detailed instructions are as well. Composer reads composer.json file in the project root and installs dependencies to the vendor/ subdirectory. To do that, run:

$ composer install

Configuration and usage

The API can be configured in config.neon and config.local.neon files in app/config directory. The first file is predefined by authors and should not be modified. The second one is not present and could be created by copying config.local.neon.example template in the config directory. Local configuration have higher precedence, so it will override default values from config.neon.

Configurable items

Description of configurable items. All timeouts are in milliseconds if not stated otherwise.

• accessManager -- configuration of access token in JWT standard. Do not modify unless you really know what are you doing.
• fileServer -- connection to fileserver
  • address -- URI of fileserver
  • auth -- username and password for HTTP basic authentication
  • timeouts -- connection timeout for establishing new connection and request timeout for completing one request
• broker -- connection to broker
  • address -- URI of broker
  • auth -- username and password for broker callback authentication back to API
  • timeouts -- ack timeout for first response that broker receives the message, send timeout how long try to send new job to the broker and result timeout how long to wait for confirmation if job can be processed or not
• monitor -- connection to monitor
  • address -- URI of monitor
• CAS -- CAS external authentication
  • serviceId -- visible identifier of this service
  • ldapConnection -- parameters for connecting to LDAP, hostname, base_dn, port, security and bindName
  • fields -- names of LDAP keys for informations as email, firstName and lastName
• emails -- common configuration for sending email (addresses and template variables)
  • apiUrl -- base URL of API server including port (for referencing pictures in messages)
  • footerUrl -- link in the message footer
  • siteName -- name of frontend (ReCodEx, or KSP for unique instance for KSP course)
  • githubUrl -- URL to GitHub repository of this project
  • from -- sending email address
• failures -- admin messages on errors
  • emails -- additional info for sending mails, to is admin mail address, from is source address, subjectPrefix is prefix of mail subject
• forgottenPassword -- user messages for changing passwords
  • redirectUrl -- URL of web application where the password can be changed
  • tokenExpiration -- expiration timeout of temporary token (in seconds)
  • emails -- additional info for sending mails, from is source address and subjectPrefix is prefix of mail subject
• mail -- configuration of sending mails
  • smtp -- using SMTP server, have to be "true"
  • host -- address of the server
  • port -- sending port (common values are 25, 465, 587)
  • username -- login to the server
  • password -- password to the server
  • secure -- security, values are empty for no security, "ssl" or "tls"
  • context -- additional parameters, depending on used mail engine. For examle self-signed certificates can be allowed as verify_peer and verify_peer_name to false and allow_self_signed to true under ssl key (see example).

Outside the parameters section of configuration is configuration for Doctrine. It is ORM framework which maps PHP objects (entities) into database tables and rows. The configuration is simple, required items are only user, password and host with dbname, i.e. address of database computer (mostly localhost) with name of ReCodEx database.

Example local configuration file

parameters:
  accessManager:
    leeway: 60
    issuer: https://recodex.projekty.ms.mff.cuni.cz
    audience: https://recodex.projekty.ms.mff.cuni.cz
    expiration: 86400  # 24 hours in seconds
    usedAlgorithm: HS256
    allowedAlgorithms:
      - HS256
    verificationKey: "recodex-123"
  fileServer:
    address: http://127.0.0.1:9999
    auth:
      username: "user"
      password: "pass"
    timeouts:
      connection: 500
  broker:
    address: tcp://127.0.0.1:9658
    auth:
      username: "user"
      password: "pass"
    timeouts:
      ack: 100
      send: 5000
      result: 1000
  monitor:
    address: wss://recodex.projekty.ms.mff.cuni.cz:4443/ws
  CAS:
    serviceId: "cas-uk"
    ldapConnection:
      hostname: "ldap.cuni.cz"
      base_dn: "ou=people,dc=cuni,dc=cz"
      port: 389
      security: SSL
      bindName: "cunipersonalid"
    fields:
      email: "mail"
      firstName: "givenName"
      lastName: "sn"
  emails:
    apiUrl: https://recodex.projekty.ms.mff.cuni.cz:4000
    footerUrl: https://recodex.projekty.ms.mff.cuni.cz
    siteName: "ReCodEx"
    githubUrl: https://github.com/ReCodEx
    from: "ReCodEx <noreply@example.com>"
  failures:
    emails:
      to: "Admin Name <admin@example.com>"
      from: %emails.from%
      subjectPrefix: "ReCodEx Failure Report - "
  forgottenPassword:
    redirectUrl: "https://recodex.projekty.ms.mff.cuni.cz/
                  forgotten-password/change"
    tokenExpiration: 600 # 10 minues
    emails:
      from: %emails.from%
      subjectPrefix: "ReCodEx Forgotten Password Request - "
  mail:
    smtp: true
    host: "smtp.ps.stdin.cz"
    port: 587
    username: "user"
    password: "pass"
    secure: "tls"
    context:
      ssl:
        verify_peer: false
        verify_peer_name: false
        allow_self_signed: true
doctrine:
  user: "user"
  password: "pass"
  host: localhost
  dbname: "recodex-api"

Database preparation

When the API is installed and configured (doctrine section is sufficient here) the database schema can be generated. There is a prepared command to do that from command line:

$ php www/index.php orm:schema-tool:update --force

With API comes some initial values, for example default user roles with proper permissions. To fill your database with these values there is another command line command:

$ php www/index.php db:fill

Check the outputs of both commands for errors. If there are any, try to clean temporary API cache in temp/cache/ directory and repeat the action.

Webserver configuration

The simplest way to get started is to start the built-in PHP server in the root directory of your project:

$ php -S localhost:4000 -t www

Then visit http://localhost:4000 in your browser to see the welcome page of API project.

For Apache or Nginx, setup a virtual host to point to the www/ directory of the project and you should be ready to go. It is critical that whole app/, log/ and temp/ directories are not accessible directly via a web browser (see security warning). Also it is highly recommended to set up a HTTPS certificate for public access to the API.

Troubleshooting

In case of any issues first remove the Nette cache directory temp/cache/ and try again. This solves most of the errors. If it does not help, examine API logs from log/ directory of the API source or logs of your webserver.