This project contains an implementation of the OIO object model, used as a standard for data exchange by the Danish government, for use with a MOX messaging queue.

You can find the current MOX specification here:

http://www.kl.dk/ImageVaultFiles/id_55874/cf_202/MOX_specifikation_version_0.PDF

As an example, you can find the Organisation hierarchy here:

http://digitaliser.dk/resource/991439

In each installation of MOX, it is possible to only enable some of the hierarchies, but we provide the following four OIO hierarchies by default:

• Klassifikation
• Sag
• Dokument
• Organisation

This README file is part of our documentation, and an HTML version can be obtained by the following command in a command prompt:

$ make -C doc

Note that this requires Sphinx to be installed — on Ubuntu or Debian, this can be done with the following command:

$ sudo apt install python-sphinx

If you're reading this on GitHub or ReadTheDocs, you're probably seeing the HTML rendering. The official location for this documentation is:

• http://mox.readthedocs.io/

Please note that as a convention, all shell commands have been prefixed with a dollar-sign, or $, representing a prompt. You should exclude this when entering the command in your terminal.

This is a technical guide. You are not expected to have a profound knowledge of the system as such, but you do have to know your way in a Bash prompt — you should be able to change the Apache configuration and e.g. disable or change the SSL certificate on your own.

LoRA currently supports Ubuntu 16.04 (Xenial). We recommend running it on a VM with the following allocation:

You should initially provision all the storage space you expect to use, as adjusting it is somewhat cumbersome. By comparison, increasing or decreasing CPU and memory is trivial.

To install the OIO REST API, run install.sh:

$ sudo apt-get install git
$ git clone https://github.com/magenta-aps/mox
$ cd mox
$ ./install.sh

The default location of the log directory is:

/var/log/mox

The following log files are created:

• Audit log: /var/log/mox/audit.log
• OIO REST HTTP access log: /var/log/mox/oio_access.log
• OIO REST HTTP error log: /var/log/mox/oio_error.log

Additionally, a directory for file uploads is created:

# settings.py
FILE_UPLOAD_FOLDER = getenv('FILE_UPLOAD_FOLDER', '/var/mox')

The oio rest api is installed as a service, for more information, see the oio_rest.service.

By default the oio_rest service can be reached as follows:

http://localhost:8080

Example:

curl http://localhost:8080/organisation/bruger

Most configurable parameters of oio_rest can be injected with environment variables, alternatively you may set the parameters explicitly in the "settings.py" file.

(Please see $ROOT/oio_rest/settings.py)

As mentioned, most parameters are accessible. If they are NOT set by you, we have provided sensible fallback values.

Example:

# settings.py

# DB (Postgres) settings
DATABASE = getenv('DB_NAME', 'mox')
DB_USER = getenv('DB_USER', 'mox')
DB_PASSWORD = getenv('DB_PASS', 'mox')

The oio rest api is served using the wsgi server gunicorn. The gunicorn server can be configured through the "guniconfig.py" file.

(Please see $ROOT/oio_rest/guniconfig.py)

Similar to the oio rest settings file, gunicorn can be configured using environment variables: :

# guniconfig.py

# Bind address (Can be either TCP or Unix socket)
bind = env("GUNICORN_BIND_ADDRESS", '127.0.0.1:8080')

On a high level the MOX actual state database consists of three server processes and several agents joining them together.

PostgreSQL

Database server providing the storage of the bi-temporal actual state database as well as validation and verification of the basic constraints.

Gunicorn

WSGI server for the oio rest api. Ideally used with a frontend HTTP proxy in production.

RabbitMQ

AMQP message broker providing interprocess communication between the various components.

Within the context of the Mox Messaging Service, agents are small pieces of software which either listen on an AMQP queue and perform operations on the incoming data, or expose certain operations as a web service.

The default installation includes the following agents:

MoxDocumentDownload

Web service for exporting actual state contents as Excel spreadsheets.

MoxDocumentUpload

Web service for importing data from Excel spreadsheets into the actual state database.

MoxRestFrontend

AMQP agent bridging the REST API.

MoxTabel

AQMP worker agent MoxDocumentDownload & MoxDocumentUpload.

SAML token authentication is enabled by default. This requires that you have access to a SAML Identity Provider (IdP) which provides a Security Token Service (STS). We currently support two types:

• Active Directory Federation Services
• WSO2

In order to use AD FS as the Security Token Service, you first need an endpoint configured in ADFS. You should name this endpoint corresponding to the designated name of the box running LoRA, for example:

https://lora.magenta.dk

As for the attributes to send, select the following:

Please note that you should configure AD FS to sign, but not encrypt, its assertions.

Then configure the following fields in oio_rest/oio_rest/settings.py:

You should now be able to test the basic configuration, and extract the signing certificate:

$ cd /path/to/mox
$ ./auth.sh --cert-only
User: user@domain
Password: <enter password here>

Now save the results to a file, e.g. adfs-cert.pem, and set that as SAML_IDP_CERTIFICATE. You may get an SSL error, in that case, you should add your certificate authority to the system. Alternatively, you can pass the --insecure option to auth.sh temporarily bypass the error.

The open source identity provider WSO2 is useful for testing. Download the binary and follow the instructions to run it.

In the folder wso2/ you can find an example init file for running the WSO2 Identity Server as a daemon.

To configure a STS, follow the instructions on https://docs.wso2.com/display/IS500/Configuring+the+Identity+Server+to+Issue+Security+Tokens (skip the part about Holder of Key).

Restart the WSO2 server! The STS endpoint simply did not work until I restarted the WSO2 server.

This is for testing with the WSO2 Identity Server as described above -we assume that this is not the configuration which the municipalities want to use in a production setting.

Log in to the IDP with the credentials provided. The IDP could, e.g., be located at https://moxtest.magenta-aps.dk:9443/.

To create a new user, enter the "Configure" tab and select "Users and roles". Enter the user's first name, last name and email address.

Important: In the URL field, enter the user's (OIO) UUID. The URL field is currently used to map between the IDP and the OIO's user concept. If the UUID is not specified, it will not be possible to authorize users correctly, nor will it be possible to make any changes to the database.

The default IdP entity ID is called "localhost". If your IdP has a different entity ID, you must change the SAML_IDP_ENTITY_ID setting to reflect your IdP's entity ID.

For testing purposes, WSO2's IdP public certificate file is included in the distribution.

When configuring the REST API to use your IdP, you must specify your IdP's public certificate file by setting in settings.py:

export SAML_IDP_CERTIFICATE=/path/to/idp_certificate.pem

In settings.py, SAML authentication can be turned off by setting:

USE_SAML_AUTHENTICATION = False

The OIO REST service provides a convenience method for requesting a SAML token in the correct base64-encoded gzipped format for use with the API.

Visit the following URL of the OIO REST server:

http://localhost:8080/get-token

Alternatively, you can run the following command locally on the server:

$ ./auth.sh -u <username> -p

You will be presented with a form with a username/password field. Optionally, you can specify the STS address to use. This will request a token from the STS service using the given username and password. It will return the value that should be used for the HTTP "Authorization" header. If it fails due to invalid username/password, an error message will be returned.

This value can then be included in the HTTP "Authorization" header, like the following:

Authorization: <output of get-token>

For testing purposes, we recommend the browser extensions Advanced REST client for Chrome or REST Easy for Firefox.

Although the Java MOX agent does this automatically, it can be useful to request a SAML token manually, for testing purposes.

To request a SAML token, it is useful to use SoapUI.

Download SoapUI and import the project provided in oio_rest/test_auth_data/soapui-saml2-sts-request.xml.

Navigate to and double-click on:

"sts" -> "wso2carbon-stsSoap11Binding" -> "Issue token - SAML 2.0"

Note: The value of <a:Address> element in <wsp:AppliesTo> must match your SAML_MOX_ENTITY_ID setting. Change as needed.

The project assumes you are running the IdP server on https://localhost:9443/ (the default).

Execute the SOAP request. You can copy the response by clicking on the "Raw" tab in the right side of the window and then selecting all, and copying to the clipboard. Paste the response, making sure that the original whitespace/indentation is preserved. Remove all elements/text surrounding the <saml2:Assertion>..</saml2:Assertion> tag. Save to a file, e.g. /my/saml/assertion.xml.

After requesting a SAML token, to make a REST request using the SAML token, you need to pass in an HTTP Authorization header of a specific format:

Authorization: saml-gzipped <base64-encoded gzip-compressed SAML assertion>

A script has been included to generate this HTTP header from a SAML token XML file. This file must only contain the <saml2:Assertion> element.

To run it:

$ python oio_rest/oio_rest/utils/encode_token.py /my/saml/assertion.xml

The output of this script can be used in a curl request by adding the parameter -H, e.g.:

$ curl -H "Authorization saml-gzipped eJy9V1................." ...

to the curl request.

Alternately, if using bash shell:

$ curl -H "$(python oio_rest/oio_rest/utils/encode_token.py" /my/saml/assertion.xml) ...

To run the API for testing or development purposes, run:

$ oio_rest/oio_api.sh 

Then, go to http://localhost:5000/site-map to see a map of all available URLs, assuming you're running this on your local machine.

The install.sh script creates an Apache VirtualHost for oio rest and MoxDocumentUpload.

To run the OIO Rest Mox Agent (the one listening for messages and relaying them onwards to the REST interface), run:

$ agents/MoxRestFrontend/moxrestfrontend.sh

NOTE:** You can start the agent in the background by running:

$ sudo service moxrestfrontend start

To test sending messages through the agent, run:

$ ./test.sh

NOTE: The install script does not set up an IDP for SAML authentication, which is enabled by default. If you need to test without SAML authentication, you will need to turn it off as described below.

To request a token for the username from the IdP and output it in base64-encoded gzipped format, run:

$ ./auth.sh -u <username> -p

Insert your username in the command argument. You will be prompted to enter a password.

If SAML authentication is turned on (i.e., if the parameter USE_SAML_AUTHENTICATION in oio_rest/oio_rest/settings.py is True), the IDP must be configured correctly — see the corresponding sections below for instruction on how to do this.

Make sure the parameter USE_SAML_AUTHENTICATION in oio_rest/oio_rest/settings.py is False.

Make sure the parameter moxrestfrontend.rest.host in agents/MoxRestFrontend/moxrestfrontend.conf is set to http://localhost:5000.

Start the (AMQP) MOX REST frontend agent:

$ sudo service moxrestfrontend start

Start the REST API:

$ oio_rest/oio_api.sh

Run the tests:

$ ./test.sh

This should give you a lot of output like this:

Deleting bruger, uuid: 1e874f85-07e5-40e5-81ed-42f21fc3fc9e
Getting authtoken
127.0.0.1 - - [27/Apr/2016 15:55:09] "DELETE /organisation/bruger/1e874f85-07e5-40e5-81ed-42f21fc3fc9e HTTP/1.1" 200 -
Delete succeeded

Note: Currently, some of the tests will give the notice: "Result differs from the expected". This is due to a bug in the tests, i.e. you should not worry about this — if you see output as described above, the system is working.

Additionally, OIO Rest has its own unit test suite:

$ cd oio_rest
$ python setup.py test

or:

$ cd oio_rest
$ ./run_tests.sh

The latter of which will automatically generate a virtual environment, and run the tests in it.

For more advanced test or production setup, please study the rest of this README and follow your organization's best practices.

The MOX messaging queue, including the ActualState database, as found in this project is free software. You are entitled to use, study, modify and share it under the provisions of Version 2.0 of the Mozilla Public License as specified in the LICENSE file.

This software was developed by Magenta ApS. For feedback, feel free to open an issue in the GitHub repository.