Leeloo provides an API into Data Hub for Data Hub clients. Using Leeloo you can search for entities and manage companies, contacts and interactions.

More guides can be found in the docs folder.

Leeloo uses Docker compose to setup and run all the necessary components. The docker-compose.yml file provided is meant to be used for running tests and development.

1. Clone the repository:

   git clone https://github.com/uktrade/data-hub-leeloo
   cd data-hub-leeloo

2. Create a .env file from sample.env

   cp sample.env .env

3. Build and run the necessary containers for the required environment:

   docker-compose build

4. Populate the database and initialise Elasticsearch:

   docker-compose run leeloo ./manage.py migrate
   docker-compose run leeloo ./manage.py init_es
   docker-compose run leeloo ./manage.py loadinitialmetadata
   docker-compose run leeloo ./manage.py createinitialrevisions

   • NOTE: If you are using a linux system, these commands may hang on waiting for the elasticsearch container to come up (data-hub-leeloo_es_1) - it might be perpetually restarting. If the logs for that container mention something like max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144], you will need to run the following on your host machine:

     sudo sysctl -w vm.max_map_count=262144

     and append/modify the vm.max_map_count setting in /etc/sysctl.conf (so that this setting persists after restart):

     vm.max_map_count=262144

     For more information, see the elasticsearch docs on vm.max_map_count.

5. Optionally, you can load some test data and update elasticsearch:

   docker-compose run leeloo ./manage.py loaddata /app/fixtures/test_data.yaml
   docker-compose run leeloo ./manage.py sync_es

6. Create a superuser:

   docker-compose run leeloo ./manage.py createsuperuser

7. Run the services:

   docker-compose up

8. Optionally, you may want to run a local copy of the data hub frontend. By default, you can run both leeloo and the frontend under one docker-compose project. See the instructions in the frontend readme to set it up.

Dependencies:

• Python 3.7.x
• PostgreSQL 10 (note: PostgreSQL 9.6 is used for the MI database)
• redis 3.2
• Elasticsearch 6.4

1. Clone the repository:

   git clone https://github.com/uktrade/data-hub-leeloo
   cd data-hub-leeloo

2. Install Python 3.7.

   See this guide for detailed instructions for different platforms.

3. Install system dependencies:

   On Ubuntu:

   sudo apt install build-essential libpq-dev python3.7-dev python3.7-venv

   On macOS:

   brew install libpq

4. Create and activate the virtualenv:

   python3.7 -m venv env
   source env/bin/activate
   pip install -U pip

5. Install the dependencies:

   pip install -r requirements.txt

6. Create an .env settings file (it’s gitignored by default):

   cp config/settings/sample.env config/settings/.env

7. Set DOCKER_DEV=False in .env

8. Create the db. By default, the dev version uses postgres:

   psql -p5432
   create database datahub;

9. Make sure you have Elasticsearch running locally. If you don't, you can run one in Docker:

   docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" docker.elastic.co/elasticsearch/elasticsearch:6.3.2

10. Make sure you have redis running locally and that the REDIS_BASE_URL in your .env is up-to-date.

11. Populate the database and initialise Elasticsearch:

    ./manage.py migrate
    ./manage.py init_es
    
    ./manage.py loadinitialmetadata
    ./manage.py createinitialrevisions

12. Optionally, you can load some test data and update Elasticsearch:

    ./manage.py loaddata fixtures/test_data.yaml
    
    ./manage.py sync_es

13. Create a superuser:

    ./manage.py createsuperuser

14. Start the server:

    ./manage.py runserver

15. Start celery:

    celery worker -A config -l info -Q celery,long-running -B

    Note that in production the long-running queue is run in a separate worker with the -O fair --prefetch-multiplier 1 arguments for better fairness when long-running tasks are running or pending execution.

If using Docker, prefix these commands with docker-compose run leeloo.

To run the tests:

./tests.sh

To run the tests in parallel, pass -n <number of processes> to ./tests.sh. For example, for four processes:

./tests.sh -n 4

To run the linter:

flake8

The internal front end uses single sign-on. You should configure Leeloo as follows to use with the front end:

• SSO_ENABLED: True
• RESOURCE_SERVER_INTROSPECTION_URL: URL of the RFC 7662 introspection endpoint (should be the same server the front end is using). This is provided by a Staff SSO instance.
• RESOURCE_SERVER_AUTH_TOKEN: Access token for the introspection server.

The token should have the data-hub:internal-front-end scope. django-oauth-toolkit will create a user corresponding to the token if one does not already exist.

To give access to a machine-to-machine client that doesn't require user authentication:

1. Log into the Django admin applications page and add a new OAuth application with these details:

   • Client type: Confidential
   • Authorization grant type: Client credentials

2. Define the required scopes for the app by adding a new record in the OAuth application scopes page with these details:

   • Application: The application just created
   • Scope: The required scopes

The currently defined scopes can be found in datahub/oauth/scopes.py.

Further information about the available grant types can be found in the OAuthLib docs.

Leeloo can run on any Heroku-style platform. Configuration is performed via the following environment variables:

If using Docker, remember to run these commands inside your container by prefixing them with docker-compose run leeloo.

./manage.py migrate

If the database is freshly built or a new versioned model is added run:

./manage.py createinitialrevisions

These commands are generally only intended to be used on a blank database.

./manage.py loadinitialmetadata

Create the Elasticsearch index (if it doesn't exist) and update the mapping:

./manage.py init_es

Resync all Elasticsearch records:

./manage.py sync_es

You can resync only specific models by using the --model= argument.

./manage.py sync_es --model=company --model=contact

For more details including all the available choices:

./manage.py sync_es --help

Migrate modified mappings:

./manage.py migrate_es

Elasticsearch mapping migrations are fairly complex – see docs/Elasticsearch migrations.md for more detail.

Update Companies House records:

./manage.py sync_ch

This downloads the latest data from Companies House, updates the Companies House table and triggers an Elasticsearch sync.

(Note that this does not remove any records from the Companies House table.)

See Managing dependencies for information about installing, adding and upgrading dependencies.

The /v3/activity-stream/ endpoint is protected by two mechanisms:

• IP address whitelisting via the X-Forwarded-For header, with a comma separated list of whitelisted IPs in the environment variable ACTIVITY_STREAM_IP_WHITELIST.

• Hawk authentication via the Authorization header, with the credentials in the environment variables ACTIVITY_STREAM_ACCESS_KEY_ID and ACTIVITY_STREAM_SECRET_ACCESS_KEY.

The authentication blocks requests that do not have a whitelisted IP in the second-from-the-end IP in X-Forwarded-For header. In general, this cannot be trusted. However, in PaaS, this can be, and this is the only production environment. Ideally, this would be done at a lower level than HTTP, but this is not possible with the current architecture.

If making requests to this endpoint locally, you must manually add this header.

In general, Hawk authentication hashing the HTTP payload and Content-Type header, and using a nonce, are both optional. Here, as with the Activity Stream endpoints in other DIT projects, both are required. Content-Type may be the empty string, and if there is no payload, then it should be treated as the empty string.