Data Hub API provides an API into Data Hub for Data Hub clients. Using Data Hub API you can search for entities and manage companies, contacts and interactions.

More guides can be found in the docs folder.

This project uses Docker compose to setup and run all the necessary components. The docker-compose.yml file provided is meant to be used for running tests and development.

Note for Mac Users: By default, docker on Mac will restrict itself to using just 2GB of memory. This should be increased to at least 4GB to avoid running in to unexpected problems.

1. Clone the repository:

   git clone https://github.com/uktrade/data-hub-api
   cd data-hub-api

2. Create a .env file from sample.env

   cp sample.env .env

   If you're working with data-hub-frontend and mock-sso, DJANGO_SUPERUSER_EMAIL should be the same as MOCK_SSO_USERNAME in the frontend's .env file.

3. Build and run the necessary containers for the required environment:

   docker-compose up

   • It will take time for the API container to come up - it will run migrations on both DBs, load initial data, sync elasticsearch etc. Watch along in the api container's logs.

   • NOTE: If you are using a linux system, the elasticsearch container may not come up successfully (data-hub-api_es_1) - it might be perpetually restarting. If the logs for that container mention something like max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144], you will need to run the following on your host machine:

     sudo sysctl -w vm.max_map_count=262144

     and append/modify the vm.max_map_count setting in /etc/sysctl.conf (so that this setting persists after restart):

     vm.max_map_count=262144

     For more information, see the elasticsearch docs on vm.max_map_count.

4. Optionally, you may want to run a local copy of the data hub frontend. By default, you can run both the API and the frontend under one docker-compose project. See the instructions in the frontend readme to set it up.

Dependencies:

• Python 3.8.x
• PostgreSQL 10 (note: PostgreSQL 9.6 is used for the MI database)
• redis 3.2
• Elasticsearch 6.8

1. Clone the repository:

   git clone https://github.com/uktrade/data-hub-api
   cd data-hub-api

2. Install Python 3.8.

   See this guide for detailed instructions for different platforms.

3. Install system dependencies:

   On Ubuntu:

   sudo apt install build-essential libpq-dev python3.8-dev python3.8-venv

   On macOS:

   brew install libpq

4. Create and activate the virtualenv:

   python3.8 -m venv env
   source env/bin/activate
   pip install -U pip

5. Install the dependencies:

   pip install -r requirements.txt

6. Create an .env settings file (it’s gitignored by default):

   cp config/settings/sample.env config/settings/.env

7. Set DOCKER_DEV=False in .env

8. Create the required PostgreSQL databases:

   psql -p5432
   create database datahub;
   create database mi;

   (Most Django apps use the datahub database. The mi database is used only by the mi_dashboard Django app.)

9. Make sure you have Elasticsearch running locally. If you don't, you can run one in Docker:

   docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" docker.elastic.co/elasticsearch/elasticsearch:6.8.2

10. Make sure you have redis running locally and that the REDIS_BASE_URL in your .env is up-to-date.

11. Populate the databases and initialise Elasticsearch:

    ./manage.py migrate
    ./manage.py migrate --database mi
    ./manage.py migrate_es
    
    ./manage.py loadinitialmetadata
    ./manage.py createinitialrevisions

12. Optionally, you can load some test data:

    ./manage.py loaddata fixtures/test_data.yaml

    Note that this will queue Celery tasks to index the created records in Elasticsearch, and hence the loaded records won‘t be returned by search endpoints until Celery is started and the queued tasks have run.

13. Create a superuser:

    ./manage.py createsuperuser

14. Start the server:

    ./manage.py runserver

15. Start celery:

    celery worker -A config -l info -Q celery,long-running -B

    Note that in production the long-running queue is run in a separate worker with the -O fair --prefetch-multiplier 1 arguments for better fairness when long-running tasks are running or pending execution.

Automatically-generated API documentation is served at /docs (requires admin site credentials).

If using Docker, prefix these commands with docker-compose run api.

To run the tests:

./tests.sh

To run the tests in parallel, pass -n <number of processes> to ./tests.sh. For example, for four processes:

./tests.sh -n 4

To run the linter:

flake8

The internal front end uses single sign-on. You should configure the API as follows to use with the front end:

• SSO_ENABLED: True
• RESOURCE_SERVER_INTROSPECTION_URL: URL of the RFC 7662 introspection endpoint (should be the same server the front end is using). This is provided by a Staff SSO instance.
• RESOURCE_SERVER_AUTH_TOKEN: Access token for the introspection server.

The token should have the data-hub:internal-front-end scope. django-oauth-toolkit will create a user corresponding to the token if one does not already exist.

To give access to a machine-to-machine client that doesn't require user authentication:

1. Log into the Django admin applications page and add a new OAuth application with these details:

   • Client type: Confidential
   • Authorization grant type: Client credentials

2. Define the required scopes for the app by adding a new record in the OAuth application scopes page with these details:

   • Application: The application just created
   • Scope: The required scopes

The currently defined scopes can be found in datahub/oauth/scopes.py.

Further information about the available grant types can be found in the OAuthLib docs.

Data Hub API can run on any Heroku-style platform. Configuration is performed via the following environment variables:

If using Docker, remember to run these commands inside your container by prefixing them with docker-compose run api.

./manage.py migrate

./manage.py migrate --database mi

If the database is freshly built or a new versioned model is added run:

./manage.py createinitialrevisions

These commands are generally only intended to be used on a blank database.

./manage.py loadinitialmetadata

To create missing Elasticsearch indexes and migrate modified mapping types:

./manage.py migrate_es

This will also resync data (using Celery) for any newly-created indexes.

See docs/Elasticsearch migrations.md for more detail about how the command works.

To resync all records using Celery:

./manage.py sync_es

To resync all records synchronously (without Celery running):

./manage.py sync_es --foreground

You can resync only specific models by using the --model= argument.

./manage.py sync_es --model=company --model=contact

For more details including all the available choices:

./manage.py sync_es --help

See Managing dependencies for information about installing, adding and upgrading dependencies.

The /v3/activity-stream/* endpoints are protected by two mechanisms:

• IP address whitelisting via the X-Forwarded-For header, with a comma separated list of whitelisted IPs in the environment variable PAAS_IP_WHITELIST.

• Hawk authentication via the Authorization header, with the credentials in the environment variables ACTIVITY_STREAM_ACCESS_KEY_ID and ACTIVITY_STREAM_SECRET_ACCESS_KEY.

The authentication blocks requests that do not have a whitelisted IP in the second-from-the-end IP in X-Forwarded-For header. In general, this cannot be trusted. However, in PaaS, this can be, and this is the only production environment. Ideally, this would be done at a lower level than HTTP, but this is not possible with the current architecture.

If making requests to this endpoint locally, you must manually add this header or disable the check using DISABLE_PAAS_IP_CHECK environment variable.

In general, Hawk authentication hashing the HTTP payload and Content-Type header, and using a nonce, are both optional. Here, as with the Activity Stream endpoints in other DIT projects, both are required. Content-Type may be the empty string, and if there is no payload, then it should be treated as the empty string.