Contains:

• the public-facing REST API for Notification built on the GOV.UK Notify platform, which teams can integrate with using their clients
• an internal-only REST API built using Flask to manage services, users, templates, etc (this is what the admin app talks to)
• asynchronous workers built using Celery to put things on queues and read them off to be processed, sent to providers, updated, etc

• We currently do not support sending of letters
• We currently do not receive a response if text messages were delivered or not
• We currently do not receive status updates from our email provider (Granicus) while waiting for ESECC process to be completed

• Notification API
  • Functional constraints
  • Table of Contents
  • API Documentation
  • Setting Up
    • Checklist
    • Local installation instruction
    • Pre-commit hooks
    • Installation for local development
    • Other useful commands
  • To run the queues
  • Running in Docker
  • AWS Configuration
    • Install tools
    • Useful commands
  • AWS Lambda Functions
  • Terraform
    • Install tools
    • Useful commands
    • Python version
  • To update application dependencies
  • Running Code Scans
    • Bandit
    • Safety
  • Testing template changes
  • Frequent problems

VANotify OpenAPI specification can be downloaded here

Postman collection and environment files are available here

• Local installation in this section
  • PyEnv
  • Python
  • virtualenv
  • Postgres
  • project dependencies
  • Docker
• Install pre-commit hooks in this section
  • pre-commit
  • talisman
• Installation for local development in this section

pip install --upgrade pip
pip install --upgrade virtualenvwrapper

export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/Devel
source  ~/.pyenv/versions/3.6.10/bin/virtualenvwrapper.sh

brew bundle

pre-commit install

Install LocalStack, which is library that mocks AWS services, including SQS (which we use to send messages), run:

pip3 install -r requirements_for_local.txt

Set environment variables required to run LocalStack:

export SERVICES=sqs
export DEFAULT_REGION=us-east-2

To get LocalStack started, which by default will spin up a Docker container, run:

localstack start

Upon starting up LocalStack, you can visit the provided port at https://localhost:4566/. You should get

{"status": "running"}

If you have issues with LocalStack, you can downgrade to 0.11.2, run:

docker pull localstack/localstack:0.11.2
docker tag localstack/localstack:0.11.2 localstack/localstack:latest

To checks if queues are created, run:

AWS_ACCESS_KEY_ID=foo AWS_SECRET_ACCESS_KEY=bar aws --endpoint-url=http://localhost:{port} sqs list-queues

To check if messages are queued up, run:

AWS_ACCESS_KEY_ID=foo AWS_SECRET_ACCESS_KEY=bar aws --endpoint-url=http://localhost:{port} sqs receive-message --max-number-of-messages 10 --queue-url={queue url provided in from list-queues command}

• Note that the max number (n) of messages can be 1 < n <= 10

scripts/run_celery.sh

scripts/run_celery_beat.sh

Make sure to copy over the .docker-env.example file and fill in the values.

When building the docker image for the first time, run docker-compose -f ci/docker-compose-local.yml build app

To run all the tests docker-compose -f ci/docker-compose-test.yml up --build --abort-on-container-exit

To run the application and it's associated postgres instance docker-compose -f ci/docker-compose.yml up --build --abort-on-container-exit

To run the app locally, with celery using localstack docker-compose -f ci/docker-compose-local.yml up --build

At the moment, we have our own AWS account. In order to run the application you have to be authenticated with AWS because of this AWS infrastructure setup. So the following will need to be installed.

brew install awscli
brew tap versent/homebrew-taps
brew install saml2aws

export AWS_PROFILE={profile}

aws sts get-caller-identity

[//]: #:TODO:"Add notes with useful commands and other development details...this is a work in progress draft"

We house our lambda functions in /lambda_functions/. The infrastructure resources live in our infra repo as part of the utility stack.

• user_flows_handler
  • Triggers user flows tests
  • lives in /lambda_functions/user_flows/

Follow user flows lambda setup as model

1. If creating a new lambda function, the basic assets needed are:

• Create needed lambda resources in our infra repo, which requires a dummy lambda zip file deployed
• Create a subdirectory in the /lambda_functions/
• Create a file in the subdirectory using the naming convention {function name}_lambda.py
• In the lambda file, it should define the handler with similar convention: {function name}_handler(event, context)
• Define a new deploy job in 'Build Lambda Functions' workflow following the naming convention deploy-{functon name}-lambda

2. Building out the lambda function:

• Any time you make changes and push them up, the 'Build Lambda Functions' workflow will be triggered, specifically related to the lambda you're working on if the deploy job was scoped correctly. This workflow packages up the required artifacts and deploys to the AWS Lambda service. Current setup does not invoke the function upon completed deployment.
• To invoke the function, you can either use the AWS Lambda project specific console or awscli.

Resources:

• AWS Lambda Documentation

In this repository, our Terraform infrastructure files are separated into several directories. The following directories dictate the order in which terraform init needs to be separately run. Order is important because each directory configures resources that the next one needs. Currently, the order is:

1. /cd/base-infrastructure/
2. /cd/application-database/
3. /cd/application-infrastructure/

brew install terraform

terraform init

terraform validate

terraform plan

terraform apply

This codebase is Python 3 only. At the moment we run 3.6.9 in production. You will run into problems if you try to use Python 3.4 or older, or Python 3.7 or newer.

requirements.txt file is generated from the requirements-app.txt in order to pin versions of all nested dependencies. If requirements-app.txt has been changed (or we want to update the unpinned nested dependencies) requirements.txt should be regenerated with

make freeze-requirements

requirements.txt should be committed alongside requirements-app.txt changes.

The instructions below are for running locally and are implemented as jobs in our pipeline.

Scans for common security issues in Python code

make check-vulnerabiltiies

Checks python dependencies for known security vulnerabilities

Note: Some of the dependencies are generated from the utils repo, so dependencies may need to be updated there first.

make check-dependencies

Jinja templates are pulled in from the notification-utils repo. Jinja templates used in this repo: email_template.jinja2. To test jinja changes locally without needing to push changes to notification-utils, follow this procedure:

1. Make markup changes to email_template.jinja2 (notifications_utils/jinja_templates/email_template.jinja2)

2. (optional) Modify notifications_utils/version.py and add -SNAPSHOT to version number, e.g. 'version.number-SNAPSHOT'. This will allow to easily revert local copy of notifications-utils in sites-packages to official version from git.

3. From within the notification-api virtual environment reinstall the utils dependency into the api repo, run:

   pip install file:///path/to/notification-utils
   

4. See the changes locally!

   • Be sure the local server for notification-api is running to send an email.
   • Refer to section Installation for Local Development for local setup or Running in Docker for Docker command to local startup.

5. Repeat steps 1, 2 and 3 until satisfied.

6. When finished run:

   pip install -r requirements.txt
   

When adding environment variables to any <filename>-task-definition.json file, make sure you add them to the corresponding celeryand celery-beat files as well. We want to try and keep these consistent with each other.

Problem : E999 SyntaxError: invalid syntax when running flake8

Solution : Check that you are in your correct virtualenv, with python 3.5 or 3.6

Problem:

/bin/sh: 1: Syntax error: "(" unexpected
make: *** [Makefile:31: freeze-requirements] Error 2

when running make freeze-requirements

Solution: Change /bin/sh to /bin/bash in the Makefile

Problem: ImportError: failed to find libmagic. Check your installation

Solution:Install libmagic, ex: brew install libmagic

Problem: assert 'test_notification_api' in db.engine.url.database, 'dont run tests against main db

Solution: Do not specify a database in your .env

Problem: Messages are in the queue but not sending

Solution: Check that celery is running.