Install Docker for your system: https://www.docker.com/

To get eq-questionnaire-runner running the following command will build and run the containers

RUNNER_ENV_FILE=.development.env docker compose up -d

To launch a survey, navigate to http://localhost:8000/

When the containers are running you are able to access the application as normal, and code changes will be reflected in the running application. However, any new dependencies that are added would require a re-build.

To rebuild the eq-questionnaire-runner container, the following command can be used.

RUNNER_ENV_FILE=.development.env docker compose build

If you need to rebuild the container from scratch to re-load any dependencies then you can run the following

RUNNER_ENV_FILE=.development.env docker compose build --no-cache

git clone git@github.com:ONSdigital/eq-questionnaire-runner.git

In order to run locally you'll need Node.js, snappy, pyenv, jq and wkhtmltopdf installed

brew install snappy npm pyenv jq wkhtmltopdf

Create .application-version for local development

This file is automatically created and populated with the git revision id during CI for anything other than development, but the file is absent when the repo is first cloned and is required for running the app locally. Setting the contents to local removes the implication that any particular revision is used when run locally.

echo "local" > .application-version

It is preferable to use the version of Python locally that matches that used on deployment. This project has a .python_version file for this purpose.

It is recommended to install the pyenv Python version management tool to easily switch between Python versions. To install pyenv use this command:

curl https://pyenv.run | bash

After the installation it should tell you to execute a command to add pyenv to path. It should look something like this:

export PYENV_ROOT="$HOME/.pyenv"

command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"

eval "$(pyenv init -)"

Python versions can be changed with the pyenv local or pyenv global commands suffixed with the desired version (e.g. 3.11.4). Different versions of Python can be installed first with the pyenv install command. Refer to the pyenv project Readme here. To avoid confusion, check the current Python version at any given time using python --version or python3 --version.

Inside the project directory install python version, upgrade pip:

pyenv install
pip install --upgrade pip setuptools

Install poetry, poetry dotenv plugin and install dependencies:

curl -sSL https://install.python-poetry.org | python3 - --version 1.8.2
poetry self add poetry-plugin-dotenv
poetry install

We use poetry-plugin-up to update dependencies in the pyproject.toml file:

poetry self add poetry-plugin-up

To update the design system templates run:

make load-design-system-templates

To download the latest schemas from the Questionnaire Registry:

make load-schemas

Run the server inside the virtual env created by Poetry with:

make run

Runner requires five supporting services - a questionnaire launcher, a storage backend, a cache, the supplementary data service and the collection instrument registry.

To run the app locally, but the supporting services in Docker, run:

make dev-compose-up

Note that on Linux you will need to use:

make dev-compose-up-linux

docker run -e SURVEY_RUNNER_SCHEMA_URL=http://docker.for.mac.host.internal:5000 -e SDS_API_BASE_URL=http://docker.for.mac.host.internal:5003 -e CIR_API_BASE_URL=http://docker.for.mac.host.internal:5004 -it -p 8000:8000 onsdigital/eq-questionnaire-launcher:latest

docker run -it -p 5003:5003 onsdigital/eq-runner-mock-sds:latest

docker run -it -p 5004:5004 onsdigital/eq-runner-mock-cir:latest

DynamoDB

docker run -it -p 6060:8000 onsdigital/eq-docker-dynamodb:latest

or

Google Datastore

docker run -it -p 8432:8432 knarz/datastore-emulator:latest

docker run -it -p 6379:6379 redis:4

To use EQ_STORAGE_BACKEND as datastore or EQ_SUBMISSION_BACKEND as gcs directly on GCP and not a docker image, you need to set the GCP project using the following command:

gcloud config set project <gcp_project_id>

Or set the GOOGLE_CLOUD_PROJECT environment variable to your gcp project id.

There is a dev-convenience script that auto generates the lines of code for a user journey. See README for more information and how to run the script.

The frontend tests use NodeJS to run. To handle different versions of NodeJS it is recommended to install Node Version Manager (nvm). It is similar to pyenv but for Node versions. To install nvm use the command below (make sure to replace "v0.39.5" with the current latest version in releases:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash

You will need to have the correct node version installed to run the tests. To do this, use the following commands:

nvm install
nvm use

Fetch npm dependencies:

npm install

Available commands:

The tests are written using WebdriverIO, Chai, and Mocha

The functional tests use a set of selectors that are generated from each of the test schemas. These make it quick to add new functional tests.

To run the functional tests first runner needs to be spin up with:

RUNNER_ENV_FILE=.functional-tests.env make run

This will set the correct environment variables for running the functional tests.

Then you can run either:

make test-functional

or

make test-functional-headless

This will delete the tests/functional/generated_pages directory and regenerate all the files in it from the schemas.

To generate the pages manually you can run the generate_pages scripts with the schema directory. Run it from the tests/functional directory as follows:

./generate_pages.py ../../schemas/test/en/ ./generated_pages -r "../../base_pages"

To generate a spec file with the imports included, you can pass the schema name as an argument without the file extension, e.g. SCHEMA=test_address:

make generate-spec SCHEMA=<schema-name>

If you have already built the generated pages, then the functional tests can be executed with:

make test-functional

This can be limited to a single spec where argument needed is the remainder of the path after ./tests/functional/spec/ (which is included in the command):

make test-functional-spec SPEC=<spec>

To run a single test, add .only into the name of any describe or it function:

describe.only('Skip Conditions', function() {...} or

it.only('Given this is a test', function() {...}

Test suites are configured in the wdio.conf.js file. An individual test suite can be run using the suite names as the argument to this command. The suites that can be used with command below are:

• timeout_modal_expired
• timeout_modal_extended
• timeout_modal_extended_new_window
• features
• general
• components

make test-functional-suite SUITE=<suite>

To run the tests against a remote deployment you will need to specify the environment variable of EQ_FUNCTIONAL_TEST_ENV eg:

EQ_FUNCTIONAL_TEST_ENV=https://staging-new-surveys.dev.eq.ons.digital/ npm run test_functional

For deploying with Concourse see the CI README.

To deploy this application with gcloud, you must be logged in using gcloud auth login and gcloud auth application-default login.

When deploying with gcloud the environment variables specified in Deploying the app must be set.

Then call the following command with environment variables set:

./ci/deploy_app.sh

Before deploying the app to GCP you need to create the application credentials. Run the following command to provision the credentials:

PROJECT_ID=PROJECT_ID EQ_KEYS_FILE=PATH_TO_KEYS_FILE EQ_SECRETS_FILE=PATH_TO_SECRETS_FILE ./ci/deploy_credentials.sh

For example:

PROJECT_ID=eq-test EQ_KEYS_FILE=dev-keys.yml EQ_SECRETS_FILE=dev-secrets.yml ./ci/deploy_credentials.sh

The following environment variables must be set when deploying the app.

The following environment variables are optional:

To deploy the app, run the following command:

./ci/deploy_app.sh

We use flask-babel to do internationalisation. To extract messages from source and create the messages.pot file, in the project root run the following command.

make translation-templates

make translation-templates is a command that uses pybabel to extract static messages.

This will extract messages and place them in the .pot files ready for translation.

These .pot files will then need to be translated. The translation process is documented in Confluence here

Once we have the translated .po files they can be added to the source code and used by the application

The following env variables can be used

The following env variables can be used when running tests

EQ_FUNCTIONAL_TEST_ENV - the pre-configured environment [local, docker, preprod] or the url of the environment that should be targeted

Integration with the survey runner requires the use of a signed JWT using public and private key pair (see https://jwt.io, https://tools.ietf.org/html/rfc7519, https://tools.ietf.org/html/rfc7515).

Once signed the JWT must be encrypted using JWE (see https://tools.ietf.org/html/rfc7516).

The JWT payload must contain the following claims:

• exp - expiration time
• iat - issued at time

The header of the JWT must include the following:

• alg - the signing algorithm (must be RS256)
• type - the token type (must be JWT)
• kid - key identification (must be EDCRRM)

The JOSE header of the final JWE must include:

• alg - the key encryption algorithm (must be RSA-OAEP)
• enc - the key encryption encoding (must be A256GCM)

To access the application you must provide a valid JWT. To do this browse to the /session url and append a token parameter. This parameter must be set to a valid JWE encrypted JWT token. Only encrypted tokens are allowed.

There is a python script for generating tokens for use in development, to run:

python token_generator.py

Refer to our profiling document.

To add a new dependency, use:

poetry add [package-name]

This will add the required packages to your pyproject.toml and install them

To update a dependency, use:

poetry update [package-name]

This will resolve the required dependencies of the project and write the exact versions into poetry.lock

Using the poetry up plugin we can update dependencies and bump their versions in the pyproject.toml file

To update dependencies to the latest compatible version with respect to their version constraints specified in the pyproject.toml file:

poetry up

To update dependencies to their latest compatible version:

poetry up --latest

NB: both the pyproject.toml and poetry.lock files are required in source control to accurately pin dependencies.

To add a new dependency, use npm install [dev dependency] --save-dev or npm install [dependency] then use npm install to install all the packages locally.

Checkout branch with new changes on

You will need to install the Design System dependencies. If you haven't installed Yarn, install it with npm i -g yarn. To install the dependencies run yarn in the terminal. If you haven't you will also need to install gulp.

Then in the terminal run:

yarn cdn-bundle
cd build
browser-sync start --cwd -s --http --port 5678

You should now see output indicating that files are being served from localhost:5678. So main.css for example will now be served on http://localhost:5678//css/main.css

Now switch to the eQ Questionnaire Runner Repo

In a separate terminal window/tab: Checkout the runner branch you want to test on

Edit your .development.env with following:

CDN_URL=http://localhost:5678
CDN_ASSETS_PATH=

Edit the Makefile to remove load-design-system-templates from the build command. Should now look like this:

build: load-schemas translate

Run make load-design-system-templates in the terminal to make sure you have the Design System templates loaded

Then edit the first line in the templates/layout/_template.njk file to remove the version number. Should now look like this:

{% set release_version = '' %}

Then spin up launcher and runner with make dev-compose-up and make run

Now when navigating to localhost:8000 and launching a schema, this will now be using the local cdn with the changes from the Design System branch