This is the code repository for the Data Transformation Service (DTS), developed for Nomis. It is a command-line based microservice that serves to allow users to easily import census data, along with associated metadata, into the Nomis web dissemination tool. It is able to acquire data both from local files (as raw data received directly from the Office for National Statistics) as well as from the Cantabular API, and the service transforms this data into a format that is compatable with the Nomis web dissemination tools.

This repository contains

• The source code for the utility
• A suite of unit tests (/tests)
• Example dataset and metadata sets (/examples)
• The user manual, maintenance guide, and code reference (/docs/manual.pdf)
• Example commands (USAGE.txt)
• Example config file (config.json)
• Pip dependencies file (requirements.txt)

• master: Contains the most up-to-date source code for development/testing
• live: Contains a modified version, for use with the live Nomis database
• submitted: Contains all the files for the submission of the product to Durham University for marking

The utility is written in Python. To guarantee a successful run please ensure you are using Python 3.8 or above. The following libraries are required:

• requests
• pyjstat
• chardet

To install them automatically, run: pip install -r requirements.txt

The Nomis Data API and Metadata API must be available for the program to work. To compile and run the Data API, make sure you have dotnet installed. Navigate to the mock-apis directory, then to nomis-api-v0.0.5-metadata-v0.0.2 Then run the following commands:

cd fe-api
dotnet build
dotnet run bin/Debug/netcoreapp3.1/fe-api.dll

To compile and run the Metadata API, again from the mock API directory, run these commands:

cd fe-api-metadata
dotnet build
dotnet run bin/Debug/netcoreapp3.1/fe-api-metadata.dll

For transforming datasets, only the Data API must be running. For transforming metadata both must be running.

The config.json already contains connection information and credentials for the Cantabular API, as well as connection information for the mock Nomis APIs.

NOTE: The current Nomis mock APIs do not support Geography Variables, so the relevant entry in the config.json must be left as an empty list. This functionality is necessary for the real APIs however, and works properly there.

To change the configuration, please refer to the manual.

Full instructions can be found in the USAGE.txt and in the manual

python main.py data -i "SYN123" -t "CENSUS TEST 1" -f "examples/cantabular_query_example.json"

python main.py data -q "SEX, AGE" -i "SYN456" -t "CENSUS TEST 2" -d "Usual-Residents"

python main.py data -i "SYN123" -t "CENSUS TEST 1" -f "examples/cantabular_query_example.json"

python main.py data -q "SEX, AGE" -i "SYN456" -t "CENSUS TEST 2" -d "Usual-Residents"

python main.py metadata -f "examples/ons_metadata_example.json" -r "O"

python main.py metadata -f "examples/cantabular_metadata_example.json" -r "C"

With the mock Data and Metadata APIs running, you can view a Swagger Interface for each API by going to:

https://localhost:[PORT]/swagger

So with the pre-configured ports these UIs will be available at:

https://localhost:5001/swagger
https://localhost:5005/swagger

From there you can perform GET-requests to verify that the imports performed by the DTS were successful.