CKAN extension for the UNHCR RIDL project

• Requirements
• Setting up environment
• Working with docker
• Starting development server
• Running unit tests
• Running E2E tests
• Building static assets
• Working with i18n
• Loging into container
• Updating readme
• Managing docker
• Generate deposited-dataset schema
• Create data containers and data deposit
• Create development users
• Testing email notifications

This extension is being developed against CKAN 2.7.x

Please follow installation instructions of the software below if needed. Also, take a look inside the Makefile to understand what's going on under the hood:

• docker
• docker-compose
• /etc/hosts contains the 127.0.0.1 ckan-dev line

For building static assets and running end-to-end tests Node.js is required and can be installed with these commands:

$ nvm install 10
$ nvm use 10
$ npm install

Clone the ckanext-unhcr repository (assuming that we're inside the docker-ckan-unhcr/src directory):

$ git clone git@github.com:okfn/ckanext-unhcr.git
$ cd ckanext-unhcr

It's designed to support live development of extensions. The only one requirement is that the folder with the project should be inside docker-ckan-unhcr/src. See docker-ckan-unhcr for more information.

The whole docker setup is inside the docker-ckan-unhcr directory. You can tweak any CKAN instance's aspects there (e.g. patches/cron/etc). To add other CKAN extensions to the work - add its folders to docker-compose.dev.yml (see ckan-dev volumes).

Pull the latest ckan-base/dev images and build the project's images:

$ make docker

Let's start the development server. It's recommended to run this command in an additional terminal window because you need it running during the work. All changes to connected extensions will trigger live-reloading of the server:

$ make start
# see CKAN logs here

Now we can visit our local ckan instance at (you can login using ckan_admin@test1234):

http://ckan-dev:5000/

We write and store unit tests inside the ckanext/unhcr/tests. Prefer to name test files after feature/bug names. To run the tests you should have the development server up and running:

$ make test

To run only selected tests:

• add @nose.plugins.attrib.attr('only') decorator to the selected test
• run $ make test ARGS='-a only'

We write and store E2E tests inside the tests directory. Prefer to name test files after feature/bug names. To run the tests you should have the development server up and running:

$ make e2e
$ npx nightwatch tests/<testname>.js # for a single E2E test

See the how to write E2E tests guide:

• http://nightwatchjs.org/guide

Put your images/fonts/etc inside the ckanext/unhcr/fanstatic folder. It can be used as usual ckan fanstatic and public contents. At the same time, we use JS and CSS preprocessors to build. Put your scripts/styles inside the ckanext/unhcr/src and build it:

$ make assets

Processed styles will be put to the ckanext/unhcr/fanstatic folder.

To extract i18n messages and compile the catalog we have to have our development server running. In another terminal window run a command:

$ make i18n

See CKAN documentation for more on i18n management.

To issue commands inside a running container:

$ make shell

Now you can tweak the running ckan-dev docker container from inside. Please take into account that all changes will be lost after the next container restart.

To update this readme's table of contents run:

$ make readme

There are a few useful docker commands:

$ docker ps -aq # list all containers
$ docker stop $(docker ps -aq) # stop all containers
$ docker rm $(docker ps -aq) # remove all containers
$ docker rmi $(docker images -q) # remove all images
$ docker system prune -a --volumes # CAUTION: it will purge all docker projects
$ docker volume rm dockerckan<project>_ckan_storage dockerckan<project>_pg_data # remove project volumes

It will be generated based on the dataset schema (re-writing existent deposited-dataset schema).

$ python scripts/generate_deposited_dataset_schema.py

It will create all initial data containers and data deposit. For local development url should be http://ckan-dev:5000 and api_key from your user profile.

$ python scripts/initial_data_containers.py url api_key
$ python scripts/create_data_deposit.py url api_key

Create users using command line interface:

# Sysadmin
docker-compose -f ../../docker-compose.dev.yml exec ckan-dev paster --plugin=ckan sysadmin add ckan_sysadmin email=sysadmin@example.com password=test -c /srv/app/production.ini
# Depadmin
docker-compose -f ../../docker-compose.dev.yml exec ckan-dev paster --plugin=ckan user add ckan_depadmin email=depadmin@example.com password=test -c /srv/app/production.ini
# Curators
docker-compose -f ../../docker-compose.dev.yml exec ckan-dev paster --plugin=ckan user add ckan_curator1 email=curator1@example.com password=test -c /srv/app/production.ini
docker-compose -f ../../docker-compose.dev.yml exec ckan-dev paster --plugin=ckan user add ckan_curator2 email=curator2@example.com password=test -c /srv/app/production.ini
# Depositors
docker-compose -f ../../docker-compose.dev.yml exec ckan-dev paster --plugin=ckan user add ckan_user1 email=user1@example.com password=test -c /srv/app/production.ini
docker-compose -f ../../docker-compose.dev.yml exec ckan-dev paster --plugin=ckan user add ckan_user2 email=user2@example.com password=test -c /srv/app/production.ini

Add admins and and editors to data deposit using web interface:

• make ckan_depadmin admin of data-deposit
• make ckan_curator1 and ckan_curator2 editors of data-deposit

We use a fake SMTP server to test email notifications:

• log into https://mailtrap.io/inboxes
• select Demo Inbox
• copy SMTP credentials
• past to docker-ckan-ed:.env (mail service connection section)
• restart the development server

Now all email sent by from ckan.lib.mailer import mail_user should be sent to the Demo Inbox at Mailtrap.