CKAN extension for Öppnadata.se, the Swedish data management platform.

To enable, activate your CKAN virtual environment and then:

1. Add sweden_blog to ckan.plugins.

2. Install the blog plugin's requirements:

    pip install -r ckanext/sweden/blog/requirements.txt
   

3. Run the paster command to initialize the blog's database tables:

    paster --plugin=ckan sweden_blog_init -c /etc/ckan/default/development.ini
   

4. Restart CKAN.

To enable, activate your CKAN virtual environment and then:

1. Install Redis, gcc and libffi-dev:

    sudo apt-get install redis-server build-essential libffi-dev
   

2. Install sweden_dcat_rdf_harvester requirements

    pip install -r ckanext/sweden/dcat/requirements.txt
   

3. Install ckanext-harvest:

    git clone https://github.com/ckan/ckanext-harvest
    cd ckanext-harvest
    git checkout stable
    pip install -r pip-requirements.txt
    python setup.py develop
   

4. Install ckanext-dcat:

    git clone https://github.com/ckan/ckanext-dcat
    cd ckanext-dcat
    pip install -r requirements.txt
    # tmp
    pip install lxml
    python setup.py develop
   

5. Add dcat_rdf_harvester sweden_dcat_rdf_harvester harvest to ckan.plugins ensuring harvest is listed after sweden_dcat_rdf_harvester

6. Restart CKAN.

You should see the harvest pages at /harvest and Generic DCAT RDF Harvester listed as a type on /harvest/new.

The following configuration options can be used with regards to the validation of remote DCAT documents:

• ckanext.sweden.harvest.use_validation (default: True): Whether to use validation at all
• ckanext.sweden.harvest.validation_service (default: http://validator.dcat-editor.com/service): The URL of the validation service to use. The harvester will POST the contents of the remote DCAT file to this endpoint.
• ckanext.sweden.harvest.stop_on_validation_errors (default False): Whether to stop the datasets import if validation errors were found.

To enable the theme:

1. Add sweden_theme to ckan.plugins

To modify the theme of the ckanext-sweden theme you'll need to:

1. Install Node (apt-get install node) and Bower (npm install -g bower)

2. Install the front end dependancies: cd ./ckanext/sweden/theme/ && npm i && bower update

3. Re-compile assets: gulp (gulp watch will regenerate them on the whenever a change happens.)

4. Once you've made your changes make sure you commit the changes in ./ckanext/theme/resources

To enable DCAT AP 1.1 theme categories:

1. Install ckanext-ap11theme::

   pip install ckanext-ap11theme

2. Enable the DCAT AP 1.1 theme and Sweden plugins by adding ap11theme and sweden to ckan.plugins.

The sweden plugin adds the following API endpoints:

• dcat_organization_list: returns a list of all organizations that have DCAT harvesting set up. Returns a list of objects, one per organization, each of them with the following keys:

  - `id`: CKAN organization id
  - `url`: Organization website (unique across organizations)
  - `dcat_metadata_url`: DCAT output for the organization datasets (generated by CKAN)
  - `original_dcat_metadata_url`: The remote DCAT datasets that were harvested into Oppnadata.se
  - `dcat_validation`: Boolean showing whether the DCAT validation passed or not
  - `dcat_validation_date`: Date and time in which the DCAT validation last took place
  - `dcat_validation_url`: URL to the DCAT validation results
  

• dcat_validation: returns the validation output for the last harvest job of the organization harvest source. Requires an id parameter with the organization name or id.

• dcat_organization_job_list: Returns a list of harvest jobs for the given organization.

  - `id`: CKAN organization id or name
  - `status`: Filter jobs by status(New, Running, Finished)
  

  Usage:

  GET http://127.0.0.1:5000/api/3/action/dcat_organization_job_list?id=haninge-kommun&status=Finished
  
  Result:
  
        {
          "help": "http://127.0.0.1:5000/api/3/action/help_show?name=dcat_organization_job_list",
          "success": true,
          "result": [{
            "id": "c7f45a5e-459d-4226-bdbf-89cf40a8896f",
            "created": "2016-06-14 12:48:26.946219",
            "gather_started": "2016-06-14 12:48:28.408297",
            "gather_finished": "2016-06-14 12:48:30.709579",
            "finished": "2016-06-14 12:48:34.248561",
            "source_id": "8c8eeed3-1bcd-4975-856c-a285004a0072",
            "status": "Finished",
            "stats": {
              "deleted": 0,
              "updated": 3,
              "added": 0,
              "not modified": 0,
              "errored": 13
            }
          },...]
        }
  

• dcat_job_details: Returns a list of HarvestGatherError objects and HarvestObjectError objects for a given job.

  - `id`: Job id
  

  Usage:

  GET http://127.0.0.1:5000/api/3/action/dcat_job_details?id=90971e97-1f46-433b-bc43-5b2d0839e23c
  
  Result:
  
          {
            "help": "http://127.0.0.1:5000/api/3/action/help_show?name=dcat_job_details",
            "success": true,
            "result": {
              "gather_errors": [{
                "message": "{\"errors\": [], \"type\": \"http://www.w3.org/ns/dcat#Distribution\", \"uri\": \"https://haninge.entryscape.net/store/7/resource/33\", \"template\": \"dcat:Distribution\", \"warnings\": [{\"path\": \"dcterms:format\", \"code\": \"few\"}, {\"path\": \"dcterms:license\", \"code\": \"few\"}, {\"path\": \"dcterms:license\", \"code\": \"few\"}]}"
              },...]
  
              "object_errors": {
                "9a9bf7de-5d4e-4566-b976-f61a96df8461": {
                  "guid": "https://haninge.entryscape.net/store/7/resource/2",
                  "errors": [{
                    "message": "Create validation Error: {'Owner org': 'A organization must be supplied', 'Id': 'Missing value', 'Name': 'Missing value'}",
                    "type": "Import",
                    "line": null
                  }],
                  "original_url": "https://haninge.entryscape.net/store/7/metadata/dcat?recursive=dcat"
                },...
              }
          }
  

The sweden_theme extension adds a number of additional API endpoints to retrieve data about datasets in the site.

• total_datasets_by_week: the cumulative total number of datasets by week.
• weekly_dataset_activity: the number of updates to datasets per week.
• weekly_dataset_activity_new: the number of new datasets per week.

e.g.:

curl http://127.0.0.1:5000/api/3/action/weekly_dataset_activity -H "Authorization:<your-api-key>"

Groups aren't used and can be hidden with the ckanext- hidegroups extension::

pip install -e 'git+git://github.com/okfn/ckanext-hidegroups.git#egg=ckanext-hidegroups'

Then add hidegroups to ckan.plugins.

The extension includes a standalone script to automate the creation of organizations on the portal. For details, check the scripts folder.

To run the tests, first install the dev requirements (and Redis, see above):

pip install -r dev-requirements.txt

Then do:

nosetests --nologcapture --ckan --with-pylons=test.ini

To run the tests with coverage, first install coverage (pip install coverage) then do:

nosetests --nologcapture --ckan --with-pylons=test.ini --with-coverage --cover-package=ckanext.sweden --cover-inclusive --cover-erase --cover-tests