CKAN Extension for the Provincia di Trento (http://dati.trentino.it)

Master branch has been developed and tested on CKAN 2.9 and Python 3.

Available plugins in this extension:

• datitrentinoit: Theme customization
• statwebpro_harvester: harvests metadata at level "pro" from services exposed by http://www.statweb.provincia.tn.it/.
• statwebsubpro_harvester: harvests metadata at level "subpro" from services exposed by http://www.statweb.provincia.tn.it/.
• opencity_harvester: harvests OpenCity catalogs
• comunweb_harvester: harvests ComunWebV2 catalogs
• broken_links_report: a report for detecting broken links in datasets' resources

This extension requires the following extensions:

• ckanext-harvester,
• ckanext-spatial,
• ckanext-multilang
• ckanext-dcatapit
• ckanext-report

Optional extensions:

• ckanext-grouplabel: replaces "group" with "category" in labels.

1. Activate your CKAN virtual environment, for example: . /usr/lib/ckan/default/bin/activate

2. Install all the other required extensions

3. Go into your CKAN path for extension (like /usr/lib/ckan/default/src):

   git clone https://github.com/geosolutions-it/ckanext-datitrentinoit.git 
   cd ckanext-datitrentinoit 
   pip install -e .
   

4. Add datitrentinoit and all the desired plugins to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/production.ini).

5. Add datitrentinoit_ap to the ckanext.dcat.rdf.profiles setting, so that it reads:

   ckanext.dcat.rdf.profiles = euro_dcat_ap it_dcat_ap datitrentinoit_ap
   

6. Restart CKAN. For example if you've deployed CKAN with Apache on Ubuntu::

   sudo service apache2 reload
   

To install ckanext-datitrentinoit for development, activate your CKAN virtualenv and do:

git clone https://github.com/geosolutions-it/ckanext-datitrentinoit.git
cd ckanext-datitrentinoit
python setup.py develop
pip install -r dev-requirements.txt

Using the ckanext-datitrentinoit harvester you can use an additional configuration property in addition to the other allowed for the ckanext-geonetwork and ckanext-multilang harvesters:

• default_license: with this property you can specify the default license to use for the CKAN's dataset if none useLimitation has been found into the metadata. Below an example:

    {
        "default_license": "cc-zero"
    }
  

• default_values: with this property you can specify some default values to use for missing properties in CSW metadata. Default values are defined into the harvester but can be overridden in the CKAN harvest source json configuration. Below the predefined defaults:

    {
    	'dataset_theme': 'OP_DATPRO',
    	'dataset_place': 'ITA_TRT',
    	'dataset_language': '{ITA,DEU,ENG}',
    	'agent_code': 'p_TN',
    	'frequency': 'UNKNOWN',
    	'agent_code_regex': {
    	    'regex': '\(([^)]+)\:([^)]+)\)',
    	    'groups': [2] # optional, dependes by the regular expression
    	},
    	'org_name_regex': {
    	    'regex': '([^(]*)(\(IPa[^)]*\))(.+)',
    	    'groups': [1,3] # optional, dependes by the regular expression
    	},
    	'dcatapit_skos_theme_id': 'theme.data-theme-skos',
    	'dcatapit_skos_places_id': 'theme.places-skos'
    }
  

  note: the agent_code_regex and org_name_regex (regex and/or group properties) must be changed if the corresponding 'organizationName' value change in the CSW metadata. Current default value has been set to match a string type like the following:

    Provincia Autonoma di Bolzano (IPa: p_bz) - Ripartizione 28 - Natura, paesaggio e sviluppo del territorio
  

  where the agent_code_regex match the IPa code p_bz and the org_name_regex match the organization name value Provincia Autonoma di Bolzano - Ripartizione 28 - Natura, paesaggio e sviluppo del territorio.

  the dcatapit_skos_theme_id and dcatapit_skos_places_id values correspond to the thesaurus identifiers speficied in the CSW metadata (related to the eu_theme.rdf and places.rdf themes) so if the metadata values change also these configuration options must be modified accordingly.

Below an example of configuration that shows all the available properties:

    {
		"private_datasets": "False", 
		"version": "2.6",
        "default_values": {
		    'dataset_theme': 'OP_DATPRO',
		    'dataset_place': 'ITA_TRT',
		    'dataset_language': '{ITA,DEU,ENG}',
		    'agent_code': 'p_TN',
		    'frequency': 'UNKNOWN',
		    'agent_code_regex': {
				'regex': '\(([^)]+)\:([^)]+)\)',
				'groups': [2] # optional, dependes by the regular expression
		    },
		    'org_name_regex': {
				'regex': '([^(]*)(\(IPa[^)]*\))(.+)',
				'groups': [1,3] # optional, dependes by the regular expression
		    },
		    'dcatapit_skos_theme_id': 'theme.data-theme-skos',
		    'dcatapit_skos_places_id': 'theme.places-skos'
        },
		"harvest_iso_categories": "True",
		"organisation_mapping": [
			{
				"key": "nome-org-1",
				"value": "organization-name-of-responsible-party-with-role-owner-1",
				"value_it":"it-localized-org-name-1",
				"value_de":"de-localized-org-name-1"
			},
			{
				"key": "nome-org-2",
				"value": "organization-name-of-responsible-party-with-role-owner-2",
				"value_it":"it-localized-org-name-2",
				"value_de":"de-localized-org-name-2"
			}
		],
		"group_mapping": {
			"farming": "ckan-group-1", 
			"utilitiesCommunication": "ckan-group-1", 
			"transportation": "ckan-group-1", 
			"inlandWaters": "ckan-group-2", 
			"geoscientificInformation": "ckan-group-1", 
			"environment": "ckan-group-2", 
			"climatologyMeteorologyAtmosphere": "ckan-group-1", 
			"planningCadastre": "ckan-group-1", 
			"imageryBaseMapsEarthCover": "ckan-group-3", 
			"elevation": "ckan-group-4", 
			"boundaries": "ckan-group-5",
			"structure": "ckan-group-6", 
			"location": "ckan-group-7", 
			"economy": "ckan-group-1",
			"society": "ckan-group-2",
			"biota": "ckan-group-2",
			"intelligenceMilitary": "ckan-group-1",
			"oceans": "ckan-group-4",
			"health": "ckan-group-5"
		},
		"ckan_locales_mapping":{
			"ita": "it",
			"ger": "de"
		},
		"default_license": "cc-zero"
	}

The ComunWeb harvester allows for these configuration entries:

• prefix: prefix for the identification code

  • default "ipa+:" using the iPA code of the harvester owner Organization

• ref_site: reference site for all harvested datasets

  • default https://www.comune.trento.it/Aree-tematiche/Open-Data

• geo_id: GeoGraphic code for all datasets (e.g. ITA_TRT)

  • if defined, will be stored as geographical coverage

• geo_uri: GeoNames URI for all datasets (e.g.: http://www.geonames.org/3165243)

  • if defined, will be stored as geonames reference

• poc_name: point of contact name

  • default Comune di Trento

• poc_ipa: point of contact iPA

  • default c_l378

• poc_mail: point of contact e-mail

  • default comurp@comune.trento.it

The datitrentinoit extension implements the ITranslation CKAN's interface so the translations procedure of the GUI elements is automatically covered using the translations files provided in the i18n directory.

To create a new translation proceed as follow:

1. Extract new messages from your extension updating the pot file:

   python setup.py extract_messages

2. Create a translation file for your language (a po file) using the existing pot file in this plugin:

   python setup.py init_catalog --locale YOUR_LANGUAGE

   Replace YOUR_LANGUAGE with the two-letter ISO language code (e.g. es, de).

3. Do the translation into the po file

4. Once the translation files (po) have been updated, either manually or via Transifex, compile them by running::

   python setup.py compile_catalog --locale YOUR_LANGUAGE

In order to update the existing translations proceed as follow:

1. Extract new messages from your extension updating the pot file:

   python setup.py extract_messages

2. Update the strings in your po file, while preserving your po edits, by doing:

   python setup.py update_catalog --locale YOUR-LANGUAGE

3. Once the translation files (po) have been updated adding the new translations needed, compile them by running:

   python setup.py compile_catalog --locale YOUR_LANGUAGE

Customized dataset license list is available in ckanext/datitrentinoit/licenses/licenses.json. To use it, set licenses_group_url config var to https://github.com/geosolutions-it/ckanext-datitrentinoit/tree/dcatapit/ckanext/datitrentinoit/licenses/licenses.json.