The tdh.metadata package is a Python-based extension product for the Plone CMS and upon installation, enables Plone to store data and metadata for integration with the Australian National Data Service (ANDS) by way of Resource Interchange Format for Collections and Services (RIF-CS) metadata production. By providing the ANDS metadata harvester service with the relevant URL for the local metadata repository on a suitable Plone site, metadata from the Plone site can be recorded for searching by their service. The metadata is suitable for integration into other services that accept the RIF-CS metadata standard.
This project aims to create a Plone product that will provide the user with a simple metadata repository that allows users to create metadata records for collections that conform to ANDS minimum metadata requirements. It expects to be connected to data sources to provide party and activity metadata related to the collections. The product include the capability to search the repository based on the meta-data fields. It includes the ability to transform the collected metadata into RIF-CS records and can respond to DirectHTTP harvesting requests from ANDS.
This project is supported by the Australian National Data Service (ANDS) through the National Collaborative Research Infrastructure Strategy Program and the Education Investment Fund (EIF) Super Science Initiative, as well as through the Queensland Cyber Infrastructure Foundation (QCIF).
- Plone 4.1+ buildout-based installation (see plone.org)
- Ability to install system libraries and dependencies from the following Python packages:
- Plone
- Shapely
- collective.geo.bundle
- JPype
- Database drivers (cx_Oracle, for instance) and associated libraries
- Java JVM installed (OpenJDK and Oracle compatible)
- Suitable back-end databases (SQL preferred) as supporting data services.
This code utilises Git submodules so if you are cloning this project from GitHub, then you will need to run:
git submodule init
git submodule updateafter cloning in order to have all relevant files needed.
Within a Plone 4.1 or later installation, specify tdh.metadata as an egg under the [instance] section of your buildout. To download from Google Code, specify a special egg section like so and include the reference to the download in your instance section:
[buildout]
extends =
https://raw.github.com/gist/3057156/
parts +=
...
tdh-metadata
[instance]
...
eggs +=
${tdh-metadata:eggs}
[tdh-metadata]
recipe = zc.recipe.egg
eggs = tdh.metadata
find-links =
http://xyz.com/tdh.metadata-src-1.0.tar.gzWithin the buildout, you need to specify additional Zope configuration under the instance section as zope-conf-additional for database configurations and where ANDS' RIF-CS can be found:
[instance]
...
zope-conf-additional =
<product-config tdh.metadata>
research-services__db-type oracle
research-services__db-host my-db.example.com
research-services__db-host-port 1234
research-services__db-service service_pid
research-services__db-user database_user
research-services__db-password XYZ
rifcs-api-location ${rifcs-api:destination}/${rifcs-api:filename}
</product-config>
[rifcs-api]
recipe = hexagonit.recipe.download
url = http://services.ands.org.au/documentation/rifcs/java-api-1.2/rifcs-api.jarAfter specifying these customisations to your buildout, re-run your bin/buildout in your Plone instance's home directory to enable Plone to see the tdh.metadata add-on.
Once done, and assuming no errors were encountered on buildout, restart your Plone instance. When restarted, click to Site Setup and 'Add-on Products' in the control panel listing. Locate Tropical Data Hub Metadata Infrastructure in the products listing and install this.
Installation will automatically create a pre-configured Data Record Repository instance at http://localhost:8080/Plone/data (where localhost:8080/Plone is your site's web address). You can add new Data Record Repository instances wherever you'd like to in your site by way of the 'Add new' menu in your site. Similarly, this pre-existing instance can be customised accordingly without needing to be removed first.
After adding Dataset Records to the Data Record Repository on the site, RIF-CS records can be exported. Either a single record's metadata can be exported through the 'Download RIF-CS' link when viewing the record (or else appending /@@rifcs to the URL). Normal users can export this metadata for their own purposes.
For site administrators and the ANDS metadata harvester, all metadata records can be exported for the entire repository by appending /@@rifcs to the URL for the Data Record Repository (for example, http://localhost:8080/Plone/data/@@rifcs). Only users with the Manager role can access this special address as exporting all metadata for a site with many records can take time and resources to process. The ANDS metadata harvester is granted special permissions on this browser view by way of IP address.
In all cases, the RIF-CS file will incorporate all related Party and Activity records within the same file for completeness. Specifically, Party records will be included for users who have been specified as being related to the metadata, and Activity records will be included for other research and grant activities that have been associated with the exported records.
To make downloading the RIF-CS file easier, use @@rifcs?download and your browser will automatically prompt you to download the given file with a suitable filename.
Formatting of RIF-CS key identifiers for Collections, Party objects and Activity records can be customised within tdh.metadata.config. Specifically, this can take place by changing the RIFCS_KEY variable in this module.
Customise the JVM_PATH variable inside tdh.metadata.config to specify a custom location of a given JVM.
The tdh.metadata.static package features several resource files, including CSS and Javascript. Whilst these files are generally applicable to all applications of this Plone add-on, there are some inclusions of JCU-specific configuration. In particular, JCU themed colours have been included to demonstrate how to customise colouring on the relevant forms. It is expected that an installer of this product will proceed and add their own styles and scripts to suit their own needs, following the examples set forth for them.
This package features the latest ANZSRC codes for research (Fields of Research [FoR], Socio-Economic Outcomes [SEO]) as of the time of writing. Should these codes need to be added to or updated, include new CSV files in the tdh/metadata/browser/*_codes.csv files, where * is for with FoR codes and seo with SEO codes. These files should be formatted thusly:
"010000","MATHEMATICAL SCIENCES"
"010100","Pure Mathematics"
"010101","Algebra and Number Theory"
"010102","Algebraic and Differential Geometry"
...Being a complete listing of all codes and any subheads being specified with multiple zeroes at the end. In the above example, the 010000 code is for the top level of 01 and the 010100 code is the sub-level 0101 within the top-level of 01. All other 6 digit codes are the actual FoR or SEO codes. As mentioned, with new codes, update these files in the same format and location and restart the Plone process to clear any caches held.
It is to be expected that ANDS release new RIF-CS Java APIs and that the specifications of RIF-CS will eventually change. In order to download a new RIF-CS API, add a new URL to the rifcs-api section of your buildout (as specified above) and re-run your bin/buildout executable. This will notice the change in file URL and re-download the new file. In the case this is not detected, remove the parts/rifcs-api directory, empty your cache directories, and re-run buildout. Upon restarting your Plone instance, the new API will be in use for any RIF-CS output produced.
Note: Be aware that any change to an API may introduce backwards-incompatible changes and further modifications may need to be made to fix RIF-CS functionality. Updates will be released to public repositories to this code as required and as is possible. Contact is welcomed to our group if you have issues to report. In addition, RIF-CS produced by this add-on is only guaranteed in our specific test situations. Users may be able to include non-standard metadata and thus break RIF-CS structure and guidelines. This is unlikely given how the cod e has been developed, but may still be possible.
The example code provided integrates user information and grant information from JCU corporate databases. Users wishing to utilise similar should customise SQLAlchemy database structure in tdh.metadata.sources.* classes. Refer to the BaseQuerySource class in tdh/metadata/sources/base.py for documentation on how to create your own database query sources, and other files within the sources package for how to integrate your database. Depending on the backend database you are using, and other specialist requirements, such as database structure, hostnames, and more, you may need to further customise the code present and potentially installation additional packages to support your environment.
Additional metadata can be captured against Dataset Record content objects by customising the relevant schema or schemas within tdh.metadata.dataset_record. The main over-arching schema is that of IDatasetRecord, which utilises zope.schema field types to produce a fully-featured content type for the capture of metadata. Additional fields can be added to this schema and the user will be prompted for the input within the web-based interface for adding and editing Dataset Records. Please keep in mind that adding new required fields or changes to existing fields may require processing across all exisiting data records, if any exist. Custom migration steps may need to be written for this to take place. Once new fields are in place, additional provisions may be made for inclusion of field values within RIF-CS and elsewhere in the add-on.
Content types are developed with Dexterity (plone.app.dexterity; http://plone.org/products/dexterity/) -- refer to its product page for more information about development within this framework. Schemas and forms use similar patterns to that of z3c.form, integrated with plone through plone.app.z3cform. Look at these products' relevant pages for more information about the technologies being used and how to extend them.
Various fields within the Dataset Record schema have vocabularies, which are used to populate the various options that are available for selection against each field. These vocabularies translate into visual options, such as checkboxes and select lists within HTML, and are used by internal validation to ensure values against fields conform to values within the lists of terms. Most vocabularies are stored within tdh.metadata.vocabularies and can be customised here, if required. Some vocabularies, such as RELATIONSHIPS, DESCRIPTIONS, and COLLECTION_TYPES are extracted from the official ANDS RIF-CS listings for vocabularies for certain aspects, and others, such as DATASET_LOCATIONS, are specifically JCU-related. All can be customised to suit, but again, be aware that some code, particularly RIF-CS code, may rely on any or all of the vocabularies being present.
In order to support the highly-customised add and edit forms for Dataset Records within the metadata repository product, several customised widgets are present within tdh.metadata.widgets. In particular, the FoR and SEO code fields have custom collective.z3cform.datagridfield DataGridField widgets that enable them to select codes from drop down menus. Other minor customisations to pre-existing widgets are carried out in the various functions against the Dataset Record forms within tdh.metadata.dataset_record -- in particular, functions such as update(), render(), groups(), updateWidgets() and for fields contained within DataGridField widgets, the datagridInitialise and datagridUpdateWidgets functions.
Presently, the metadata functionality provided by this package offers the ability to store small amounts of data alongside the metadata records being handled. It does so by way of BLOB (Binary Large OBject) storage on the server, within an efficient filesystem structure. However, this file storage functionality should be considered to be very basic in that handingly excessively large files (1GB+) or large quantities of files will likely be either not possible or extremely unwieldy for users. This backend storage may be replaced with a more suitable solution and this can be achieved by integrating a different type of field. At present, NamedBlobFile from plone.namedfile.field is being used and this is the stock-standard solution for storing files within Plone as of the time of writing.
Bug reports and suggestions for improvement are welcomed at the contact address provided within this package. If you require more information about the integration of this package into a new installation, refer to the same address.
The source code for this package is available on GitHub at https://github.com/jcu-eresearch/tdh.metadata, where developers are invited to contribute.