Interoperability test harness for the Southampton Provenance Suite.

The test harness includes support for:

• Software:
  • ProvPy
  • ProvToolbox
• Services:
  • ProvTranslator
  • ProvStore

The test harness runs under Python 2.7+ and Python 3.

The test harness can be run under Travis CI. See, for example, the following repositories, each of which contain a Travis CI test job to run interoperability tests for each component above:

Component	Travis CI job repository	Travis CI job	Build status
ProvPy prov-convert	GitHub	TravisCI
ProvToolbox provconvert	GitHub	TravisCI
ProvTranslator	GitHub	TravisCI
ProvStore	GitHub	TravisCI

Here we have set up one repository per component so we can have one test job per component. There is no reason, though, why a single test job cannot run all the tests for all the components, to reduce the number of repositories needed.

Running the interoperability tests under Travis CI require you to:

• Create a ProvStore API Key:
  • Log in to ProvStore
  • Select Account => Developer Area
  • You will see your API key
• Define a Travis CI variable, PROVSTORE_API_KEY holding your ProvStore user name and API key:
  • Visit your job's settings page in Travis CI
  • Select settings
  • Click Environment Variables
  • Click Add a new variable
  • Name: PROVSTORE_API_KEY
  • Value: user:qwert12345
  • Ensure Display value in build logs is not selected
  • See define variables in repository settings.

Travis Client explains how to automatically trigger re-runs of interoperability tests without having to either submit a GitHub pull request on the repository holding the Travis CI configuration file, or going via Travis CI's web interface.

Jenkins is a popular, open source continuous integration server that runs under Java. See Running the interoperability test harness under Jenkins which includes an example of running all the interoperability tests from within a single Jenkins job.

The test harness can be run stand-alone. See Running the interoperability test harness standalone.

There are a number of options for what versions of tools and services are tested. For example:

• ProvPy
  • pip package
  • GitHub repository stable branch (e.g. 1.3.2)
  • GitHub repository latest version (e.g. master)
  • Running under Python 2.7 or Python 3.4
• ProvToolbox:
  • GitHub repository stable branch (i.e. master)
  • GitHub repository stable branch source code ZIP
  • Maven binary release ZIP
  • rpm package
• ProvStore
  • Live, public service.
  • Development version of service hosted locally.
• ProvTranslator
  • Live, public service.
  • Development version of service hosted locally.

It is possible (and, indeed, desirable) to set up a Travis CI or Jenkins job for each component to be tested, deploying it and any required comparators, so the test dashboards shows the status of the interoperability testing for that component alone, rather than the status of the tests across all components.

For Jenkins, there is also the option of whether the test jobs clone the repositories of each component every time, or update existing clones that the test jobs assume you have cloned into a directory to which Jenkins has access.

What combination of options is used is purely a configuration issue, relating to how the Travis CI or Jenkins jobs are written and how the test harness is configured (e.g. whether it uses prov-convert or python ProvPy/scripts/prov-convert, or a public or private service URL).

The interoperability test harness includes unit tests for the harness itself. This respository contains a TravisCI, .travis.yml, job configuration file to run these unit tests.

For details, see Interoperability test harness design and implementation.

To create API documentation in apidocs/_build/html:

$ pip install sphinx
$ make apidocs

Developed by The Software Sustainability Institute and the Provenance Tool Suite team at Electronics and Computer Science at the University of Southampton.

For more information, see our document repository.

The code is released under the MIT license.