Tools for automated model validation in Open Source Brain projects, but which can be used for testing model behaviour on many simulation engines, both locally and on Travis-CI.

To see this framework in action, click on some of the green buttons below:

OSB project	Tests on Travis-CI	.travis.yml script
FitzHugh Nagumo		.travis.yml
Auditory cortex network		.travis.yml
SBML Showcase		.travis.yml

Any OSB project can have automated testing incorporated (currently, only LEMS and NEURON implementations can be tested thoroughly, though). In order to enable it:

• create a dry-run OSB Model Test (omt) file, containing the target model file and the simulator, e.g.

  echo -e "target: LEMS_hh_nostim.xml\nengine: LEMS" > NeuroML2/hh.omt
  

• copy the travis config file template to the projects' root dir, and rename it to .travis.yml (don't forget the leading dot!):

  cd OSB_proj_dir
  wget https://raw.githubusercontent.com/borismarin/osb-model-validation/master/templates/travis.yml.template
  mv travis.yml.template .travis.yml
  

Once these changes are pushed to the github repo, travis will run the tests automatically. You can check the results at http://travis-ci.org/OpenSourceBrain/project_name (and ideally add a travis build badge to the projects' Readme.md file). Check this project for a working example.

After this initial simple test passes, you can start writing more elaborate tests by creating a Model Emergent Properties (mep) file and corresponding omt tests. Examples of supported constructs can be found [here] (https://github.com/borismarin/hh-testing). Notice that omt tests will be automatically discovered by travis, regardless of their location.

If you are wise and want to run the tests locally before submitting your changes to github, install the omv python package

pip install git+https://github.com/OpenSourceBrain/osb-model-validation.git

This package provides the omv command line utility:

• omv test <filename.omt>, which runs a specific test locally

• omv all, which recursively discovers all .omt files in the project and runs them (this is the command used by travis).

• omv validate_mep <filename.mep>, to validate a .mep file against the current mep schema.

• additional options: omv -h

    OpenSourceBrain Model Validation and Testing
    ============================================
    
    Usage:
      omv all [-V | --verbose]
      omv test <testMe.omt> [-V | --verbose]
      omv autogen [options]
      omv install <backend>
      omv list-backends
      omv validate-mep <mepfile>
      omv validate-omt <omtfile>
      omv (-h | --help)
      omv --version
  
    Options:
      -h --help     Show this screen.
      -d --dryrun   Generate dry-run tests only [default: False].
      -V --verbose  Display additional diagnosis messages [default: False].
      --version     Show version.
      -y            Auto-select default options (non-interactive mode)
  

Evidently, we will provide a validate_omt script as soon as we agree on its schema.

In other words, to run all omt tests inside a project:

cd OSB_proj_dir
omv all