This a regression test suite for JKind. This suite is built in C-Python v 3.4.2 using only the libraries from the standard distribution from python.org.

• C-Python v3.4.2 or later (recommend retrieving from python.org)
• Yices distribution on system path (this suite tested against v1.0.40)
• Yices2 distribution on system path (this suite tested against v2.4.1)
• CVC4 distribution on system path (this suite tested against v1.4)
• Z3 distribution on system path (this suite tested against 4.4.2)

This suite runs JKind against Lustre files using varying arguments and solvers. The suite checks that the JKind results do not change given a different set or arguments. The arguments applied to each Lustre file include the desired solver and any additional flags (see the JKind help description for all options). Each Lustre file is run against all the arguments and if any irregularities are encountered these are logged as failures.

The results are determined from the XML file generated by JKind for each Lustre file and argument set. The results of the "Property" elements are saved internally and then compared against each other once all the argument sets have been run on the file.

The final results are reported on the command line or log file called "jkr.log" as:

<unittest.runner.TextTestResult run=1 errors=0 failures=0>

In this instance the "run=1" indicates that one Lustre file was tested and no irregularities were found.

If irregularities were found you may encounter an output such as:

<unittest.runner.TextTestResult run=2 errors=0 failures=4>

In this case the "run=2" indicates that two Lustre files were tested. The "failures=4" indicates that four sets of arguments resulted in an irregularity. Looking above in the command line or log file output will show the particular argument sets that were irregular.

Along with the final report line seen above the jkr.log will also show each of the test cases that were executed, the command line strings called for each test case and the durations. If any irregularities are found the details are printed just below the command line string in question for easier troubleshooting. A subdirectory "jkr" is also generated, containing an individual log file for each lustre file tested.

• If there are a mis-matched number of Properties between argument set executions. There is no further attempt to match Properties for further checks.
• If two same named Properties have different Answer elements. The exception to this rule is if one of the Property Answer elements is "unknown", this case is deemed to be ok.
• If two same named Properties have Answer element = "falsifiable", the K elements are tested as follows:
  • If either src1 or src2 are NOT from bmc or pdr will fail and complain
  • If src1==bmc and src2==pdr, then k1==k2; otherwise fail
  • If src1==src2==pdr, don't care. Pass unconditionally.
  • If src1==bmc and src2==pdr, then k1 < k2; otherwise fail
  • If scr1==pdr and src2==bmc, then k1 > k2; otherwise fail

Command Line usage (also available from the python help command):

$ python jkindtest.py <-file FILE OR -dir DIR> <-argfile FILE> <-logfile FILE> <--gui> <--recur>

1. -file : One of the mandatory-ish arguments. Specify the Lustre file to test. May substitute -dir or --gui.
2. -dir : One of the mandatory-ish arguments. Specify the folder/directory to search for *.lus files to test. May substitute -file or --gui.
3. --gui : Optional flag. If this is specified will launch a simple Tk GUI. Special note that the -dir/-file and -logfile may be specified at the command line with the --gui flag to "pre-load" the options.
4. -argfile : Optional. Specify an alternate XML Argument File. The default is ./test_arguments.xml.
5. -logfile : Optional. Specify a text file to log the output. Default is stdout (console).
6. -jkind : Optional. Specify an alternate JKind to run (default is what is on system path)
7. -java : Optional. Specify an alternate Java executable (default is what is on system path)
8. --recur : Optional flag. If -dir was specified, this flag indicates that the search for Lustre files should recurse through sub-directories.
9. --quite : Optional flag. Suppresses non-failing warnings or errors.

The GUI gives menu options to manually specify a file or directory and a log file. When the execute button is pressed will give a running update on the file count, the file under test and the argument set under test. The GUI can be useful for monitoring progress when logging to file as the stdout is redicted to the file and does not update the console.

Note that unless specified it is expected that the default_args.xml file is at the root level (included as part of this git repo). This XML file specifies the arguments to be applied to the Lustre files. The arguments are specified by one or more elements named ArgumentGroup. The test suite will read the XML file and then construct the command line arguments to call based upon the possible combinations of the Argument Groups.

<Configuration>

  <!-- Solvers and PDR-only -->
  <ArgumentGroup>
    <arg>-solver yices</arg>
    <arg>-solver yices2</arg>
    <arg>-solver cvc4</arg>
    <arg>-solver z3</arg>
    <arg>-solver smtinterpol</arg>
    <arg>-no_bmc -no_k_induction -no_inv_gen</arg>
  </ArgumentGroup>
  
  <!-- With and without PDR -->
  <ArgumentGroup>
    <arg&rt</arg&rt
    <arg>-pdr_max 0</arg>
  </ArgumentGroup>
  
  <!-- Always product inductive counter-examples -->
  <ArgumentGroup>
    <arg>-induct_cex</arg>
  </ArgumentGroup>
  
  <!-- Miscellaneous options -->
  <ArgumentGroup>
    <arg&rt</arg&rt
    <arg>-smooth</arg>
    <arg>-interval</arg>
    <arg>-support</arg>
  </ArgumentGroup>
  
</Configuration>

A separate Python unittest based suite is included as a self test. This test suite interacts with various parts of the jktest package modules to test the functionality. This includes reading of an arguments xml file, results xml files and testing of expected and read results. Tests include happy path and failure conditions. Two example Lustre files, pre.lus and tuple.lus, are run in their entirety as well.

Autogenerated HTML documentation for the python modules exists in the subfolder:

./doc/_build/html/index.html

This test suite uses the standard C-Python distribution v3.4.2 obtained from python.org. No libraries other than what is included in the basic distribution are used. Must use v3.4.2 or later as this suite makes use of the unittest.TestCase.subTest.

Typical usage is through the root level jkindtest.py module which accepts the command line arguments. The guts of the suite are in the jktest Package and may be imported in to other applications, the entry execution point being jktest.testsuite.py. Note that if you access testsuite.py directly you must first instantiate and populate the setup class within jktest.config.py.

The test suite itself is based on the Python unittest library. Typical usage of the unittest framework is to define unique, stand-alone test cases that are independent of other tests and data. Python unittest wants to register these defined classes and does not inherently accept run-time inputs. This test suite however wants to execute the same test steps over different inputs (files and arguments) and this poses a couple special challenges to using the unittest framework.

The goal is to iterate over multiple files with multiple arguments and not have to manually write a test case for each file and argument set. To make this possible our test suite makes use of a class factory. A single test case base class is defined in the test suite. The class factory dynamically creates sub-classes and adds the file and argument set to the newly created sub-classes. It is these dyncamically created sub-class definitions that are fed to the unittest.TestSuite and this is how we are able to iterate through files and arguments while still using the built-in unittest framework.

The GUI is located in the gui package, but the interface to drive the GUI is located in jktest.guiIF.py. This interface class accepts updates from the test suite and attempts to pass them along. If the GUI has not been instantiated the interface class will catch and pass any exceptions.