Objective: test the Qucs (GUI) and Qucsator (simulator).

The test projects under the testsuite directory are all run by default.

To skip failing projects use the --skip option with a list of projects to be ignored.

usage: run.py [-h] [--prefix PREFIX] [--qucs] [--qucsator] [-p]
              [--add-test ADD_TEST] [--skip SKIP] [--project PROJECT]
              [--compare-qucsator COMPARE_QUCSATOR [COMPARE_QUCSATOR ...]]
              [-v [VERBOSE]] [--reset] [--timeout TIMEOUT]

Qucs testing script.

optional arguments:
  -h, --help            show this help message and exit
  --prefix PREFIX       prefix of installed Qucs (default: /usr/local/bin/)
  --qucs                run qucs tests
  --qucsator            run qucsator tests
  -p                    run qucs and prints the schematic to file
  --add-test ADD_TEST   add schematic to the testsuite
  --skip SKIP           file listing skipped test projects
  --project PROJECT     path to a test project
  --compare-qucsator COMPARE_QUCSATOR [COMPARE_QUCSATOR ...]
                        two full paths to directories containing qucsator
                        binaries for comparison test
  -v [VERBOSE], --verbose [VERBOSE]
                        increase verbosity: 0 = progress and errors, 1 = all
                        info. Default is low verbosity.
  --reset               Reset (overwrite) data and log files of test
                        projects.Run qucsator given with --prefix.
  --timeout TIMEOUT     Abort test if longer that timeout (default: 15 s).

• The script is rather verbose and the progress is printed to the terminal (with colors, yeah!).

• A table with the run project, Qucs schematic version and simulation runtim is saved automatically and time-stamped.

• A report table with all components available to Qucs and the coverage with respect to the types of simulations.

Example of running the test-suite for Qucsator while skipping a few test projects:

$python run.py --prefix /home/user/local/qucs-master/bin/ --skip skip.txt --qucsator

See below examples of printing schematics to file (pdf):

• printing all
• skipking a list of projects
• a single project from the testsuite

$python run.py --prefix /home/user/local/qucs-master/bin/ --p
$python run.py --prefix /home/user/local/qucs-master/bin/ --skip skip_print.txt -p
$python run.py --prefix /home/user/local/qucs-master/bin/ --p --project AC_SW_resonance_prj

Qucs is featured with an equation system available in the schematic area.

The script below tests several of the built-in functions.

In short, it works as follows:

• For each entry in src/application.h.
• Generate random (or constrained) arguments (double, complex, matrix...)
• Create a test equation and computes the expected result (with Python and Numpy).
• Save test equation into a netlist
• Run the netlist and read the simulator output.
• Compare simulator output to the expected result.

Run it with:

$ python run_equations.py /home/user/git/qucs-clone/

Mac OSX 10.8.5, Python 2.7.5, Numpy 1.7.1

Debian 6.0.7 (Squeeze), Python 2.6.6, Numpy 1.4.1

You might need to install argparse which is not included with Python 2.6. Something like:

$sudo apt-get update
$sudo apt-get install python-setuptools
$sudo easy_install pip
$sudo pip install setuptools --no-use-wheel --upgrade
$sudo pip install argparse

• List of Qucs components

• Testsuite simulation report

• Testsuite component coverage report

• Currently failing projects

• Output without errors

• Output with errors

under construction

It considers every directory under testsuite as a project to be run. Then,

• It run tests to check the schematic to netlist conversion, using the Qucs-GUI code.

• It compares (diff) a reference netlist with the generated netlist on the step above.

• A simulation is run for each project

• Results are for the dependent variables are compared with reference results

• It collects : failed netlist diffs, simulation runtime, failed variable comparison, output and warnings (not used currently).

Running Qucs and Qucsator from the command line makes it easy to automated testing.

It is possible to use Qucs GUI from the command line to read a schematic and create the netlist.

$qucs -n -i file.sch -o netlist.txt

The Qucs GUI can be used from the command line to read a schematic and crete a printout (feature in development).

$qucs -p -i file.sch -o netlist.pdf

Having a netlist file it is possible to run the Qucsator backed to read the netlist and create the results (and log messages).

$qucsator -i netlist.txt -o result.dat

• Tests should not take too much time to run.
• Tests should focus into a particular feature which can be tested in isolation
• Tests could be categorised, ex. DC, AC analysis, a component...
• Tests should exercise corners that are crucial for stable releases.
• Before releasing, the test set should be run with no regressions.
• Git can be used to version test result (timing history?)
• Use git diff -G '^([^#])' to ignore the changes on the header line of the netlist files.

A test case is stored on a directory, similar to a Qucs project

Suggested contents:

• test.txt
  • test config file
• Schematic (.sch)
  • Test schematic (name convention)
  • Subcircuits dependencies
• Netlist (.txt)
  • reference netlist
• Result (.dat)
  • reference results
• Optional
  • reference log file (.txt)
  • PDF dump of schematic/data display

What about asco, verilog, vhdl?

The test-case is run depending on the available inputs. The main steps are listed below

• convert schematic into test netlist
• compare reference netlist with test netlist
• run netlist (or test netlist) store test result
• compare reference result with test result

Schematic ==> Netlist

input:  [name].sch
output: netlist_test.txt

Netlist ==> Result

input:  netlist.txt | netlist_test.txt
output: result.dat | result_test.txt
        log.txt
        [asco.log]

compare:
  - netlist X netlist_test
  - result  X result_test

Simple diff, skip first line. Retun different lines.

Qucsator has the option to check a netlist. ?? what for ??

It uses the numpy allclose function to check how close together are all the dependent variables (arrays) on the reference and test data files.

Tests should be similar to projects. So making it easy to open on Qucs an use as examples.

qucs-test/
  run.py

  testsuite
    TR_collpits_prj/
      collpits.sch


    AC_bandpass_prj/
      bandpass.sch
      opa.sch
      bandpass.dat
      netlist.txt
      log.txt

    ASCO_AC_fobar_prj/

    VERILOG_

    VHDL_

Notes:

• Need to identify the test schematic among the subcircuits
  • directory name, and the name of the schematic to be netlisted/simulated.

Whish-list

• what about verilog, vhdl, asco?
• pack/unpack tarballs/examples?

• Create/update list of test/prj:

  • check consistency of the test-suite.
  • it looks what tests are available.
  • it adds new tests as they are found.
  • it preserves the ones that are commented out #, give warning

• Test behavior depends on the provided input/reference files

  • if [sch] provided -> netlist, check
  • if [sch, dat] provided -> nelist, [check], simulate, check
  • if [sch, dat, log] provided -> do netlist, [check], simulate, check, statistics

• Give option to create reference files when missing.

• Table Report pass, fail, missing resource

• Catch errors, time out, kill process if taking more than expected

• Report checks, time improvement/regression

• Html report?