This repo contains several scripts that can be used to run several static analysis tools over the modified ITC-Toyota benchmark available here.

In this README you can explore:

• Main directory stucture
• How to use the scripts
• Use case scenarios
• List of available tools

.
├── bash
├── benchmark.py
├── csv-data
├── latex
└── python

The folders are explained below:

• bash: contains a suite of scripts which are meant to count errors or warnings. The most important ones are gather-errors-by-line.sh which generates a CSV file containg all the errors in the Toyota-ITC benchmarks, and merge-csv.sh which generates a merged CSV where, for each variation in the benchmark we have the lines from both positive and negative tests, together with other metadata about defects.
• csv-data: this folder contains all the results generated by the scripts when running several analysis tools. Even if they are generated by the scripts, some of these CSV files were added on GitHub on purpose: they can be used to compute various statistics about the results generated by the static analyzers.
• latex: this is a folder which is filled by the scripts with latex reports.
• python: contains scripts for running various static analysis tools. Each static analysis tool has its own script which runs the tool and parses the output.

The main script is benchmark.py. In order for this script to work, you have to make sure that you have cloned the ITC-benchmarks repository from here. If you clone the ITC-benchmarks repository at the same directory level as this repository (i.e., you have the itc-testing-tools and itc-benchmarks side by side in the same folder) then no additional setup is required. Otherwise, you can edit benchmark.py, right at the begining, by changing the location of your itc-benchmarks folders. That is, you modify this code:

W_C_DEFECTS_DIR = "../itc-benchmarks/01.w_Defects/"
W_CPP_DEFECTS_DIR = "../itc-benchmarks/03.w_Defects_Cpp/"

WO_C_DEFECTS_DIR = "../itc-benchmarks/02.wo_Defects/"
WO_CPP_DEFECTS_DIR = "../itc-benchmarks/04.wo_Defects_Cpp/"

into something like this:

W_C_DEFECTS_DIR = "<CUSTOM_PATH>/itc-benchmarks/01.w_Defects/"
W_CPP_DEFECTS_DIR = "<CUSTOM_PATH>/itc-benchmarks/03.w_Defects_Cpp/"

WO_C_DEFECTS_DIR = "<CUSTOM_PATH>/itc-benchmarks/02.wo_Defects/"
WO_CPP_DEFECTS_DIR = "<CUSTOM_PATH>/itc-benchmarks/04.wo_Defects_Cpp/"

where <CUSTOM_PATH> is the custom path to itc-benchmarks.

python3 benchmark.py <csv-data-folder> setup

There no need to run command unless you want to use another folder instead of csv-data. The current repo already contains the results of running the above command in csv-data.

python3 benchmark.py <csv-data-folder> run <toolname>

Before you run this command make sure that your <toolname> is available in our list of tools and it is installed on your computer: you should be able to run <toolname> from the command line.

Example: python3 benchmark.py csv-data run cppcheck.

This command will generate a number of CSV files in the csv-data directory, namely a file called timing.csv and a folder called temp which will include other CSV files.

• The file timing.csv has only one row and three columns: tool name, number of seconds consumed by the tool to run the ITC test suite with defects, and number of seconds consumed by the tool to run the ITC test suite without defects. An example is available here.
• The temp folder includes CSV files containg the errors per line detected by the current tool, and temporary output files of the tool. You can check the contents of such a temp folder here.

Note: We have included in this repository a folder named csv-data which already contains the files generated by several tools. Thus, there is no need to install the tools on your local machine to generate various statistics (see the next use case scenarios).

python3 benchmark.py <csv-data-folder> stat <toolname>

The <csv-data-folder> should be populated by the command run, explained at step 1.

Example: python3 benchmark.py csv-data stat cppcheck

This command will generate statistics for the cppcheck tool, all stored in CSV files: the total statistics (c_total.csv and cpp_total.csv), the statistics per defects (c_defects.csv and cpp_defects), the statistics per subdefects (c_subdefects and cpp_subdefects).

Note: cppcheck is the tool alias from list of tools.

Note: Since we have already included a folder named csv-data which contains the files generated by several tools, this command can be run using the content of csv-data directly, without installing tools.

There are three types of latex reports that we generate:

1. Latex reports for all tools: python3 benchmark.py <csv-data-folder> total
2. Latex reports on defects for all tools: python3 benchmark.py <csv-data-folder> defects
3. Latex reports on subdefects for all tools: python3 benchmark.py <csv-data-folder> subdefects

All these will generate some latex tables in the latex directory which can be included in latex files.