Automatically find diff lines that need test coverage. Also finds diff lines that have violations (according to tools such as pycodestyle, pyflakes, flake8, or pylint). This is used as a code quality metric during code reviews.
Diff coverage is the percentage of new or modified lines that are covered by tests. This provides a clear and achievable standard for code review: If you touch a line of code, that line should be covered. Code coverage is every developer's responsibility!
The diff-cover
command line tool compares an XML coverage report with the output of git diff
. It then reports coverage information for lines in the diff.
Currently, diff-cover
requires that:
- You are using
git
for version control. - Your test runner generates coverage reports in Cobertura XML format.
Cobertura XML coverage reports can be generated with many coverage tools, including:
- Cobertura (Java)
- Clover (Java)
- coverage.py (Python)
- JSCover (JavaScript)
diff-cover
is designed to be extended. If you are interested in adding support for other version control systems or coverage report formats, see below for information on how to contribute!
To install the latest release:
To install the development version:
- Set the current working directory to a
git
repository. - Run your test suite under coverage and generate a Cobertura XML report. For example, if you are using nosetests and coverage.py:
This will create a coverage.xml
file in the current working directory.
NOTE: If you are using a different coverage generator, you will need to use different commands to generate the coverage XML report.
- Run
diff-cover
:
This will compare the current git
branch to origin/master
and print the diff coverage report to the console.
You can also generate an HTML version of the report:
In the case that one has multiple xml reports form multiple test suites, you can get a combined coverage report (a line is counted as covered if it is covered in ANY of the xml reports) by running diff-cover
with multiple coverage reports as arguments. You may specify any arbitrary number of coverage reports:
You can use diff-cover to see quality reports on the diff as well by running diff-quality
.
Where tool
is the quality checker to use. Currently pycodestyle
, pyflakes
, flake8
, pylint
, checkstyle
, checkstylexml
are supported, but more checkers can (and should!) be integrated. There's no way to run findbugs
from diff-quality
as it operating over the generated java bytecode and should be integrated into the build framework.
Like diff-cover
, HTML reports can be generated with
If you have already generated a report using pycodestyle
, pyflakes
, flake8
, pylint
, checkstyle
, checkstylexml
, or findbugs
you can pass the report to diff-quality
. This is more efficient than letting diff-quality
re-run pycodestyle
, pyflakes
, flake8
, pylint
, checkstyle
, or checkstylexml
.
# For pylint < 1.0
pylint -f parseable > pylint_report.txt
# For pylint >= 1.0
pylint --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" > pylint_report.txt
# Use the generated pylint report when running diff-quality
diff-quality --violations=pylint pylint_report.txt
# Use a generated pycodestyle report when running diff-quality.
pycodestyle > pycodestyle_report.txt
diff-quality --violations=pycodestyle pycodestyle_report.txt
Note that you must use the -f parseable
option to generate the pylint
report for pylint versions less than 1.0 and the --msg-template
option for versions >= 1.0.
diff-quality
will also accept multiple pycodestyle
, pyflakes
, flake8
, or pylint
reports:
If you need to pass in additional options you can with the options
flag
By default, diff-cover
compares the current branch to origin/master
. To specify a different compare branch:
To have diff-cover
and diff-quality
return a non zero status code if the report quality/coverage percentage is below a certain threshold specify the fail-under parameter
The above will return a non zero status if the coverage or quality score was below 80%
Issue: diff-cover
always reports: "No lines with coverage information in this diff."
Solution: diff-cover
matches source files in the coverage XML report with source files in the git diff
. For this reason, it's important that the relative paths to the files match. If you are using coverage.py to generate the coverage XML report, then make sure you run diff-cover
from the same working directory.
Issue: GitDiffTool._execute()
raises the error:
This is known to occur when running diff-cover
in Travis CI
Solution: Fetch the remote master branch before running diff-cover
:
Issue: diff-quality
reports "diff_cover.violations_reporter.QualityReporterError: No config file found, using default configuration"
Solution: Your project needs a pylintrc file. Provide this file (it can be empty) and diff-quality
should run without issue.
Issue: diff-quality
reports "Quality tool not installed"
Solution: diff-quality
assumes you have the tool you wish to run against your diff installed. If you do not have it then install it with your favorite package manager.
The code in this repository is licensed under the Apache 2.0 license. Please see LICENSE.txt
for details.
Contributions are very welcome. The easiest way is to fork this repo, and then make a pull request from your fork. The first time you make a pull request, you may be asked to sign a Contributor Agreement.
diff-cover is written to support many versions of python. The best way to set your machine up for development is to make sure you have tox
installed which can be installed using pip
.
Now by simply running tox
from the project root you will have environments for all the supported python versions. These will be in the .tox
directory.
To create a specific python dev environment just make a virtualenv for your python version and then install the appropriate test-requirements file.
For example, setting up python 3:
Shout out to the original author of diff-cover Will Daly and the original author of diff-quality Sarina Canelake.
Originally created with the support of edX.