Coverage with pytest-cov

A crucial metric in testing is code coverage, indicating the percentage of code covered by at least one test. The goal is 100% code coverage.

Define an operate function that takes two parameters a and b and performs an addition or subtraction based on the operation.

# example.py
def operate(a, b, operation):
    if operation == "sum":
        return a + b
    if operation == "subtraction":
        return a - b
    else:
        raise ValueError(f"Invalid operation: {operation}")

Create a file with a test.

# example_test.py
import pytest
from example import operate

def test_operate():
    assert operate(2, 3, "sum") == 5

Install the pytest-cov package.

pip install pytest-cov

Running tests with --cov provides a coverage report. Here, it is 50%.

pytest --cov=example
# Name Stmts Miss Cover
# --------------------------------
# example.py 6 3 50%

It is 50% because only the sum operation is tested, not other paths. It is an incomplete test. To achieve 100%, test all paths:

• ➕ Addition.
• ➖ Subtraction.
• ⚠️ An unrecognized operator.

The following test covers all three paths.

def test_operate():
    assert operate(2, 3, "sum") == 5
    assert operate(2, 3, "subtraction") == -1
    with pytest.raises(ValueError):
        operate(2, 3, "multiplication")

Now, coverage is 100%.

pytest --cov=example
# Name Stmts Miss Cover
# --------------------------------
# example.py 6 0 100%
# --------------------------------
# TOTAL 6 0 100%

Although 100% coverage is ideal, it does not guarantee anything. Code with 100% coverage may still have bugs. A test shows the presence of a bug but not its absence.

Let us prove it. Imagine a mistake where the operation subtract uses + instead of -. This code is incorrect.

def operate(a, b, operation):
    if operation == "sum":
        return a + b
    if operation == "subtraction":
        return a + b # <- Mistake. WRONG!
    else:
        raise ValueError(f"Invalid operation: {operation}")

In the test, if 0 and 0 are subtracted, the result is 0. The correct result is achieved with incorrect code.

def test_operate():
    assert operate(2, 3, "sum") == 5
    
    # This assert passes, but there is a bug.
    assert operate(0, 0, "subtraction") == 0

This test passes, and coverage is 100%. This could falsely assure that the code is correct. But it is not.

The bug can be detected with the following code, a counterexample showing something is wrong.

def test_operate():
    assert operate(5, 1, "subtraction") == 4

It is important to test as many combinations as possible, including positive numbers, negative numbers, zero, decimals, and unexpected inputs.

To exclude a function from the coverage metric, use the comment # pragma: no cover. Python will not consider it in the percentage calculation.

def operate(a, b, operation): # pragma: no cover
    # ...

pytest generates interesting coverage reports. When you have hundreds of functions, a nice user interface to navigate is helpful.

pytest --cov=example --cov-report=html