Other pytest features

pytest has a wide variety of plugins to extend its functionality. Some are default, others require installation.

• ⏭️ skip
• ⏳ timeout
• ⚠️ flaky
• ⏱️ freeze time
• 📏 benchmark

To install them:

pip install pytest-timeout
pip install pytest-rerunfailures
pip install pytest-freezegun
pip install pytest-benchmark

⏭️ skip: Skip a test so it does not run. Remember, if a test fails, the solution is not to skip it but to understand and fix the problem.

@pytest.mark.skip(reason="Does not run")
def test_that_does_not_run():
    pass

Running the tests shows which are skipped and why.

iss_test.py::test_that_does_not_run SKIPPED (Does_not_run)

⏳ timeout: Set a maximum time for tests. If not finished within that time, it fails.

In this example, the test should pass, but it takes 2 seconds, and the timeout is 1 second, so it fails.

import time
@pytest.mark.timeout(1)
def test_with_timeout():
    time.sleep(2)
    assert 1 == 1

Running it results in:

iss_test.py::test_with_timeout FAILED

⚠️ flaky: Mark a test as flaky or unstable, if it fails sometimes. Specify the maximum retry attempts. This will try to run the test multiple times. If not passed after all attempts, it fails.

Tests should be deterministic, but sometimes flaky tests are unavoidable.

import random

@pytest.mark.flaky(reruns=10)
def test_flaky():
    die = random.choice([1, 2, 3, 4, 5, 5, 6])
    assert die == 1

Running it shows multiple attempts, up to 10. If at least one passes, it is considered passed.

iss_test.py::test_flaky RERUN
iss_test.py::test_flaky RERUN
iss_test.py::test_flaky PASSED
1 passed, 2 rerun in 0.09s

⏱️ freeze_time: Freeze the current date, similar to a mock.

@pytest.mark.freeze_time('2023-01-01')
def test_today_date():
    assert date.today() == date(2023, 1, 1)

📏 benchmark: Measure test execution time.

def heavy_task():
    return sum(i * i for i in range(100000000))

@pytest.mark.benchmark()
def test_heavy_task_benchmark(benchmark):
    result = benchmark(heavy_task)
    assert result == 333333328333333350000000

The time taken is displayed when the test runs, averaged over multiple runs. In this case, it takes about 4.4 seconds.

Name (time in s) Min Max Mean
----------------------------------------------
test_heavy_task_benchmark 4.48 4.61 4.54

You can set a condition with an assert where the test only passes if it takes less than a certain time. In this example, it fails if it takes more than 5 seconds.

@pytest.mark.benchmark()
def test_heavy_task_benchmark(benchmark):
    result = benchmark(heavy_task)
    assert result == 333333328333333350000000
    assert benchmark.stats['mean'] < 5

This protects against future code modifications that may reduce efficiency.