Link Search Menu Expand Document
The Python Book

Fuzzing with hypothesis

Fuzzing generates numerous random inputs with various values to find cases where behavior is unexpected. Unlike previous tests, fuzzing is automatic.

Automatic generation is advantageous, allowing thousands of combinations. Writing this manually would be tedious.

Fuzzing detects:

  • 🎯 Possible edge cases not considered.
  • ⚑ Performance issues. Some inputs may take too long.
  • πŸ”’ Inconsistencies, known as invariants and idempotency. For example, sorting a list and re-sorting it should yield the same result. Reversing a list twice should return the initial list.

Here is how to use the hypothesis package. First, install it.

pip install hypothesis

Generate an example of an int, a random int.

import hypothesis.strategies as st
print(st.integers().example())
# -14251

Add restrictions, such as between 0 and 100.

print(st.integers(min_value=0, max_value=100).example())
# 95

Generate text in str, including all kinds of letters.

import hypothesis.strategies as st
print(st.text().example())
# ÈΒͺΒͺ󫦬aûæ4c

For multiple examples:

for _ in range(5):
    print(st.integers().example())

hypothesis can generate almost anything:

  • integers: Integers or int.
  • text: Text or str.
  • floats: Float numbers.
  • booleans: bool types.
  • lists: Lists or list.
  • tuples: Tuples or tuple.
  • dictionaries: Dictionaries or dict.
  • sets: Sets or set.

Now let us focus on integrating hypothesis with pytest.

Define a function that reverses an str, changing abc to cba.

def reverse(s: str) -> str:
    return s[::-1]

A good fuzzing example is verifying that using reverse twice returns the initial value.

The following test generates 10000 random str examples, verifying the property.

from hypothesis import given, settings
from hypothesis import strategies

@settings(max_examples=10000)
@given(strategies.text())
def test_reverse(s):
    assert reverse(reverse(s)) == s

Run it as follows, and it will show statistics of the tested examples. In this case, everything worked correctly.

pytest -v --hypothesis-show-statistics
# 10000 passing examples
# 0 failing examples
# 0 invalid examples

In summary:

  • Always write tests, even for simple code. They help verify functionality and protect against future changes.
  • Include tests with unexpected inputs to ensure code resilience.
  • Use coverage metrics as a guide, but remember that 100% coverage does not guarantee bug-free code.