Introduction to testing

Tests are code that verify other code is working correctly. They allow you to:

✅ Ensure that your code functions as expected today.
🛡️ Safeguard your code against future changes, ensuring it continues to work tomorrow.

Writing tests involves:

📩 Demonstrating that given known inputs, the outputs are as expected.
❓ Ensuring that unexpected inputs yield expected outputs. For example, ordering %&? beers in a bar should not cause the bartender to crash, and neither should your program.

Let us explore how to use pytest to protect your code from bugs.

The purpose of testing is to detect bugs, which are errors or defects where the code does not behave as expected. The term bug is said to originate from an actual moth found in a computer in 1947, causing it to malfunction.

Consider a sum function. Any code, no matter how simple, should be accompanied by tests to verify its correct behavior. It is as simple as this.

# sum.py
def sum(a, b):
    return a + b

def test_sum():
    assert sum(1, 2) == 3
    assert sum(5, 5) == 10

In test_sum, we verify that:

The output is 3 when the input is 1 and 2.
The output is 10 when the input is 5 and 5.

This is our first unit test. Now, let us run them. First, install pytest.

pip install pytest

With pytest, you can run the tests. The PASSED message indicates that all asserts were successful.

pytest -v sum.py
# sum.py::test_sum PASSED

It is crucial to push tests to their limits by testing unexpected inputs. For instance, adding 1 and "2" should raise an exception, as these values should not be added together.

import pytest

def test_sum_exception():
    with pytest.raises(TypeError):
        sum(1, "2")

Testing the addition of % and ? might also be necessary. While it may seem nonsensical, our function currently returns %? instead of an error. It is up to you to decide if this is acceptable.

import pytest

def test_sum_exception():
    with pytest.raises(TypeError):
        sum("%", "?")

Generally, more tests are better, but they do not guarantee anything, especially if they are incomplete or do not test enough combinations. Remember:

⚠️ A test can prove the presence of a bug, but not its absence.

If you can write a test that fails, you have proven something does not work. However, if all tests pass, it does not guarantee the code is 100% bug-free. The more comprehensive the tests, the more secure the code.

Tests are also code and are usually separated from the main code. Here are some tips for organizing your tests:

📁 Create a tests folder to store all your tests.
📝 If a module is named module.py, place its tests in test_module.py.

Before diving into testing, let us clarify some jargon:

🐛 Bug: A defect in the code where something does not work as expected. Tests aim to eliminate them.
📝 Test Case: A test that verifies a specific use case, like test_sum.
📚 Test Suite: A collection of test cases for better organization.
🎲 Flaky Test: A test that sometimes passes and sometimes fails, exhibiting random behavior. Ideally, there should be none, but they occasionally occur.
📊 Coverage: A metric measuring the percentage of code covered by tests. The goal is 100% coverage, but it does not guarantee anything.
🌪️ Fuzzing: A testing technique providing random and unexpected inputs to detect bugs, similar to asking a waiter for %&? beers.
🎁 Black Box: A type of testing viewing the system as a black box, where inputs are given, outputs are received, but the internal workings are unknown.
🔄 Continuous Integration: A software development practice where every code change is verified by running all tests. If tests fail, the change is not included. This provides immediate feedback and improves code quality.
🃏 Mock: Used in testing to replace an object with a mock. For example, if testing communications with an aircraft, a mock can simulate the aircraft. That allows you to test the code without the need for an actual aircraft.
🧭 Test Vector: A set of expected inputs and outputs. For example, for the sum function, a test vector could be 5, 3, 8, meaning if the inputs are 5 and 3, the expected output is 8.

The concept of test is broad, with many types:

🧩 Unit: Focus on testing small units of code like functions or classes.
🔗 Integration: Focus on testing the integration of all system components.
⚡️ Performance: Focus on testing speed and efficiency.

Depending on your program’s complexity, you may not need all these types, but any program should have at least unit tests. Tests are not just for advanced developers; even simple programs benefit from them.

Writing tests is an art. It is often better if the person writing the code does not write the tests, as they may be biased. An outsider with an unbiased perspective is ideal.

A person who thinks outside the box is usually good at writing tests. It is not enough to test the happy path; you must explore rare cases where the code might fail.

One advantage of writing tests as code is that they run automatically, making it easy to verify nothing is broken.

Ideally, code changes should not be supported until all tests pass. Tools like Jenkins or GitHub Actions facilitate this. Until all tests are passing, new changes cannot be added.

A common misconception is that tests guarantee the absence of bugs. They can detect presence but not absence. If not written correctly, bugs can still occur.

With this, we are ready to start writing our first tests with pytest.