Learnings

Python in practice, part 2 of 9

Python Testing

A test is a small piece of code that runs your real code and checks the result. This page starts from a plain assert and builds up to pytest, fixtures, parametrised tests, fakes and mocks, coverage and test-driven development. Every example tests the same small bank module, and every pytest output shown here was copied from a real run.

First tests

Part 1 of 4 · from a plain assert to pytest

You know Python but have never written a test. This part shows what a test is, how pytest finds and runs tests, and how to read what it prints when a test passes and when it fails.

Why test

Without tests you check code by running it and looking. That works once. It does not work the tenth time you change the code, because nobody re-checks everything by hand.

  • Tests catch mistakes early. A wrong balance is found on your machine, not in a customer's account.
  • Tests make change safe. Rewrite a function, run the tests, and you know in a second whether the old behaviour still holds.
  • Tests show how code is meant to be used. A test is a small working example that cannot go out of date without failing.

Most of this page uses pytest, the test tool nearly every Python project uses. It is a third-party package, so it needs pip install pytest. The coverage chapter needs pip install pytest-cov. Everything else here, including unittest, unittest.mock and doctest, is part of the standard library.

Setup: install packages into a virtual environment, not into the system Python. Python Project Setup shows how.

The bank under test

Save this as bank.py. It has a Person, an Account that refuses bad deposits and withdrawals by raising ValueError, a transfer function, and a TransferService that also sends a message through a notifier object it is given. The rest of the page tests this one file.

Used by hand, it looks like this.

Layout: keep bank.py and all the test files from this page in one folder and run every command from that folder.

Arrange, act, assert

Every test has the same three steps. Arrange: build the objects you need. Act: do the one thing you are testing. Assert: check that the result is what you expected.

Python's assert statement is enough to write one. assert x does nothing when x is true and raises AssertionError when it is false.

This already is a test. It has two weak points. The script stops at the first failed assert, so later checks never run. And a failed assert says very little.

The error carries no message: it does not say that the balance was 1500. A test runner fixes both problems. It runs every test even when some fail, and it reports exactly what was compared.

A first pytest run

Install pytest into your virtual environment.

pytest finds tests by name, so the names matter:

  • test files are named test_*.py (or *_test.py);
  • inside them, test functions are named test_*;
  • a test passes if it finishes without raising, and fails if an assert is false or any other error escapes.

Save this next to bank.py. There is nothing to import from pytest: a test is an ordinary function with an ordinary assert.

Run pytest with no arguments in that folder. It searches the folder, collects every test and runs them all.

Read it from the top: pytest found 3 tests ("collected 3 items"). Each dot after the file name is one passing test. The last line is the result. Your first lines will show your own system, Python version and folder.

Tip: if the pytest command is not found, run python -m pytest. It does the same thing with the Python you name.

Reading a failure

A failing test is the useful case, so learn to read one. These two tests are wrong on purpose.

How to read it:

  • FF on the progress line: two tests failed. A dot is a pass, F is a failure.
  • Under FAILURES each failed test gets its own block with the test's source code. The line starting with > is the line that failed.
  • Lines starting with E explain the failure. assert 1500 == 1600 shows the real values, and where 1500 = Account(...).balance shows where the 1500 came from. For the lists, pytest points at the first item that differs.
  • test_broken.py:7 is the file and line number.
  • The short summary at the bottom lists every failed test on one line.

Concept: showing the values inside a failed assert is called assertion introspection. pytest rewrites the assert statements in test files to make it possible, which is why a plain assert is all you need.

Useful flags

A flag is an option you add after pytest. These are the ones you will use every day.

CommandWhat it does
pytest test_bank.pyRun one file
pytest test_bank.py::test_transfer_moves_moneyRun one test
pytest -vVerbose: one line per test with its name
pytest -qQuiet: no header, just dots and the result
pytest -k depositRun only tests whose name contains "deposit"
pytest -xStop at the first failure
pytest --lfLast failed: run only the tests that failed last time
pytest -sShow what the tests print (pytest hides it by default)
pytest --tb=shortShorter tracebacks for failures

-v on the first test file:

-q and -k together. Two tests are "deselected" because their names do not contain "deposit".

-x on the broken file stops after the first failure, so the second test never runs.

Tip: -k accepts small expressions too: pytest -k "deposit or transfer", pytest -k "not withdraw".

Testing errors

Refusing a bad withdrawal is behaviour too, and it needs a test. pytest.raises is a context manager: the test passes only if the code inside the with block raises the named exception.

  • match= also checks the error message. It is a regular expression searched for in the message, so part of the text is enough.
  • as excinfo gives you the caught exception as excinfo.value for any further checks.

If the code does not raise, the test fails. This test wrongly expects a small withdrawal to be refused, and these are the two key lines of the failure report pytest prints for it:

Habit: keep only the one call that should raise inside the with block. If the setup lines are inside too, an error in the setup can make the test pass for the wrong reason.

Comparing floats

Floats are stored in binary, so some sums are off by a tiny amount. Comparing them with == gives surprising failures.

pytest.approx(value) compares with a small tolerance, by default one part in a million. Pass abs= to set the allowed difference yourself.

Real money: a real bank would store paise as whole numbers or use decimal.Decimal, where sums are exact. approx is for values that are floats by nature, such as rates and averages.

Fixtures and parameters

Part 2 of 4 · less repeated code in tests

The tests so far each build their own accounts. With thirty tests that is thirty copies of the same lines. This part covers the two pytest features that remove the repetition: fixtures for shared setup, and parameters for running one test with many inputs.

Fixtures

A fixture is a function that prepares something a test needs. Mark it with @pytest.fixture. A test asks for it by having a parameter with the same name, and pytest calls the fixture and passes in what it returns.

The fixture runs again for every test that asks for it. test_withdraw gets a brand new account with 9000 in it, whatever test_deposit did to its own. That keeps tests independent of each other.

Concept: you never call a fixture yourself. Naming it as a parameter is the request. This is a small form of dependency injection: the test says what it needs and something else supplies it.

Fixtures using fixtures

A fixture can ask for other fixtures in the same way a test does. Small fixtures then combine into bigger starting points.

Cleaning up with yield

Some setup has to be undone: a file closed, a temporary record deleted. Write yield where you would write return. The code before yield runs before the test, and the code after it runs when the test is over, whether it passed or failed.

Run it with -s so the prints are visible, and the order shows.

Fixture scope

By default a fixture is rebuilt for every test. scope= changes how long one result is kept and shared.

ScopeBuilt once per
"function" (default)test function
"class"test class
"module"test file
"session"whole pytest run

Both tests pass, and the second one shows the cost: it sees the deposit made by the first. A shared object carries changes from one test to the next.

Rule of thumb: keep the default scope. Use a wider scope only for something slow to build that tests read and do not change, such as a large data file loaded once.

Sharing with conftest.py

Fixtures defined in a test file are visible only in that file. Put them in a file named conftest.py and every test file in the same folder, and in folders below it, can use them. There is nothing to import: pytest loads conftest.py by itself.

This test file defines no fixtures and does not import any. It is not run by this site's example checker, because it only works under pytest with conftest.py beside it. It was run with pytest: 2 passed.

Tip: if a test file defines a fixture with the same name as one in conftest.py, the one in the test file wins for that file.

Built-in fixtures

pytest ships with fixtures you can ask for by name. Three are worth knowing from the start.

FixtureWhat the test gets
tmp_pathA new empty folder, as a pathlib.Path, different for every test
capsysEverything the code printed, as text you can check
monkeypatchA tool to replace attributes, dictionary items and environment variables for one test, and put them back afterwards

An environment variable is a named setting that the operating system passes to a program. After test_monkeypatch ends, BANK_BRANCH is removed again. tmp_path and monkeypatch come back in the chapters on files, the clock and mocking.

Parametrised tests

Often you want the same check for many inputs. Copying the test five times is noise. @pytest.mark.parametrize takes the parameter names and a table of rows, and runs the test once per row.

Three functions become nine tests. Each row passes or fails by itself and gets its own name in square brackets. Without ids the name is built from the values. With ids=, or pytest.param(..., id=...) on a row, you choose it.

Spelling: the decorator is spelled parametrize, with a z and no second e. Any other spelling is an error.

Skip and xfail

A mark is a label on a test, written as a decorator. parametrize is one. Three more control whether a test runs.

  • skip: do not run this test.
  • skipif: do not run it when a condition is true, such as an old Python version.
  • xfail: "expected to fail". Run it, but a failure is reported as xfailed and does not make the run fail. Use it for a known bug you have not fixed yet.

-rsx adds the reasons for skipped (s) and xfailed (x) tests to the summary.

This output is from Python 3.9, so the skipif test was skipped. On Python 3.10 or newer it runs and you get 2 passed, 1 skipped, 1 xfailed.

Habit: always give a reason. A skipped test with no reason stays skipped for years because nobody remembers why.

Fakes and mocks

Part 3 of 4 · testing code that talks to the outside world

Some code sends messages, calls another server, reads the clock or writes files. A test must not send a real SMS, and it must give the same result every run. The answer is to replace the outside part with a stand-in during the test.

Test doubles

A test double is any object that stands in for a real one during a test, like a stunt double in a film. There are four common kinds. People often call all of them "mocks", but the differences are worth knowing.

NameWhat it doesBank example
StubReturns fixed answers and nothing moreA rate service that always says 4%
FakeA simple working version of the real thingA notifier that keeps messages in a list instead of sending them
SpyRecords how it was called so the test can check afterwards"Was send called once, with this text?"
MockAn object made by a mocking library that can be a stub and a spy at onceunittest.mock.Mock()

Use a double for things that are slow, cost money, need a network, or give a different answer every time. Do not use one for your own plain code: test a real Account, not a stand-in for it.

A hand-written fake

TransferService does not create its notifier. It receives one. The dependency inversion chapter of Design Principles promised that this pays off in tests: "pass a fake notifier that stores messages in a list". Here is that fake. It needs no library at all.

No SMS, no email, no network, and the whole transfer rule is tested, including the exact message text. Because the fake records what it was given, it also works as a spy.

Design: this was only possible because the notifier is passed in. If TransferService created an SmsSender inside __init__, the test would have nowhere to put the fake. Code that receives its helpers is easy to test.

Mock objects

Writing a small class for every stand-in gets tiring. unittest.mock, in the standard library, makes them for you. A Mock accepts any method call, returns another Mock, and remembers every call.

assert_called_once_with is the spy part: it raises AssertionError unless there was exactly one call with exactly those arguments. call_args holds the arguments of the most recent call if you want to inspect them yourself.

Choosing what a mock returns

return_value sets a fixed answer. That is the stub part. side_effect does more: give it a list to return a different value on each call, or an exception to make the call raise.

Mocks in a test

The two notifier tests again, with a Mock in place of the hand-written class. The second test uses side_effect to see what happens when sending fails, a case that is very hard to produce with a real SMS service.

MagicMock and spec

MagicMock is a Mock that also supports Python's special methods, so it works with len(), with, loops and indexing. A plain Mock does not.

A mock that accepts anything also accepts mistakes. A misspelled method name is just another call. Pass spec= with the real class and the mock allows only the names that class has.

Patching

The notifier was easy because it is passed in. Often the thing you want to replace is a function that the code imports and calls directly. Two small modules show the case. rates.py stands for a slow call to another server; here it simply fails, as it would on a machine with no network.

savings.py uses it. This block and the test files that import savings are marked as not run by this site's example checker, because they need these helper files next to them. They were all run with pytest instead.

patch("module.name") replaces that name with a MagicMock for a while and then puts the original back. It can be used in two ways. As a decorator the patch lasts for the whole test, and the mock is passed in as an argument. As a context manager it lasts for the with block.

Both tests pass without any network, and the real fetch_rate is never called.

Patch where it is used

Look again at the target string: "savings.fetch_rate", not "rates.fetch_rate". The function is defined in rates, so patching it there looks natural. It does not work.

The real function ran. The reason is how import works. from rates import fetch_rate creates a second name, savings.fetch_rate, that points at the same function. add_interest looks the function up through that name. Patching rates.fetch_rate changes the name inside rates only, and savings still holds the original.

Rule: patch the name in the module that uses it, not in the module that defines it. For from rates import fetch_rate inside savings.py, the target is "savings.fetch_rate". Had savings.py written import rates and called rates.fetch_rate(), the lookup would go through rates, and "rates.fetch_rate" would be the right target.

monkeypatch

pytest's monkeypatch fixture does the same job without unittest.mock. You give it the object, the attribute name and the replacement. Everything is put back when the test ends.

The same rule applies: the attribute is replaced on savings, where it is used. This file needs savings.py and rates.py, so it was verified with pytest (1 passed) and is not run by the example checker.

NeedReach for
Swap in a simple replacement value or functionmonkeypatch.setattr
Set or remove an environment variablemonkeypatch.setenv, monkeypatch.delenv
Check afterwards how the replacement was calledpatch, which gives you a mock
Tests written with unittest, without pytestpatch

Time, random and the environment

Three things make a function give a different answer on every run: the current time, random numbers, and settings read from the environment. A test cannot assert on a value that keeps changing, so pin each one down.

Each test below fixes the changing part to a known value, in both styles. patch.dict changes a dictionary for the length of a with block; os.environ behaves like a dictionary. This file imports branch, so it was verified with pytest (5 passed) and is not run by the example checker.

Why delenv: test_limit_default removes the variable first. Without that line the test would fail on any machine where BANK_DAILY_LIMIT happens to be set. A test should not depend on the machine it runs on.

Code that writes files

Code that reads and writes files is tested with real files, in a folder that belongs to the test. tmp_path gives each test its own new folder, so tests never see each other's files and nothing is left in your project.

Three useful checks: what exactly lands in the file, a full round trip of saving and loading, and the error case. This file imports statement, so it was verified with pytest (3 passed) and is not run by the example checker.

Design: the functions take the path as an argument. If save_statement had a fixed file name written inside it, the test could not send it to a temporary folder. It is the same idea as passing in the notifier.

Code that reads the clock

A function that calls datetime.now() returns something different every minute. There are two ways to test it. The cleaner one is to inject the clock: let the function accept the "what time is it" function as a parameter, with the real clock as the default.

The first test passes its own clock to receipt, with no patching at all. is_weekend reads the clock directly, so the other two tests patch the name datetime inside the receipt module and tell the mock what now() returns. This file imports receipt, so it was verified with pytest (3 passed) and is not run by the example checker.

Gotcha: patch("receipt.datetime.now") does not work. It fails with TypeError: can't set attributes of built-in/extension type 'datetime.datetime', because datetime is written in C and its methods cannot be replaced. Patch the whole name receipt.datetime as above, or inject the clock and avoid the problem.

Good habits

Part 4 of 4 · tests that stay useful

Knowing the tools is half of it. The other half is writing tests that people trust and keep. This part covers what a good test looks like, writing the test first, measuring what the tests reach, and the two other test styles you will meet in Python code.

What makes a good test

  • One behaviour per test. When it fails, the name alone tells you what broke. Several assert lines are fine if they all check the same behaviour, like both balances after one transfer.
  • A clear name. test_withdraw_more_than_balance_is_refused reads as a sentence about the code. test_withdraw_2 says nothing.
  • Independent. Each test builds what it needs and passes when run alone. Fixtures give every test a new start.
  • Fast. Tests that take milliseconds get run after every change. Tests that take minutes get run once a day, if at all.
  • No hidden order. A test must not rely on another test having run first. pytest runs a file's tests from top to bottom, but -k, --lf and running a single test all change which tests come before it.

A good test also checks what the code does, not how. "After the transfer my mother has 3500" stays true when you rewrite transfer. "transfer calls withdraw and then deposit" breaks on every rewrite, even a correct one.

The testing pyramid

Tests come in sizes. A unit test checks one small piece by itself, like Account.withdraw. An integration test checks that several pieces work together, such as saving a statement to a real file and loading it back, or code talking to a real database. An end-to-end test drives the whole program the way a user would. The testing pyramid is advice about how many of each to have: many unit tests at the bottom because they are fast and point straight at the fault, fewer integration tests in the middle, and only a handful of end-to-end tests at the top because they are slow and break for many reasons. Almost every test on this page is a unit test.

Test-driven development

Test-driven development, or TDD, means writing the test before the code. It is a loop of three steps:

  • Red: write a test for behaviour that does not exist yet, run it, and watch it fail.
  • Green: write the least code that makes it pass.
  • Refactor: tidy the code while the tests keep passing.

A worked example: the bank wants an overdraft limit, an amount by which an account may go below zero. Start with the tests. They describe the feature before any of it exists.

Red

Two tests fail, and for the expected reason: Account has no overdraft_limit yet. Seeing the failure first matters. It proves the tests can fail, so a later pass means something. The third test passes already because it describes behaviour the bank has today, and it will guard that behaviour during the change.

Green

Two changed lines in bank.py are enough: a new field with a default of 0, and one changed condition in withdraw. This is a fragment of the Account class, not a whole file, so it is not run by the example checker.

Refactor

The sum "balance plus overdraft limit" is a real idea in a bank: the amount available. Give it a name. The behaviour does not change, so the tests must stay green.

Run the new tests together with the older ones, to be sure nothing else broke.

Note: the rest of this page still describes the original bank.py from the top. The overdraft change is compatible with it: every passing test on this page also passes against the changed file.

Coverage

Coverage measures which lines of your code ran while the tests ran. The tool is coverage.py, and pytest-cov connects it to pytest. Both are third-party.

--cov=bank says which module to measure. --cov-report=term-missing adds the line numbers that never ran. With only the three tests from the first test file:

The missing lines tell you what is untested. Lines 20, 25 and 27 are the three raise statements: these three tests never try a bad amount. Line 10 is first_name, and lines 38 and 41 to 42 are TransferService. The later test files on this page cover all of them.

Why 100% is not the goal

Coverage shows that a line ran. It does not show that anything checked the result. A test that calls deposit and asserts nothing still counts as covering it. So a low number is a useful warning, but a high number is not proof of good tests. Chasing 100% also leads to tests written to touch lines, not to check behaviour. Use the report to find code you forgot to test, then decide whether that code matters.

The unittest style

The standard library has its own test framework, unittest. It is older than pytest and you will meet it in many projects, so learn to recognise it. Tests are methods on a class that inherits from unittest.TestCase, checks are methods like assertEqual, and setUp runs before every test.

pytest runs unittest tests too. pytest test_bank_unittest.py reports 2 passed, so a project can move from one style to the other a file at a time.

Taskunittestpytest
InstallNothing, it is in the standard librarypip install pytest
A test isA test_ method on a TestCase subclassA plain test_ function
Check equalityself.assertEqual(a, b)assert a == b
Check truthself.assertTrue(x)assert x
Check membershipself.assertIn(x, items)assert x in items
Expect an errorwith self.assertRaises(ValueError):with pytest.raises(ValueError):
Setup and cleanupsetUp and tearDown methodsFixtures
Many inputsA loop with self.subTest(...)@pytest.mark.parametrize
Runpython -m unittestpytest

doctest

A doctest is an example inside a docstring, written as it would look in the Python prompt: a line starting with >>>, then the expected output. The standard library's doctest module runs the examples and compares what really comes out.

There are three ways to run them: run the file as above, run python -m doctest -v interest.py, or let pytest collect them with pytest --doctest-modules.

pytest counts all the examples in one docstring as one test. Doctests are good for keeping the examples in your documentation honest. They are a poor fit for anything that needs setup, so use them next to ordinary tests, not in place of them.

Quick reference

To do thisWrite this
Check a valueassert account.balance == 1500
Expect an exceptionwith pytest.raises(ValueError, match="funds"):
Compare floatsassert total == pytest.approx(0.3)
Share setup@pytest.fixture, then name it as a test parameter
Clean up after a testyield in the fixture, cleanup code after it
Share fixtures between filesPut them in conftest.py
Run one test with many inputs@pytest.mark.parametrize("a, b", [(1, 2), (3, 4)])
Skip a test@pytest.mark.skip(reason="...") or skipif(condition, reason="...")
Mark a known bug@pytest.mark.xfail(reason="...")
Get a temporary folderThe tmp_path fixture
Check printed outputcapsys.readouterr().out
Set an environment variablemonkeypatch.setenv("NAME", "value")
Replace a function for one testmonkeypatch.setattr(module, "name", replacement)
Make a stand-in objectMock(), or Mock(spec=RealClass)
Set what a mock returnsmock.method.return_value = 0.04
Make a mock raisemock.method.side_effect = ConnectionError("down")
Check how a mock was calledmock.method.assert_called_once_with(...)
Replace a name in a module@patch("module_that_uses_it.name")
Run one testpytest test_bank.py::test_transfer_moves_money
Run tests by namepytest -k deposit
Stop at the first failurepytest -x
Re-run only the failurespytest --lf
Measure coveragepytest --cov=bank --cov-report=term-missing

Common mistakes

Tests that depend on each other, and tests that assert nothing

This file has both problems. All three tests share one account created at the top of the file.

Run the whole file and pytest reports 3 passed. Run test_withdraw alone and it fails, because the deposit it silently relied on never happened.

The fix for the first problem is a fixture, so that each test gets its own account. The fix for the second is to end every test with a check. test_asserts_nothing only proves that deposit did not crash.

Over-mocking, and testing how instead of what

This test replaces both accounts with mocks and then checks which methods were called.

It passes, and it proves very little. No real Account is involved, so it would still pass if withdraw added money instead of taking it. It also repeats the body of transfer line by line, so any rewrite of transfer breaks it, even a correct one. Compare it with test_transfer_moves_money near the top of this page, which uses real accounts and checks the balances.

MistakeWhat goes wrongDo this
Tests share stateThey pass together and fail alone, or the other way roundBuild a new object per test with a fixture
No assertThe test cannot fail, so it tells you nothingEnd with a check on the result
Mocking your own simple codeThe test checks the mock, not the codeUse real objects; mock only the outside world
Testing implementationEvery refactor breaks tests although behaviour is unchangedAssert on results: return values, balances, messages sent
Patching where the function is definedThe real function still runsPatch the name in the module that uses it
Files not named test_*.pypytest does not collect them, so their tests never runFollow the naming rules
== on floatsFails on tiny rounding differencespytest.approx

When a test does fail and the message is not enough to see why, the next step is to look inside the running program: continue with Python Debugging and Logging.