Python in practice, part 3 of 9
Python Debugging and Logging
Debugging is finding out why a program does something you did not expect. Logging is making the program write down what it is doing, so that you can find out later without being there. This page covers both, using a small bank as the example. Everything here is in the standard library, so nothing needs pip install.
Debugging
Part 1 of 2 · finding out what went wrong
A bug is a difference between what you think the code does and what it really does. Debugging closes that gap. You read what Python tells you, look at the real values, and test one idea at a time.
The tools go from simple to powerful. Most bugs are solved with the first two.
- Reading a traceback: the error report, read from the bottom up
- Chained exceptions: when one error happens while handling another
- Common exceptions: what each one usually means
- Print debugging: looking at values the quick way
- assert: checks that state what must be true
- breakpoint() and pdb: stop the program and look around
- A pdb session: the commands in use
- Post-mortem debugging: start the debugger after a crash
- Conditional breakpoints: stop only when something is odd
- Switching breakpoints off, and IDE debuggers
- A method for finding bugs: five steps that work
- Warnings: problems that are not errors yet
- The traceback module: a traceback as text
Reading a traceback
When an exception is not caught, Python stops and prints a traceback: the list of function calls that were in progress when the error happened. Here is a small script with a bug. It is marked as not runnable on purpose, because it crashes.
Running it with python3 transfer.py prints this. Only the folder name has been shortened.
Read it from the bottom up.
- The last line is the error itself: the exception type (
KeyError) and its message ('Hemkesh'). A dictionary was asked for a key it does not have. - The pair of lines just above it is where the error happened: file, line number, function name, then the line of code. Here it is line 5, inside
get_balance. - Each pair above that is the caller of the one below.
get_balancewas called bytransferon line 10, which was called bymainon line 14, which was called from the top of the file (<module>) on line 18. - The first line says "most recent call last". That is why the place of the crash is at the bottom and the start of the program is at the top.
Each pair is called a frame: one function call that has started and not finished yet. The bug is often not in the bottom frame. Line 5 is fine. The real question is one frame up: why did transfer ask for an account that does not exist?
Tip: In a long traceback, look for the lowest frame that is in your own file. Frames inside library code usually only show how your bad value travelled. Python 3.11 and later also draw ^^^^ marks under the exact part of the line that failed.
Chained exceptions
Sometimes a traceback has two parts joined by a sentence. There are two such sentences, and they mean different things.
During handling of the above exception
A second error happened by accident inside an except block. This block is not runnable because it crashes: the name nme is a typing mistake.
The program died of the last error, the NameError. The first part shows what the code was busy handling at the time. Fix the bottom one first.
The above exception was the direct cause
Here the second error was raised on purpose with raise ... from .... The code turns a low-level error into one that makes sense for a bank, and keeps the original attached. Also not runnable, because it ends in an uncaught exception.
The original error is stored on the new one as __cause__, so code can look at it too.
Tip: Write raise NewError(...) from None when the original error would only confuse the reader. The first part of the traceback is then left out.
Common exceptions
A handful of exception types cover most crashes. Knowing what each usually means saves a lot of reading.
| Exception | What it usually means | First thing to check |
|---|---|---|
SyntaxError | Python could not even read the file | A missing bracket, quote or colon, often on the line before the one reported |
IndentationError | The spaces at the start of a line do not line up | Mixed tabs and spaces, or a block that was not indented |
NameError | A name is used that does not exist | A spelling mistake, or a variable used before it is assigned |
UnboundLocalError | A function reads a local variable before giving it a value | A variable assigned only inside an if, or a global assigned inside the function |
TypeError | The value has the wrong type for the operation | A string where a number was expected, or the wrong number of arguments |
ValueError | The type is right but the value is not acceptable | int("12,000"), or unpacking the wrong number of items |
KeyError | A dictionary does not have that key | Print the keys. Is it a spelling, case or type difference ("7" and 7)? |
IndexError | A list position is past the end | An empty list, or counting from 1 instead of 0 |
AttributeError | The object has no attribute of that name | If the message says 'NoneType' object, a function returned None and you did not expect it |
ZeroDivisionError | Division by zero | An empty list when working out an average |
FileNotFoundError | No file at that path | The folder the program was started from, shown by os.getcwd() |
ModuleNotFoundError | The module in an import cannot be found | The package is not installed in this environment, or the wrong Python is running |
RecursionError | A function called itself too many times | A missing or unreachable stopping condition |
AssertionError | An assert statement was false | The condition on the reported line |
KeyboardInterrupt | Someone pressed Ctrl+C | Not a bug. The traceback shows where the program was when it was stopped, which helps find an endless loop |
Two of them, caught and printed so that the block runs to the end:
Catching and handling exceptions is covered in Python Errors and Files.
Print debugging
The quickest way to look at a value is to print it. Done well, it is a real tool. The = inside an f-string prints the expression and its value together, so you never wonder which number is which.
The output shows two bugs at once. The name has a space at the end, and the balance is text, not a number. That is why doubling it gave '1200012000'.
Plain print(x) hides exactly these things. repr(x) shows the value the way you would type it in code, with quotes and invisible characters made visible.
Three habits make print debugging work:
- Print a label with every value.
f"{value=}"does it for you. - Print the type when a value looks right but behaves wrongly.
- Print before and after the line you suspect, so you see what changed.
Watch out: Remove debug prints before you share the code. If a message is worth keeping, turn it into a logger.debug() call, as shown in the second part of this page.
assert as a sanity check
assert condition, message does nothing when the condition is true and raises AssertionError when it is false. Use it to state something that must be true if your own code is correct. When it fails, you have found a bug close to where it started.
Do not use assert to check input from users, files or the network. Python has a switch, -O (optimise), that removes every assert from the program. The block below runs the same two lines twice, once normally and once with -O.
With -O the check is simply gone. For input, raise a real exception instead, because it can never be switched off:
Rule: assert is for mistakes made by the programmer. raise is for bad data from outside.
breakpoint() and pdb
A debugger stops a running program so that you can look at every variable and run the code one line at a time. Python comes with one called pdb. Put breakpoint() on any line, and the program pauses there and gives you a (Pdb) prompt.
This script has a bug: one balance was typed as text. It is not runnable here because breakpoint() waits for you to type commands.
At the prompt you type short commands. These are the ones that matter.
| Command | Long form | What it does |
|---|---|---|
l | list | Show the code around the current line. ll shows the whole function |
n | next | Run the current line and stop at the next one. A function call is run in one go |
s | step | Like n, but if the line calls a function, go inside it |
c | continue | Run until the next breakpoint or the end of the program |
p expr | print | Print the value of any expression, such as p balance or p len(accounts) |
pp expr | Pretty-print: the same, but long lists and dictionaries are laid out over several lines | |
w | where | Show the stack: the chain of calls that led here, like a traceback |
u | up | Move to the caller's frame, to see its variables |
d | down | Move back down one frame |
b 9 | break | Set a breakpoint on line 9. b alone lists the breakpoints. b yearly_interest stops when that function is called |
r | return | Run until the current function returns |
q | quit | Leave the debugger and stop the program |
!statement | Run a Python statement, for example !balance = 0. The ! stops pdb from reading it as one of its own commands | |
h | help | List all commands. h b explains one of them |
Tip: Pressing Enter on an empty line repeats the last command. That makes it easy to walk through code with n, Enter, Enter, Enter.
A pdb session
This is a real session with interest.py from the previous chapter, started with python3 interest.py. The lines after (Pdb) are what was typed. A line starting with > shows the file, line number and function where the program is paused, and the -> line below it is the line that will run next.
What happened, step by step:
pp accountsshowed the data before anything ran.sstepped intoreport. Threencommands walked to line 9, andp name, balanceshowed the first account.sstepped intoyearly_interest. After two more lines,p interestshowed200.0, which is correct.wprinted the stack. The>marks the current frame.umoved up toreport, wheretotalis still0because line 9 has not finished.b 9set a breakpoint andcran to it. Now it is Sandhya's turn, and her balance is the text'12000'. That is the bug.!balance = int(balance)repaired the value in the running program, andclet it finish with the right total,680.0.
Remember: Changing a value inside pdb only proves your idea about the bug. The real fix still has to go into the file: store 12000 as a number.
Post-mortem debugging
Post-mortem means "after death". You do not add breakpoint() anywhere. You let the program crash, and the debugger opens at the exact line that failed, with all the variables still there.
The script is not runnable here because it crashes. Start it under pdb with python3 -m pdb crash.py. pdb pauses on the first line, so type c to let it run.
Two other ways to get the same thing:
- Run
python3 -i crash.py. The-iswitch leaves you at the>>>prompt after the crash. Then typeimport pdbandpdb.pm().pmstands for post-mortem, and it opens the debugger on the last traceback. - Inside an
exceptblock, callpdb.post_mortem()to open the debugger on the exception being handled.
When to use it: Post-mortem is the fastest route when the program crashes with an exception. Use breakpoint() when nothing crashes but the answer is wrong.
Conditional breakpoints
A loop may run a thousand times and go wrong only once. Stopping on every pass is no use. A conditional breakpoint stops only when a condition is true.
The simplest form is an ordinary if in the code. Not runnable here, because it would wait for commands.
The other form needs no change to the file. In pdb, write the condition after the line number, separated by a comma. This is crash.py from the previous chapter, started with python3 -m pdb crash.py:
The first account passed line 2 without stopping. The debugger stopped only for the one whose balance is a string, before the line crashed.
Switching breakpoints off, and IDE debuggers
breakpoint() reads an environment variable called PYTHONBREAKPOINT. An environment variable is a named setting that the shell passes to a program. Set it to 0 and every breakpoint() call does nothing, without editing the code.
The block below proves it. It writes a two-line script that contains breakpoint(), then runs it with the variable set, so nothing waits for the keyboard.
Watch out: A breakpoint() left in code that runs on a server will make it hang, waiting for a keyboard that is not there. Search for breakpoint( before every commit.
IDE debuggers
An IDE (integrated development environment) is an editor with tools built in. VS Code and PyCharm both have a graphical debugger that does the same job as pdb. You click next to a line number to set a breakpoint, press the debug button, and the editor shows all variables in a side panel and the call stack as a list. Buttons called Step Over, Step Into and Continue are n, s and c. Right-clicking a breakpoint lets you add a condition. The ideas are exactly the ones in this page, so learning pdb first means you can debug anywhere, including on a server that has no editor.
A method for finding bugs
Tools help, but a steady method helps more. Changing lines at random until the error goes away is slow and leaves bugs behind.
- Reproduce it. Find the exact steps or input that make the bug happen every time. A bug you cannot repeat is a bug you cannot prove fixed.
- Reduce it. Remove everything that is not needed to show the bug: fewer accounts, fewer steps, a smaller file. Often the cause shows itself during this step.
- Read the error. The whole message, slowly, from the bottom up. It usually says what is wrong and on which line.
- Form one hypothesis. A hypothesis is a guess you can test: "the balance is a string". Check it with a print or the debugger before changing anything.
- Change one thing. Make one change, run again, and look. If it did not help, undo it. Two changes at once tell you nothing about which one mattered.
Apply it to transfer.py from the first chapter. The error says KeyError: 'Hemkesh' on the receiver's line. Hypothesis: the money had already left the sender when the crash came. One print proves it.
So there are two bugs. The crash is the small one. The serious one is that 500 rupees vanished. The fix is to check everything first and change the balances only after that.
Tip: Once a bug is fixed, write a test that fails with the old code and passes with the new. Then it cannot come back unnoticed. See Python Testing.
Warnings
A warning is a message that something is likely to become a problem, while the program carries on. The most common kind is DeprecationWarning: a function still works but will be removed one day. The warnings module issues and controls them.
Normally a warning is printed once to the error stream with the file and line that caused it. stacklevel=2 makes that line the caller's line, which is the one that has to change. The catch_warnings block is used here only to collect the warning so it can be printed in a fixed form.
While hunting a bug, it helps to turn warnings into exceptions. Then you get a full traceback showing who made the call.
From the terminal: python3 -W error script.py does the same for a whole run without touching the code, and python3 -W default script.py shows warnings that are hidden by default.
The traceback module
Sometimes a program must survive an error and still keep the full report, for example a bank job that processes many accounts overnight. The traceback module gives you the traceback as text, so you can save it and carry on.
The loop finished all three accounts, and the string in problems holds the same text Python would have printed, with file and line number.
| Function | What it gives |
|---|---|
traceback.format_exc() | The traceback of the exception being handled, as a string |
traceback.print_exc() | The same, printed straight to the error stream |
traceback.print_stack() | The current call stack, with no exception needed. Good for "who called this function?" |
In real programs you rarely call these yourself. logger.exception(), shown in a later chapter, does the same and writes the result to the log.
Logging
Part 2 of 2 · a record of what the program did
A log is a list of messages a program writes while it runs: what it did, when, and what went wrong. When something fails at night on a server, the log is the only witness. Python's logging module is the standard way to write one.
- Why not print: what logging adds
- The five levels: how important is this message?
- basicConfig: set up logging in one line
- A logger for each module:
getLogger(__name__) - Logging an exception: keep the traceback
- Lazy formatting:
%sarguments instead of f-strings - Handlers and formatters: file and console at once
- Rotating log files: stop the file growing for ever
- The logger hierarchy: parents, children and propagation
- Configure once, at the start:
dictConfig - Structured logging: one JSON object per line
- What never to log: passwords, cards, account numbers
- Common mistakes
- Quick reference
Why not print
print() is fine while you sit in front of the program. It is a poor record once the program runs on its own. Logging gives four things that print does not.
| Need | logging | |
|---|---|---|
| Levels | Every message looks equally important | Each message has a level, from DEBUG to CRITICAL |
| Destinations | The screen only | Screen, file, several files, or all at once |
| Context | Only what you typed | Time, level, module name and line number added for you |
| Turning it off | Delete or comment out every call | Change one setting. The calls stay in the code |
About the examples: Logging writes to the error stream (stderr) by default and normally includes the time. The runnable blocks on this page send it to sys.stdout and leave the time out, so the output is the same on every run.
The five levels
Every log message has a level that says how serious it is. A logger also has a level, and it drops every message below it. So one setting decides how much detail you get.
| Level | Number | Use it for | Bank example |
|---|---|---|---|
DEBUG | 10 | Detail that only helps while finding a bug | "Balance before the deposit was 5000" |
INFO | 20 | Normal events worth recording | "Deposited 500 for Rishikesh" |
WARNING | 30 | Something odd happened, but the work was done | "Balance for Hemkesh is below the minimum" |
ERROR | 40 | One piece of work failed | "Transfer to Malinikesh failed" |
CRITICAL | 50 | The whole program cannot continue | "Cannot reach the accounts database" |
The level was set to WARNING, so the DEBUG and INFO messages were dropped. WARNING is also the default when you set nothing.
Rule of thumb: Use DEBUG while developing and INFO or WARNING in production. If an ERROR appears in the log, a person should look at it.
basicConfig
logging.basicConfig() sets up logging in one call: the lowest level to show, where to write, and what each line looks like. The format string is built from named fields.
| Field | Meaning |
|---|---|
%(asctime)s | Date and time, such as 2026-11-17 10:15:02,123 |
%(levelname)s | DEBUG, INFO, WARNING, ERROR or CRITICAL |
%(name)s | Name of the logger. root when you call logging.info() directly |
%(message)s | The message itself |
%(module)s, %(funcName)s, %(lineno)d | The file (without .py), function and line number that made the call |
A usual format for real work is "%(asctime)s %(levelname)s %(name)s: %(message)s". To write to a file, pass filename="bank.log" instead of stream.
Watch out: basicConfig() works only the first time. If logging is already set up, later calls do nothing. Pass force=True (Python 3.8 and later) to replace the old setup.
A logger for each module
Calling logging.info() uses one shared logger called the root logger. In a real program, each file should have its own logger, created once at the top with logging.getLogger(__name__).
__name__ is the module's name. Inside bank/transfers.py it is "bank.transfers". In the file you run directly it is "__main__".
Why bother:
- Every line in the log says which module wrote it.
- You can make one noisy module quieter, or one suspect module more detailed, without touching the others.
getLogger("bank.transfers")always returns the same object for the same name, so nothing has to be passed around.
Logging an exception
Inside an except block, logger.exception("message") writes the message at ERROR level and adds the full traceback below it. You get the whole story, and the program can carry on.
The output, with the folder name shortened:
logger.error("...") alone would log only the message. logger.exception("...") is the same as logger.error("...", exc_info=True). Use exc_info=True with another level, such as logger.warning, when a failure is expected and not serious.
Watch out: Call logger.exception() only inside an except block. Outside one there is no exception to report.
Lazy formatting
Log calls take a message with %s placeholders and the values as separate arguments. The logger joins them only if the message will really be written. An f-string builds the text first, even when the level is switched off and the text is thrown away.
Neither call logged anything, but the f-string still did the work. There is a second reason to prefer %s arguments: the message text stays the same for every call, so tools that group log lines can see that they are the same event.
Fair point: For cheap values the speed difference is tiny, and many teams allow f-strings in log calls. The habit matters most for DEBUG messages inside loops.
Handlers and formatters
Three kinds of object work together:
- A logger is what your code calls. It decides whether a message is important enough.
- A handler sends the message somewhere:
StreamHandlerto the screen,FileHandlerto a file. - A formatter turns the message into a line of text. Each handler has its own.
One logger can have several handlers, each with its own level and format. A common setup: short warnings on the screen, everything in a file.
A message has to pass two gates: the logger's level, then each handler's level. In a real file format you would start with %(asctime)s, which gives lines such as 2026-11-17 10:15:02,123 INFO bank withdraw: Withdrew 4500 for Hemkesh. A second argument to Formatter changes the time layout, for example datefmt="%Y-%m-%d %H:%M:%S".
Rotating log files
A log file that only grows will fill the disk one day. RotatingFileHandler starts a new file when the current one reaches a size limit and keeps only a fixed number of old ones. The newest lines are always in bank.log. Older ones move to bank.log.1, then bank.log.2, and the oldest file is deleted.
Thirty lines were written, but only the last fifteen are kept, five in each of three small files. The limit here is tiny so the effect is visible. A real setting looks like maxBytes=5_000_000, backupCount=5: five megabytes per file, five old files.
Also useful: TimedRotatingFileHandler("bank.log", when="midnight", backupCount=14) starts a new file every day and keeps two weeks.
The logger hierarchy
Logger names with dots form a family tree. bank.transfers is a child of bank, and bank is a child of the root logger. A message logged on a child is also passed up to the handlers of its parents. This is called propagation.
So you attach a handler once, high up, and every module below it is covered.
Two rules explain the output:
- Level: a logger with no level of its own uses its nearest parent's.
bank.accountshas none, so it uses INFO (20) frombank. - Handlers: the child loggers have no handlers. Their messages travel up to
bank, which prints them.
If a message appears twice in your log, a handler is attached at two levels of the tree. Remove one, or set propagate = False on the child to stop messages travelling up.
Configure once, at the start
Logging should be set up in one place, at the very start of the program, usually in main(). Every other file only creates its logger and uses it. For anything beyond basicConfig, describe the whole setup as a dictionary and pass it to logging.config.dictConfig().
The dictionary has the same three parts as before: formatters, handlers that use a formatter, and loggers that use handlers. "version": 1 is required. "disable_existing_loggers": False keeps loggers that were created before this call working, which is nearly always what you want.
Tip: Because it is plain data, the dictionary can be loaded from a JSON file, so the logging setup can change without editing code.
Structured logging
A line such as "Deposited 500 for Rishikesh" is easy for a person to read and hard for a program to search. Structured logging writes each event as data, usually one JSON object per line, so a log tool can filter by field: every event for one account, or every amount over 10,000.
A formatter is just a class with a format(record) method, so a tiny one of your own is enough. Extra fields are passed with the extra argument and become attributes of the record.
A real version would also add the time, for example self.formatTime(record). It is left out here so the output does not change between runs. Packages such as python-json-logger and structlog do this job fully. They need pip install.
What never to log
Log files are copied, backed up, sent to other systems and read by many people. Anything written to a log should be treated as public inside the company. Never log:
- Passwords, PINs, one-time codes and security answers
- Full card numbers, CVV codes and expiry dates
- Full account numbers and government identity numbers
- API keys, tokens and session cookies
Log an identifier that is safe, such as an internal customer id, or mask the value so only the last four digits remain.
People forget, so add a safety net. A filter is an object that sees every record before it is written. This one replaces any run of 9 to 18 digits, wherever it appears in a message.
Watch out: A filter like this is a second line of defence, not the first. It cannot recognise a password. The first rule stays: do not pass secrets to the logger at all.
Common mistakes
Calling basicConfig in a library
A library is code that other programs import. It must not decide where log messages go. That is the job of the program that uses it. A library only creates its logger. At most it adds a NullHandler, which does nothing.
The root logger everywhere
Writing logging.info(...) in every file puts every message under the name root. You can no longer tell which module wrote a line, or make one module quieter. Use logger = logging.getLogger(__name__) at the top of each file.
Logging in a tight loop
One log line for each of a million rows makes the program slow and the log useless. Log a summary, or one line every so often.
Swallowing exceptions
except Exception: pass hides the error completely. The program goes on with wrong data and nobody knows why. If you catch an exception, either handle it properly or log it with its traceback.
Also avoid: logging the same error at every level of the call chain. Log it once, at the place that decides what to do about it.
Quick reference
| I want to | Use |
|---|---|
| Understand a crash | Read the traceback from the bottom up |
| See a value and its name | print(f"{value=}") |
| See hidden spaces, quotes and the type | repr(value), type(value) |
| State something that must be true | assert condition, "message" |
| Reject bad input | raise ValueError("message") |
| Pause the program and look around | breakpoint(), then n, s, p, c |
| Open the debugger where it crashed | python3 -m pdb script.py, or pdb.pm() |
| Stop only in the odd case | b 9, balance < 0, or if ...: breakpoint() |
| Ignore all breakpoints for one run | PYTHONBREAKPOINT=0 |
| Find who triggers a warning | python3 -W error script.py |
| Keep a traceback as text | traceback.format_exc() |
| Set up logging quickly | logging.basicConfig(level=..., format=...) |
| Get a logger in a module | logger = logging.getLogger(__name__) |
| Log an error with its traceback | logger.exception("message") inside except |
| Log to a file and the screen | Two handlers on one logger |
| Stop the log file growing for ever | RotatingFileHandler |
| Set up a whole application | logging.config.dictConfig(...), once, at the start |
Next in the practice path: Python Code Quality.