Learnings

Python in practice, part 3 of 9

Python Debugging and Logging

Debugging is finding out why a program does something you did not expect. Logging is making the program write down what it is doing, so that you can find out later without being there. This page covers both, using a small bank as the example. Everything here is in the standard library, so nothing needs pip install.

Debugging

Part 1 of 2 · finding out what went wrong

A bug is a difference between what you think the code does and what it really does. Debugging closes that gap. You read what Python tells you, look at the real values, and test one idea at a time.

The tools go from simple to powerful. Most bugs are solved with the first two.

Reading a traceback

When an exception is not caught, Python stops and prints a traceback: the list of function calls that were in progress when the error happened. Here is a small script with a bug. It is marked as not runnable on purpose, because it crashes.

Running it with python3 transfer.py prints this. Only the folder name has been shortened.

Read it from the bottom up.

  • The last line is the error itself: the exception type (KeyError) and its message ('Hemkesh'). A dictionary was asked for a key it does not have.
  • The pair of lines just above it is where the error happened: file, line number, function name, then the line of code. Here it is line 5, inside get_balance.
  • Each pair above that is the caller of the one below. get_balance was called by transfer on line 10, which was called by main on line 14, which was called from the top of the file (<module>) on line 18.
  • The first line says "most recent call last". That is why the place of the crash is at the bottom and the start of the program is at the top.

Each pair is called a frame: one function call that has started and not finished yet. The bug is often not in the bottom frame. Line 5 is fine. The real question is one frame up: why did transfer ask for an account that does not exist?

Tip: In a long traceback, look for the lowest frame that is in your own file. Frames inside library code usually only show how your bad value travelled. Python 3.11 and later also draw ^^^^ marks under the exact part of the line that failed.

Chained exceptions

Sometimes a traceback has two parts joined by a sentence. There are two such sentences, and they mean different things.

During handling of the above exception

A second error happened by accident inside an except block. This block is not runnable because it crashes: the name nme is a typing mistake.

The program died of the last error, the NameError. The first part shows what the code was busy handling at the time. Fix the bottom one first.

The above exception was the direct cause

Here the second error was raised on purpose with raise ... from .... The code turns a low-level error into one that makes sense for a bank, and keeps the original attached. Also not runnable, because it ends in an uncaught exception.

The original error is stored on the new one as __cause__, so code can look at it too.

Tip: Write raise NewError(...) from None when the original error would only confuse the reader. The first part of the traceback is then left out.

Common exceptions

A handful of exception types cover most crashes. Knowing what each usually means saves a lot of reading.

ExceptionWhat it usually meansFirst thing to check
SyntaxErrorPython could not even read the fileA missing bracket, quote or colon, often on the line before the one reported
IndentationErrorThe spaces at the start of a line do not line upMixed tabs and spaces, or a block that was not indented
NameErrorA name is used that does not existA spelling mistake, or a variable used before it is assigned
UnboundLocalErrorA function reads a local variable before giving it a valueA variable assigned only inside an if, or a global assigned inside the function
TypeErrorThe value has the wrong type for the operationA string where a number was expected, or the wrong number of arguments
ValueErrorThe type is right but the value is not acceptableint("12,000"), or unpacking the wrong number of items
KeyErrorA dictionary does not have that keyPrint the keys. Is it a spelling, case or type difference ("7" and 7)?
IndexErrorA list position is past the endAn empty list, or counting from 1 instead of 0
AttributeErrorThe object has no attribute of that nameIf the message says 'NoneType' object, a function returned None and you did not expect it
ZeroDivisionErrorDivision by zeroAn empty list when working out an average
FileNotFoundErrorNo file at that pathThe folder the program was started from, shown by os.getcwd()
ModuleNotFoundErrorThe module in an import cannot be foundThe package is not installed in this environment, or the wrong Python is running
RecursionErrorA function called itself too many timesA missing or unreachable stopping condition
AssertionErrorAn assert statement was falseThe condition on the reported line
KeyboardInterruptSomeone pressed Ctrl+CNot a bug. The traceback shows where the program was when it was stopped, which helps find an endless loop

Two of them, caught and printed so that the block runs to the end:

Catching and handling exceptions is covered in Python Errors and Files.

assert as a sanity check

assert condition, message does nothing when the condition is true and raises AssertionError when it is false. Use it to state something that must be true if your own code is correct. When it fails, you have found a bug close to where it started.

Do not use assert to check input from users, files or the network. Python has a switch, -O (optimise), that removes every assert from the program. The block below runs the same two lines twice, once normally and once with -O.

With -O the check is simply gone. For input, raise a real exception instead, because it can never be switched off:

Rule: assert is for mistakes made by the programmer. raise is for bad data from outside.

breakpoint() and pdb

A debugger stops a running program so that you can look at every variable and run the code one line at a time. Python comes with one called pdb. Put breakpoint() on any line, and the program pauses there and gives you a (Pdb) prompt.

This script has a bug: one balance was typed as text. It is not runnable here because breakpoint() waits for you to type commands.

At the prompt you type short commands. These are the ones that matter.

CommandLong formWhat it does
llistShow the code around the current line. ll shows the whole function
nnextRun the current line and stop at the next one. A function call is run in one go
sstepLike n, but if the line calls a function, go inside it
ccontinueRun until the next breakpoint or the end of the program
p exprprintPrint the value of any expression, such as p balance or p len(accounts)
pp exprPretty-print: the same, but long lists and dictionaries are laid out over several lines
wwhereShow the stack: the chain of calls that led here, like a traceback
uupMove to the caller's frame, to see its variables
ddownMove back down one frame
b 9breakSet a breakpoint on line 9. b alone lists the breakpoints. b yearly_interest stops when that function is called
rreturnRun until the current function returns
qquitLeave the debugger and stop the program
!statementRun a Python statement, for example !balance = 0. The ! stops pdb from reading it as one of its own commands
hhelpList all commands. h b explains one of them

Tip: Pressing Enter on an empty line repeats the last command. That makes it easy to walk through code with n, Enter, Enter, Enter.

A pdb session

This is a real session with interest.py from the previous chapter, started with python3 interest.py. The lines after (Pdb) are what was typed. A line starting with > shows the file, line number and function where the program is paused, and the -> line below it is the line that will run next.

What happened, step by step:

  • pp accounts showed the data before anything ran.
  • s stepped into report. Three n commands walked to line 9, and p name, balance showed the first account.
  • s stepped into yearly_interest. After two more lines, p interest showed 200.0, which is correct.
  • w printed the stack. The > marks the current frame. u moved up to report, where total is still 0 because line 9 has not finished.
  • b 9 set a breakpoint and c ran to it. Now it is Sandhya's turn, and her balance is the text '12000'. That is the bug.
  • !balance = int(balance) repaired the value in the running program, and c let it finish with the right total, 680.0.

Remember: Changing a value inside pdb only proves your idea about the bug. The real fix still has to go into the file: store 12000 as a number.

Post-mortem debugging

Post-mortem means "after death". You do not add breakpoint() anywhere. You let the program crash, and the debugger opens at the exact line that failed, with all the variables still there.

The script is not runnable here because it crashes. Start it under pdb with python3 -m pdb crash.py. pdb pauses on the first line, so type c to let it run.

Two other ways to get the same thing:

  • Run python3 -i crash.py. The -i switch leaves you at the >>> prompt after the crash. Then type import pdb and pdb.pm(). pm stands for post-mortem, and it opens the debugger on the last traceback.
  • Inside an except block, call pdb.post_mortem() to open the debugger on the exception being handled.

When to use it: Post-mortem is the fastest route when the program crashes with an exception. Use breakpoint() when nothing crashes but the answer is wrong.

Conditional breakpoints

A loop may run a thousand times and go wrong only once. Stopping on every pass is no use. A conditional breakpoint stops only when a condition is true.

The simplest form is an ordinary if in the code. Not runnable here, because it would wait for commands.

The other form needs no change to the file. In pdb, write the condition after the line number, separated by a comma. This is crash.py from the previous chapter, started with python3 -m pdb crash.py:

The first account passed line 2 without stopping. The debugger stopped only for the one whose balance is a string, before the line crashed.

Switching breakpoints off, and IDE debuggers

breakpoint() reads an environment variable called PYTHONBREAKPOINT. An environment variable is a named setting that the shell passes to a program. Set it to 0 and every breakpoint() call does nothing, without editing the code.

The block below proves it. It writes a two-line script that contains breakpoint(), then runs it with the variable set, so nothing waits for the keyboard.

Watch out: A breakpoint() left in code that runs on a server will make it hang, waiting for a keyboard that is not there. Search for breakpoint( before every commit.

IDE debuggers

An IDE (integrated development environment) is an editor with tools built in. VS Code and PyCharm both have a graphical debugger that does the same job as pdb. You click next to a line number to set a breakpoint, press the debug button, and the editor shows all variables in a side panel and the call stack as a list. Buttons called Step Over, Step Into and Continue are n, s and c. Right-clicking a breakpoint lets you add a condition. The ideas are exactly the ones in this page, so learning pdb first means you can debug anywhere, including on a server that has no editor.

A method for finding bugs

Tools help, but a steady method helps more. Changing lines at random until the error goes away is slow and leaves bugs behind.

  1. Reproduce it. Find the exact steps or input that make the bug happen every time. A bug you cannot repeat is a bug you cannot prove fixed.
  2. Reduce it. Remove everything that is not needed to show the bug: fewer accounts, fewer steps, a smaller file. Often the cause shows itself during this step.
  3. Read the error. The whole message, slowly, from the bottom up. It usually says what is wrong and on which line.
  4. Form one hypothesis. A hypothesis is a guess you can test: "the balance is a string". Check it with a print or the debugger before changing anything.
  5. Change one thing. Make one change, run again, and look. If it did not help, undo it. Two changes at once tell you nothing about which one mattered.

Apply it to transfer.py from the first chapter. The error says KeyError: 'Hemkesh' on the receiver's line. Hypothesis: the money had already left the sender when the crash came. One print proves it.

So there are two bugs. The crash is the small one. The serious one is that 500 rupees vanished. The fix is to check everything first and change the balances only after that.

Tip: Once a bug is fixed, write a test that fails with the old code and passes with the new. Then it cannot come back unnoticed. See Python Testing.

Warnings

A warning is a message that something is likely to become a problem, while the program carries on. The most common kind is DeprecationWarning: a function still works but will be removed one day. The warnings module issues and controls them.

Normally a warning is printed once to the error stream with the file and line that caused it. stacklevel=2 makes that line the caller's line, which is the one that has to change. The catch_warnings block is used here only to collect the warning so it can be printed in a fixed form.

While hunting a bug, it helps to turn warnings into exceptions. Then you get a full traceback showing who made the call.

From the terminal: python3 -W error script.py does the same for a whole run without touching the code, and python3 -W default script.py shows warnings that are hidden by default.

The traceback module

Sometimes a program must survive an error and still keep the full report, for example a bank job that processes many accounts overnight. The traceback module gives you the traceback as text, so you can save it and carry on.

The loop finished all three accounts, and the string in problems holds the same text Python would have printed, with file and line number.

FunctionWhat it gives
traceback.format_exc()The traceback of the exception being handled, as a string
traceback.print_exc()The same, printed straight to the error stream
traceback.print_stack()The current call stack, with no exception needed. Good for "who called this function?"

In real programs you rarely call these yourself. logger.exception(), shown in a later chapter, does the same and writes the result to the log.

Logging

Part 2 of 2 · a record of what the program did

A log is a list of messages a program writes while it runs: what it did, when, and what went wrong. When something fails at night on a server, the log is the only witness. Python's logging module is the standard way to write one.

Why not print

print() is fine while you sit in front of the program. It is a poor record once the program runs on its own. Logging gives four things that print does not.

Needprintlogging
LevelsEvery message looks equally importantEach message has a level, from DEBUG to CRITICAL
DestinationsThe screen onlyScreen, file, several files, or all at once
ContextOnly what you typedTime, level, module name and line number added for you
Turning it offDelete or comment out every callChange one setting. The calls stay in the code

About the examples: Logging writes to the error stream (stderr) by default and normally includes the time. The runnable blocks on this page send it to sys.stdout and leave the time out, so the output is the same on every run.

The five levels

Every log message has a level that says how serious it is. A logger also has a level, and it drops every message below it. So one setting decides how much detail you get.

LevelNumberUse it forBank example
DEBUG10Detail that only helps while finding a bug"Balance before the deposit was 5000"
INFO20Normal events worth recording"Deposited 500 for Rishikesh"
WARNING30Something odd happened, but the work was done"Balance for Hemkesh is below the minimum"
ERROR40One piece of work failed"Transfer to Malinikesh failed"
CRITICAL50The whole program cannot continue"Cannot reach the accounts database"

The level was set to WARNING, so the DEBUG and INFO messages were dropped. WARNING is also the default when you set nothing.

Rule of thumb: Use DEBUG while developing and INFO or WARNING in production. If an ERROR appears in the log, a person should look at it.

basicConfig

logging.basicConfig() sets up logging in one call: the lowest level to show, where to write, and what each line looks like. The format string is built from named fields.

FieldMeaning
%(asctime)sDate and time, such as 2026-11-17 10:15:02,123
%(levelname)sDEBUG, INFO, WARNING, ERROR or CRITICAL
%(name)sName of the logger. root when you call logging.info() directly
%(message)sThe message itself
%(module)s, %(funcName)s, %(lineno)dThe file (without .py), function and line number that made the call

A usual format for real work is "%(asctime)s %(levelname)s %(name)s: %(message)s". To write to a file, pass filename="bank.log" instead of stream.

Watch out: basicConfig() works only the first time. If logging is already set up, later calls do nothing. Pass force=True (Python 3.8 and later) to replace the old setup.

A logger for each module

Calling logging.info() uses one shared logger called the root logger. In a real program, each file should have its own logger, created once at the top with logging.getLogger(__name__).

__name__ is the module's name. Inside bank/transfers.py it is "bank.transfers". In the file you run directly it is "__main__".

Why bother:

  • Every line in the log says which module wrote it.
  • You can make one noisy module quieter, or one suspect module more detailed, without touching the others.
  • getLogger("bank.transfers") always returns the same object for the same name, so nothing has to be passed around.

Logging an exception

Inside an except block, logger.exception("message") writes the message at ERROR level and adds the full traceback below it. You get the whole story, and the program can carry on.

The output, with the folder name shortened:

logger.error("...") alone would log only the message. logger.exception("...") is the same as logger.error("...", exc_info=True). Use exc_info=True with another level, such as logger.warning, when a failure is expected and not serious.

Watch out: Call logger.exception() only inside an except block. Outside one there is no exception to report.

Lazy formatting

Log calls take a message with %s placeholders and the values as separate arguments. The logger joins them only if the message will really be written. An f-string builds the text first, even when the level is switched off and the text is thrown away.

Neither call logged anything, but the f-string still did the work. There is a second reason to prefer %s arguments: the message text stays the same for every call, so tools that group log lines can see that they are the same event.

Fair point: For cheap values the speed difference is tiny, and many teams allow f-strings in log calls. The habit matters most for DEBUG messages inside loops.

Handlers and formatters

Three kinds of object work together:

  • A logger is what your code calls. It decides whether a message is important enough.
  • A handler sends the message somewhere: StreamHandler to the screen, FileHandler to a file.
  • A formatter turns the message into a line of text. Each handler has its own.

One logger can have several handlers, each with its own level and format. A common setup: short warnings on the screen, everything in a file.

A message has to pass two gates: the logger's level, then each handler's level. In a real file format you would start with %(asctime)s, which gives lines such as 2026-11-17 10:15:02,123 INFO bank withdraw: Withdrew 4500 for Hemkesh. A second argument to Formatter changes the time layout, for example datefmt="%Y-%m-%d %H:%M:%S".

Rotating log files

A log file that only grows will fill the disk one day. RotatingFileHandler starts a new file when the current one reaches a size limit and keeps only a fixed number of old ones. The newest lines are always in bank.log. Older ones move to bank.log.1, then bank.log.2, and the oldest file is deleted.

Thirty lines were written, but only the last fifteen are kept, five in each of three small files. The limit here is tiny so the effect is visible. A real setting looks like maxBytes=5_000_000, backupCount=5: five megabytes per file, five old files.

Also useful: TimedRotatingFileHandler("bank.log", when="midnight", backupCount=14) starts a new file every day and keeps two weeks.

The logger hierarchy

Logger names with dots form a family tree. bank.transfers is a child of bank, and bank is a child of the root logger. A message logged on a child is also passed up to the handlers of its parents. This is called propagation.

So you attach a handler once, high up, and every module below it is covered.

Two rules explain the output:

  • Level: a logger with no level of its own uses its nearest parent's. bank.accounts has none, so it uses INFO (20) from bank.
  • Handlers: the child loggers have no handlers. Their messages travel up to bank, which prints them.

If a message appears twice in your log, a handler is attached at two levels of the tree. Remove one, or set propagate = False on the child to stop messages travelling up.

Configure once, at the start

Logging should be set up in one place, at the very start of the program, usually in main(). Every other file only creates its logger and uses it. For anything beyond basicConfig, describe the whole setup as a dictionary and pass it to logging.config.dictConfig().

The dictionary has the same three parts as before: formatters, handlers that use a formatter, and loggers that use handlers. "version": 1 is required. "disable_existing_loggers": False keeps loggers that were created before this call working, which is nearly always what you want.

Tip: Because it is plain data, the dictionary can be loaded from a JSON file, so the logging setup can change without editing code.

Structured logging

A line such as "Deposited 500 for Rishikesh" is easy for a person to read and hard for a program to search. Structured logging writes each event as data, usually one JSON object per line, so a log tool can filter by field: every event for one account, or every amount over 10,000.

A formatter is just a class with a format(record) method, so a tiny one of your own is enough. Extra fields are passed with the extra argument and become attributes of the record.

A real version would also add the time, for example self.formatTime(record). It is left out here so the output does not change between runs. Packages such as python-json-logger and structlog do this job fully. They need pip install.

What never to log

Log files are copied, backed up, sent to other systems and read by many people. Anything written to a log should be treated as public inside the company. Never log:

  • Passwords, PINs, one-time codes and security answers
  • Full card numbers, CVV codes and expiry dates
  • Full account numbers and government identity numbers
  • API keys, tokens and session cookies

Log an identifier that is safe, such as an internal customer id, or mask the value so only the last four digits remain.

People forget, so add a safety net. A filter is an object that sees every record before it is written. This one replaces any run of 9 to 18 digits, wherever it appears in a message.

Watch out: A filter like this is a second line of defence, not the first. It cannot recognise a password. The first rule stays: do not pass secrets to the logger at all.

Common mistakes

Calling basicConfig in a library

A library is code that other programs import. It must not decide where log messages go. That is the job of the program that uses it. A library only creates its logger. At most it adds a NullHandler, which does nothing.

The root logger everywhere

Writing logging.info(...) in every file puts every message under the name root. You can no longer tell which module wrote a line, or make one module quieter. Use logger = logging.getLogger(__name__) at the top of each file.

Logging in a tight loop

One log line for each of a million rows makes the program slow and the log useless. Log a summary, or one line every so often.

Swallowing exceptions

except Exception: pass hides the error completely. The program goes on with wrong data and nobody knows why. If you catch an exception, either handle it properly or log it with its traceback.

Also avoid: logging the same error at every level of the call chain. Log it once, at the place that decides what to do about it.

Quick reference

I want toUse
Understand a crashRead the traceback from the bottom up
See a value and its nameprint(f"{value=}")
See hidden spaces, quotes and the typerepr(value), type(value)
State something that must be trueassert condition, "message"
Reject bad inputraise ValueError("message")
Pause the program and look aroundbreakpoint(), then n, s, p, c
Open the debugger where it crashedpython3 -m pdb script.py, or pdb.pm()
Stop only in the odd caseb 9, balance < 0, or if ...: breakpoint()
Ignore all breakpoints for one runPYTHONBREAKPOINT=0
Find who triggers a warningpython3 -W error script.py
Keep a traceback as texttraceback.format_exc()
Set up logging quicklylogging.basicConfig(level=..., format=...)
Get a logger in a modulelogger = logging.getLogger(__name__)
Log an error with its tracebacklogger.exception("message") inside except
Log to a file and the screenTwo handlers on one logger
Stop the log file growing for everRotatingFileHandler
Set up a whole applicationlogging.config.dictConfig(...), once, at the start

Next in the practice path: Python Code Quality.