Python advanced, part 1 of 5
Python Decorators
A decorator is a function that takes another function and gives back a new one that does a little more. It lets you add logging, checks, caching or retries to many functions without touching the code inside them. This page builds the idea one small step at a time, starting from plain functions, and uses a small family bank as the running example.
Foundations
Part 1 of 4 · what a function really is
A decorator uses nothing new. It is built from three things Python functions can already do. Once these three feel ordinary, the decorator itself is a short step.
- Functions are objects: store them, pass them, return them
- Inner functions: define a function inside a function
- Closures: an inner function that remembers values
Functions are objects
A function is a value, just like a number or a list. The name deposit is a variable that points to it. Without brackets you get the function itself. With brackets you call it.
Pass a function to a function
Because a function is a value, you can hand it to another function as an argument.
Return a function from a function
A function can also give a function back as its result.
Remember: a decorator does both at once. It takes a function in and returns a function out.
Inner functions
You can write a def inside another def. The inner function exists only while the outer one runs, like any other local variable.
The outer function can also return the inner function instead of calling it. The caller then holds a function that was made on the spot.
Closures
An inner function can use the variables of the function around it. If the inner function is returned, it keeps those variables alive and carries them along. A function that remembers values like this is called a closure.
make_interest finished long before savings(10000) ran, yet rate was still there. Each call to make_interest makes a new closure with its own copy.
A closure can also change what it remembers
To assign to a remembered variable, declare it with nonlocal. Without that word Python would create a new local variable instead.
Concept: a closure remembers the variables it uses from the enclosing function, not the whole function. In a decorator, the remembered value is the original function.
Writing decorators
Part 2 of 4 · wrap a function, step by step
Now put the three ideas together. Take a function in, define an inner function that calls it and does something extra, and return the inner function. Each chapter here fixes one weakness of the chapter before it.
- The first decorator: written by hand, no special syntax
- The @ syntax: a shorter way to write the same thing
- Arguments and return value: make the wrapper fit any function
- functools.wraps: keep the name and docstring
- Stacking decorators: more than one, and their order
- Decorating methods: the same thing inside a class
The first decorator
We want a line printed before and after every deposit, without editing deposit. announce takes the function, builds a wrapper around it and returns the wrapper. The wrapper is a closure: it remembers func.
After the line deposit = announce(deposit), the name deposit points to wrapper. Every caller gets the extra behaviour without knowing it.
Definition: a decorator is any callable that takes a function and returns something to use in its place, usually a new function that wraps the old one.
The @ syntax
Writing deposit = announce(deposit) after every function is easy to forget. Python has a shorthand: put @announce on the line above the def. It means exactly the same thing.
Timing: the decorator runs once, when Python reads the def. The wrapper runs every time the function is called.
Arguments and return value
The wrapper so far has two problems. It only fits functions with exactly two parameters, and it throws away what the function returns.
The fix has two parts. Accept *args, **kwargs, which collect any positional and keyword arguments, and pass them straight on. Then return whatever the function returned.
Rule: unless the decorator needs to look at specific arguments, write the wrapper as wrapper(*args, **kwargs) and return the result. Then it works on any function.
functools.wraps
After decorating, the name transfer points to wrapper. So the function's own name and docstring are gone. That confuses help(), error messages and debugging tools.
functools.wraps is a decorator for the wrapper. It copies the name, docstring and a few other details from the original function onto the wrapper.
wraps also stores the original function as __wrapped__, which is handy in tests when you want to call it without the decorator.
Habit: put @functools.wraps(func) on every wrapper you write. The rest of this page always does.
Stacking decorators
A function can have several decorators. Python applies them from the bottom up, so the one closest to the def wraps first and the top one ends up on the outside.
Think of layers around the function. A call enters through the top decorator, moves inward to the function, then comes back out the same way.
Order matters: with @audit on top, a refused deposit still passes through the audit layer. Swap the two lines and the check runs first, so a refused deposit never reaches audit.
Decorating methods
A method is a function whose first argument is self. A wrapper written with *args, **kwargs passes self through without any change. If the decorator needs the object, name self as the wrapper's first parameter.
With built-in decorators: when you combine your own decorator with @property, @staticmethod or @classmethod, put the built-in one on top so that yours wraps the plain function.
Decorators with parameters
Part 3 of 4 · settings, state and classes
So far a decorator did one fixed thing. Often you want to set it up first: retry 3 times, allow only a manager. That needs one more layer of function. This part also shows decorators written as classes, and decorators applied to classes.
- A decorator that takes arguments:
@require_role("manager") - Why three levels: what runs when
- With or without brackets: one decorator, both spellings
- Class-based decorators: an object with
__call__ - Decorating classes: the same idea, one level up
A decorator that takes arguments
Start with a decorator that lets only a manager through. The role is fixed inside the code.
A teller-only decorator would be a copy with one word changed. Better to pass the role in. Wrap the whole decorator in one more function that takes the role, and return the decorator from it.
Name: the outer function, require_role, is often called a decorator factory, because calling it makes a decorator.
Why three levels
Python allows any expression after @. It works that expression out first, then uses the result as the decorator. So @require_role("manager") first calls require_role, and what comes back must be an ordinary decorator. Each level has its own job and runs at its own time.
| Level | Receives | Runs | Returns |
|---|---|---|---|
| 1. factory | the settings, such as "manager" | once, on the @ line | the decorator |
| 2. decorator | the function | once, right after the def | the wrapper |
| 3. wrapper | the arguments of one call | on every call | the result |
Prints at each level show the order.
Written without @, the two steps are easy to see.
Concept: the wrapper is a closure over two things now: func from level 2 and the settings from level 1.
With or without brackets
This chapter is optional. Some decorators can be written both as @audit and as @audit(label="LOAN"). The trick: when used bare, Python passes the function as the first argument. When used with brackets, the first argument is missing, so the function returns a decorator instead.
Why the star: the * forces label to be given by name. Then the first positional argument can only ever be the function, and the two cases cannot be mixed up.
Class-based decorators
An object can be called like a function if its class defines __call__. So a class can be a decorator: __init__ receives the function and __call__ plays the wrapper. This reads well when the decorator keeps state, such as a count.
For a class-based decorator with parameters, the roles shift by one: __init__ takes the settings and __call__ takes the function and returns a wrapper.
Limit: the CountCalls shape does not work on methods as it stands, because the object replaces the function and self is no longer passed in. For methods, use a function-based decorator.
Decorating classes
The @ line also works above a class. The decorator receives the class and returns the class to use. A common use is a registry: a dictionary that collects classes as they are defined.
A class decorator can also add something to the class before returning it.
You have met one: @dataclass is a class decorator. It reads the fields of the class and adds __init__, __repr__ and __eq__ for you.
Use cases
Part 4 of 4 · where decorators earn their place
A decorator fits when the same extra step is needed around many functions and has nothing to do with what each function is for. These are the jobs you will meet most often, each as a small working example.
- Logging and audit: record every transfer
- Timing: measure how long a call takes
- Retry on failure: try again when a call fails
- Caching: remember answers already worked out
- Permission and input checks: stop a bad call before it starts
- Registering handlers: collect functions in a table
- Limiting calls: count and refuse
- Built-in decorators: the ones Python ships with
- Quick reference and common mistakes
Logging and audit
A bank must record every transfer, including the ones that fail. The decorator writes one line per call. transfer itself stays about moving money.
In real code: replace the list with the logging module or a database table. The decorator stays the same shape.
Timing
Read a clock before the call and after it. time.perf_counter() is the clock made for measuring short durations.
About the output: the real time is a tiny fraction of a second and differs on every run, so this example rounds it to whole seconds. In your own code print more digits, for example {elapsed:.4f}.
Retry on failure
Calls over a network sometimes fail and then work a moment later. A retry decorator calls the function again, up to a limit. The number of tries is a setting, so this is a three-level decorator.
In real code: wait between tries with time.sleep(), and retry only errors that can go away by themselves. Retrying a ValueError would fail the same way every time.
Caching
If a function always gives the same answer for the same arguments, the answer can be stored and reused. This is called memoizing. The store is a dictionary kept in the closure, with the arguments as the key.
functools.lru_cache
You rarely need to write this yourself. functools.lru_cache does the same job, handles keyword arguments, and can limit how many answers it keeps. LRU means "least recently used": when the cache is full, the answer that has gone unused the longest is dropped.
Limits: the arguments must be hashable, so a list or dictionary argument raises TypeError. And do not cache a function whose answer changes over time, such as a live balance.
Permission and input checks
A check decorator looks at the arguments before the function runs and raises an error if something is wrong. The function body can then assume its input is fine.
Permission
Only the owner may take money out of an account.
Input validation
The limits are settings, so this one takes parameters.
Registering handlers
Some decorators do not wrap anything. They note the function down in a table and return it unchanged. The program can then find the right function by name, and adding a new command means writing one new function, with no if chain to edit. Plugin systems and web frameworks use this shape for their routes.
Limiting calls
A decorator can count calls in its closure and refuse once a limit is reached. Here a cash machine allows three withdrawals.
Rate limiting: a real rate limiter, such as "5 calls per minute", works the same way. It stores the time of each call instead of a plain count and forgets the ones older than a minute.
Built-in decorators
You have already used decorators that come with Python. Now you can read them for what they are: functions or classes that take your function or class and return something in its place.
| Decorator | Goes on | What it does |
|---|---|---|
@property | method | Lets you read the method like an attribute, without brackets |
@staticmethod | method | A plain function kept inside the class. It gets no self |
@classmethod | method | Receives the class as cls instead of an object. Often used for extra ways to build an object |
@dataclass | class | Writes __init__, __repr__ and __eq__ from the listed fields |
@functools.cached_property | method | Like @property, but worked out once per object and then stored |
@functools.lru_cache | function | Remembers results by argument |
@functools.wraps | wrapper | Copies name and docstring onto a wrapper |
Quick reference
Find what you need in the left column. The right column is the shape to write.
| You need | Shape of the decorator | Chapter |
|---|---|---|
| Do something before or after every call | Two levels: decorator(func) returns wrapper(*args, **kwargs) | Arguments and return value |
| Keep the name and docstring | @functools.wraps(func) on the wrapper | functools.wraps |
| Settings such as a role or a count | Three levels: factory(settings) returns decorator(func) returns wrapper | A decorator that takes arguments |
Both @name and @name(...) | def name(func=None, *, option=...) | With or without brackets |
| State you want to read from outside | A class with __init__(func) and __call__ | Class-based decorators |
| State only the wrapper needs | A variable in the closure, changed with nonlocal | Limiting calls |
| Check arguments or permissions | Wrapper raises before calling func | Permission and input checks |
| Try again after an error | Wrapper calls func in a loop with try | Retry on failure |
| Remember results | @functools.lru_cache, or a dictionary in the closure | Caching |
| Collect functions or classes in a table | No wrapper: store it and return it unchanged | Registering handlers |
| Use it on a method | Same as a function. self is the first argument | Decorating methods |
| Change or record a class | decorator(cls) returns cls | Decorating classes |
Common mistakes
Forgetting to return the wrapper
A function without return gives back None. So the decorated name becomes None, and calling it fails.
Forgetting to return the result
The wrapper calls the function but does not pass the answer on. Nothing fails. The caller just gets None, which is harder to notice.
Forgetting functools.wraps
Every decorated function then reports the name wrapper and loses its docstring. See functools.wraps.
Leaving the brackets off a decorator with parameters
A three-level decorator must be called on the @ line. Without brackets, the function is passed in as the setting, and the name ends up pointing to the middle level. There is no error at the definition. The first call quietly returns a wrapper instead of doing the work.
The opposite slip is adding brackets to a two-level decorator, as in @announce(). That one fails at once with a TypeError, because announce is called without the function it needs.
Check: if a decorated function returns a function when you expected a value, look at the @ line first.
Next, see how a for loop really works and how to produce values one at a time in Python Iterators and Generators.