Learnings

Python advanced, part 1 of 5

Python Decorators

A decorator is a function that takes another function and gives back a new one that does a little more. It lets you add logging, checks, caching or retries to many functions without touching the code inside them. This page builds the idea one small step at a time, starting from plain functions, and uses a small family bank as the running example.

Foundations

Part 1 of 4 · what a function really is

A decorator uses nothing new. It is built from three things Python functions can already do. Once these three feel ordinary, the decorator itself is a short step.

Functions are objects

A function is a value, just like a number or a list. The name deposit is a variable that points to it. Without brackets you get the function itself. With brackets you call it.

Pass a function to a function

Because a function is a value, you can hand it to another function as an argument.

Return a function from a function

A function can also give a function back as its result.

Remember: a decorator does both at once. It takes a function in and returns a function out.

Inner functions

You can write a def inside another def. The inner function exists only while the outer one runs, like any other local variable.

The outer function can also return the inner function instead of calling it. The caller then holds a function that was made on the spot.

Closures

An inner function can use the variables of the function around it. If the inner function is returned, it keeps those variables alive and carries them along. A function that remembers values like this is called a closure.

make_interest finished long before savings(10000) ran, yet rate was still there. Each call to make_interest makes a new closure with its own copy.

A closure can also change what it remembers

To assign to a remembered variable, declare it with nonlocal. Without that word Python would create a new local variable instead.

Concept: a closure remembers the variables it uses from the enclosing function, not the whole function. In a decorator, the remembered value is the original function.

Writing decorators

Part 2 of 4 · wrap a function, step by step

Now put the three ideas together. Take a function in, define an inner function that calls it and does something extra, and return the inner function. Each chapter here fixes one weakness of the chapter before it.

The first decorator

We want a line printed before and after every deposit, without editing deposit. announce takes the function, builds a wrapper around it and returns the wrapper. The wrapper is a closure: it remembers func.

After the line deposit = announce(deposit), the name deposit points to wrapper. Every caller gets the extra behaviour without knowing it.

Definition: a decorator is any callable that takes a function and returns something to use in its place, usually a new function that wraps the old one.

The @ syntax

Writing deposit = announce(deposit) after every function is easy to forget. Python has a shorthand: put @announce on the line above the def. It means exactly the same thing.

Timing: the decorator runs once, when Python reads the def. The wrapper runs every time the function is called.

Arguments and return value

The wrapper so far has two problems. It only fits functions with exactly two parameters, and it throws away what the function returns.

The fix has two parts. Accept *args, **kwargs, which collect any positional and keyword arguments, and pass them straight on. Then return whatever the function returned.

Rule: unless the decorator needs to look at specific arguments, write the wrapper as wrapper(*args, **kwargs) and return the result. Then it works on any function.

functools.wraps

After decorating, the name transfer points to wrapper. So the function's own name and docstring are gone. That confuses help(), error messages and debugging tools.

functools.wraps is a decorator for the wrapper. It copies the name, docstring and a few other details from the original function onto the wrapper.

wraps also stores the original function as __wrapped__, which is handy in tests when you want to call it without the decorator.

Habit: put @functools.wraps(func) on every wrapper you write. The rest of this page always does.

Stacking decorators

A function can have several decorators. Python applies them from the bottom up, so the one closest to the def wraps first and the top one ends up on the outside.

Think of layers around the function. A call enters through the top decorator, moves inward to the function, then comes back out the same way.

Order matters: with @audit on top, a refused deposit still passes through the audit layer. Swap the two lines and the check runs first, so a refused deposit never reaches audit.

Decorating methods

A method is a function whose first argument is self. A wrapper written with *args, **kwargs passes self through without any change. If the decorator needs the object, name self as the wrapper's first parameter.

With built-in decorators: when you combine your own decorator with @property, @staticmethod or @classmethod, put the built-in one on top so that yours wraps the plain function.

Decorators with parameters

Part 3 of 4 · settings, state and classes

So far a decorator did one fixed thing. Often you want to set it up first: retry 3 times, allow only a manager. That needs one more layer of function. This part also shows decorators written as classes, and decorators applied to classes.

A decorator that takes arguments

Start with a decorator that lets only a manager through. The role is fixed inside the code.

A teller-only decorator would be a copy with one word changed. Better to pass the role in. Wrap the whole decorator in one more function that takes the role, and return the decorator from it.

Name: the outer function, require_role, is often called a decorator factory, because calling it makes a decorator.

Why three levels

Python allows any expression after @. It works that expression out first, then uses the result as the decorator. So @require_role("manager") first calls require_role, and what comes back must be an ordinary decorator. Each level has its own job and runs at its own time.

LevelReceivesRunsReturns
1. factorythe settings, such as "manager"once, on the @ linethe decorator
2. decoratorthe functiononce, right after the defthe wrapper
3. wrapperthe arguments of one callon every callthe result

Prints at each level show the order.

Written without @, the two steps are easy to see.

Concept: the wrapper is a closure over two things now: func from level 2 and the settings from level 1.

With or without brackets

This chapter is optional. Some decorators can be written both as @audit and as @audit(label="LOAN"). The trick: when used bare, Python passes the function as the first argument. When used with brackets, the first argument is missing, so the function returns a decorator instead.

Why the star: the * forces label to be given by name. Then the first positional argument can only ever be the function, and the two cases cannot be mixed up.

Class-based decorators

An object can be called like a function if its class defines __call__. So a class can be a decorator: __init__ receives the function and __call__ plays the wrapper. This reads well when the decorator keeps state, such as a count.

For a class-based decorator with parameters, the roles shift by one: __init__ takes the settings and __call__ takes the function and returns a wrapper.

Limit: the CountCalls shape does not work on methods as it stands, because the object replaces the function and self is no longer passed in. For methods, use a function-based decorator.

Decorating classes

The @ line also works above a class. The decorator receives the class and returns the class to use. A common use is a registry: a dictionary that collects classes as they are defined.

A class decorator can also add something to the class before returning it.

You have met one: @dataclass is a class decorator. It reads the fields of the class and adds __init__, __repr__ and __eq__ for you.

Use cases

Part 4 of 4 · where decorators earn their place

A decorator fits when the same extra step is needed around many functions and has nothing to do with what each function is for. These are the jobs you will meet most often, each as a small working example.

Logging and audit

A bank must record every transfer, including the ones that fail. The decorator writes one line per call. transfer itself stays about moving money.

In real code: replace the list with the logging module or a database table. The decorator stays the same shape.

Timing

Read a clock before the call and after it. time.perf_counter() is the clock made for measuring short durations.

About the output: the real time is a tiny fraction of a second and differs on every run, so this example rounds it to whole seconds. In your own code print more digits, for example {elapsed:.4f}.

Retry on failure

Calls over a network sometimes fail and then work a moment later. A retry decorator calls the function again, up to a limit. The number of tries is a setting, so this is a three-level decorator.

In real code: wait between tries with time.sleep(), and retry only errors that can go away by themselves. Retrying a ValueError would fail the same way every time.

Caching

If a function always gives the same answer for the same arguments, the answer can be stored and reused. This is called memoizing. The store is a dictionary kept in the closure, with the arguments as the key.

functools.lru_cache

You rarely need to write this yourself. functools.lru_cache does the same job, handles keyword arguments, and can limit how many answers it keeps. LRU means "least recently used": when the cache is full, the answer that has gone unused the longest is dropped.

Limits: the arguments must be hashable, so a list or dictionary argument raises TypeError. And do not cache a function whose answer changes over time, such as a live balance.

Permission and input checks

A check decorator looks at the arguments before the function runs and raises an error if something is wrong. The function body can then assume its input is fine.

Permission

Only the owner may take money out of an account.

Input validation

The limits are settings, so this one takes parameters.

Registering handlers

Some decorators do not wrap anything. They note the function down in a table and return it unchanged. The program can then find the right function by name, and adding a new command means writing one new function, with no if chain to edit. Plugin systems and web frameworks use this shape for their routes.

Limiting calls

A decorator can count calls in its closure and refuse once a limit is reached. Here a cash machine allows three withdrawals.

Rate limiting: a real rate limiter, such as "5 calls per minute", works the same way. It stores the time of each call instead of a plain count and forgets the ones older than a minute.

Built-in decorators

You have already used decorators that come with Python. Now you can read them for what they are: functions or classes that take your function or class and return something in its place.

DecoratorGoes onWhat it does
@propertymethodLets you read the method like an attribute, without brackets
@staticmethodmethodA plain function kept inside the class. It gets no self
@classmethodmethodReceives the class as cls instead of an object. Often used for extra ways to build an object
@dataclassclassWrites __init__, __repr__ and __eq__ from the listed fields
@functools.cached_propertymethodLike @property, but worked out once per object and then stored
@functools.lru_cachefunctionRemembers results by argument
@functools.wrapswrapperCopies name and docstring onto a wrapper

Quick reference

Find what you need in the left column. The right column is the shape to write.

You needShape of the decoratorChapter
Do something before or after every callTwo levels: decorator(func) returns wrapper(*args, **kwargs)Arguments and return value
Keep the name and docstring@functools.wraps(func) on the wrapperfunctools.wraps
Settings such as a role or a countThree levels: factory(settings) returns decorator(func) returns wrapperA decorator that takes arguments
Both @name and @name(...)def name(func=None, *, option=...)With or without brackets
State you want to read from outsideA class with __init__(func) and __call__Class-based decorators
State only the wrapper needsA variable in the closure, changed with nonlocalLimiting calls
Check arguments or permissionsWrapper raises before calling funcPermission and input checks
Try again after an errorWrapper calls func in a loop with tryRetry on failure
Remember results@functools.lru_cache, or a dictionary in the closureCaching
Collect functions or classes in a tableNo wrapper: store it and return it unchangedRegistering handlers
Use it on a methodSame as a function. self is the first argumentDecorating methods
Change or record a classdecorator(cls) returns clsDecorating classes

Common mistakes

Forgetting to return the wrapper

A function without return gives back None. So the decorated name becomes None, and calling it fails.

Forgetting to return the result

The wrapper calls the function but does not pass the answer on. Nothing fails. The caller just gets None, which is harder to notice.

Forgetting functools.wraps

Every decorated function then reports the name wrapper and loses its docstring. See functools.wraps.

Leaving the brackets off a decorator with parameters

A three-level decorator must be called on the @ line. Without brackets, the function is passed in as the setting, and the name ends up pointing to the middle level. There is no error at the definition. The first call quietly returns a wrapper instead of doing the work.

The opposite slip is adding brackets to a two-level decorator, as in @announce(). That one fails at once with a TypeError, because announce is called without the function it needs.

Check: if a decorated function returns a function when you expected a value, look at the @ line first.

Next, see how a for loop really works and how to produce values one at a time in Python Iterators and Generators.