Python advanced, part 5 of 5
Python Power Features
This page is a tour of the remaining features an experienced Python developer is expected to know: context managers, special methods, attribute control, class machinery, the handy parts of functools and collections, type hints and some newer syntax. Each chapter is short: the idea, one small example from a family bank, and a note where it helps.
Context managers
Part 1 of 6 · set-up and clean-up that cannot be forgotten
A context manager is an object that knows what to do when a with block starts and when it ends. It puts the clean-up next to the set-up, and the clean-up runs even if the block fails.
- What with does
- Writing one as a class: a transaction that rolls back
- The short way:
contextlib.contextmanager - Ready-made helpers:
suppress,closing,ExitStack
What with does
with open(...) as f: opens the file, runs the block, and closes the file when the block ends, whether it ends normally or with an error. It is a short form of try and finally.
Concept: Python calls __enter__() on the object when the block starts, and the value it returns is what as names. It calls __exit__() when the block ends. Any object with these two methods works in a with statement.
A context manager as a class
A bank transaction is a group of changes that must all happen or not happen at all. This one remembers the balance on the way in. If the block raises an error, it puts the old balance back, which is called a rollback.
Concept: __exit__ receives the error's class, the error object and its traceback, or three None values if the block ended normally. Returning True swallows the error. Return False (or nothing) unless you really mean to hide it.
The short way: contextmanager
contextlib.contextmanager is a decorator that turns a generator function into a context manager. The code before yield is the set-up, the yielded value goes to as, and the code after yield is the clean-up.
Tip: Always put the yield inside try when there is clean-up to do. An error in the with block is raised at the yield line, so without try the code after it would be skipped.
suppress, closing and ExitStack
contextlib also has ready-made context managers. suppress ignores the errors you name. closing calls close() on any object at the end. ExitStack holds a number of context managers that is known only while the program runs and closes all of them together.
Special methods
Part 2 of 6 · making your objects work with built-in syntax
Methods whose names start and end with two underscores, such as __init__, are special methods, often called "dunder" methods (short for double underscore). You do not call them yourself. Python calls them when your object is used with print, ==, +, len(), in and so on.
- __repr__ and __str__: how an object is shown
- __eq__ and __hash__: equality, sets and dict keys
- Ordering:
<,sortedandtotal_ordering - Container methods:
len,in,[] - Arithmetic:
+andsumfor a Money class - __bool__ and __call__: truth and objects you can call
__repr__ and __str__
Without help, printing an object shows something like <__main__.Money object at 0x10a3f>. __repr__ gives the text for developers, ideally the code that would rebuild the object. __str__ gives the friendly text for users, and is what print and f-strings use.
Tip: If you write only one, write __repr__. Python falls back to it when __str__ is missing, and it is what you see inside lists and in the debugger.
__eq__ and __hash__
By default == on your own objects is true only for the very same object. __eq__ lets you say that two objects with the same values are equal. __hash__ returns a number used by sets and dict keys to find objects quickly, and equal objects must return the same number.
Caution: A class that defines __eq__ but not __hash__ becomes unhashable: its objects can no longer go in a set or be dict keys. Build the hash from the same fields as the equality, and only from fields that do not change.
Ordering with total_ordering
To use <, sorted(), min() and max(), a class needs comparison methods. Writing all of __lt__, __le__, __gt__ and __ge__ is tedious. With functools.total_ordering you write __eq__ and one of them, and the decorator fills in the rest.
Container methods: len, in and []
Three methods make an object behave like a collection. __len__ answers len(x), __contains__ answers item in x, and __getitem__ answers x[key]. Add __iter__ and it works in a for loop too.
Arithmetic: + for Money
a + b calls a.__add__(b). If that does not know what to do, Python tries the right-hand object with b.__radd__(a). That second method is what makes sum() work, because sum starts by adding the first item to the number 0.
Concept: Returning NotImplemented tells Python "not my job, ask the other object". If nobody can do it, Python raises TypeError for you. Other operators follow the same pattern: __sub__, __mul__, __truediv__, and __iadd__ for +=.
__bool__ and __call__
__bool__ decides whether an object counts as true in if obj:. __call__ lets you use an object like a function, obj(...), which is handy for a function that needs to remember settings.
Concept: Without __bool__, Python uses __len__ (length 0 is false). Without both, every object is true.
Attribute control
Part 3 of 6 · deciding what happens when an attribute is read or set
Reading acc.balance or writing acc.balance = 5 looks like plain data access. Python gives you several hooks to run code at that moment, from the simple property to the general descriptor.
- property: a method that looks like an attribute
- Descriptors: a reusable property
- __getattr__ and __getattribute__: catching attribute reads
- __slots__: a fixed list of attributes
property in one example
@property turns a method into something you read like an attribute. Add a setter and you can check every value before it is stored, without changing how the class is used.
Descriptors: a reusable property
A property is tied to one attribute of one class. If five fields need the same check, you would write it five times. A descriptor is a small class with __get__ and __set__ methods. Put an instance of it in a class body, and Python calls those methods whenever that attribute is read or set.
__set_name__ is called when the class is created and tells each descriptor which name it was given. That is how one PositiveAmount class can report "principal" for one field and "monthly_payment" for the other.
Concept: You have used descriptors all along. property, methods, classmethod and staticmethod are all built on this same mechanism.
__getattr__ and __getattribute__
__getattr__ is called only when the normal lookup fails, so it is a safe place for fallbacks, such as passing unknown attributes on to another object. __getattribute__ is called for every attribute read, found or not. It is easy to break things with it and it is rarely needed.
Caution: Inside __getattribute__, reading self.anything calls __getattribute__ again, for ever. Use super().__getattribute__(name) there. The matching hook for writes is __setattr__.
__slots__
Every normal object carries its own dict to hold its attributes. That is flexible but costs memory. __slots__ lists the only attribute names allowed, and Python then stores them in fixed places with no per-object dict.
| What it saves | What it forbids |
|---|---|
| Memory per object, which matters when you make hundreds of thousands of them | Adding attributes that are not in the list |
| A little time on attribute access | Tools that rely on __dict__, such as vars(obj) |
Typos such as acc.balanse = 5 become errors | Easy multiple inheritance when several parents have slots |
Class machinery
Part 4 of 6 · tools that write, check and customise classes
These features work on classes themselves: generating the boring methods, fixing a set of allowed values, saying what a subclass must provide, and running code when a class is created.
- Dataclasses in depth:
field,default_factory,__post_init__ - Frozen and ordered dataclasses
- Enum: a fixed set of named values
- Abstract base classes: methods a subclass must write
- Protocol: matching by shape, not by parent
- __init_subclass__: a hook for new subclasses
- Metaclasses: the class of a class
- Class decorators: the lighter alternative
Dataclasses in depth
@dataclass reads the annotated fields of a class and writes __init__, __repr__ and __eq__ for you. field() tunes one field. default_factory gives each object its own fresh list or dict, and __post_init__ runs right after the generated __init__, which is the place for checks and computed values.
Caution: Writing history: list = [] is refused with a ValueError, because that one list would be shared by every account. Use default_factory for lists, dicts and sets.
Frozen and ordered dataclasses
frozen=True makes objects read-only after creation, which also makes them hashable, so they can be set members and dict keys. order=True adds < and friends, comparing the fields from first to last like a tuple.
Enum
An Enum is a fixed set of named values. Use it instead of loose strings such as "savings", so that a typo becomes an error and not a silent bug.
Tip: Compare members with is or ==. When the values do not matter, write SAVINGS = auto() (from enum) and Python numbers them for you.
Abstract base classes
An abstract base class (ABC) is a parent that cannot be used directly. It marks some methods with @abstractmethod, and Python refuses to create an object of any subclass that has not written them all.
Protocol: matching by shape
A Protocol describes which methods an object must have, without asking it to inherit from anything. Any class with the right methods fits. This is called structural typing, the checked form of duck typing ("if it has a balance() method, it will do").
| ABC | Protocol | |
|---|---|---|
| A class fits when | It inherits from the ABC | It has the listed methods |
| Checked | When an object is created | Mostly by a type checker, before the program runs |
| Can share code | Yes, normal methods in the parent | Not its purpose |
| Use it for | Your own family of classes | Accepting classes you do not own or cannot change |
__init_subclass__
A parent class can define __init_subclass__. Python calls it every time a new subclass is defined, and passes any keyword arguments written after the parent's name. It covers the most common reason people once needed a metaclass: keeping a register of subclasses.
Metaclasses
Classes are objects too, and every object has a class. The class of a class is called its metaclass, and for normal classes it is type. A metaclass is a subclass of type, and it can change how classes themselves are built.
You rarely need this: Metaclasses are used inside frameworks such as Enum, ABC and database model libraries. In your own code, reach for a class decorator or __init_subclass__ first. Know what a metaclass is so that you can read such code, not so that you write one.
Class decorators
A decorator can wrap a class just as it wraps a function: it receives the class, may change or record it, and returns it. @dataclass and @total_ordering are class decorators. The decorators page explains the mechanism.
Functions and data tools
Part 5 of 6 · small standard-library helpers used every day
These save a few lines each, and experienced developers reach for them without thinking.
- functools:
partial,reduce,singledispatch - collections:
Counter,defaultdict,deque,namedtuple,ChainMap - Unpacking tricks: stars, merging dicts and the walrus operator
functools: partial, reduce, singledispatch
partial makes a new function with some arguments already filled in. reduce folds a list into one value by applying a two-argument function again and again. singledispatch lets one function name have a different body for each type of its first argument.
Tip: For plain totals, sum(), max() and min() are clearer than reduce. Keep reduce for combining steps that have no built-in.
collections
The collections module holds special-purpose containers. Counter counts things, defaultdict creates missing values by itself, deque is a list that is fast at both ends, namedtuple is a tuple with field names, and ChainMap looks through several dicts in order.
Unpacking tricks
A star collects "the rest" when unpacking, and spreads a collection out when building one or calling a function. Two stars do the same for dicts. The walrus operator := assigns a value and uses it in the same expression.
Version: The walrus operator needs Python 3.8 and the | operator for dicts needs 3.9. In both merges the right-hand dict wins when a key appears twice.
Typing and newer syntax
Part 6 of 6 · saying what types you expect, and syntax from Python 3.10
Type hints are notes in the code that say what kind of value a variable, argument or result should be. The last chapters show two pieces of syntax that arrived after Python 3.9, then a table of everything on this page.
- Type hints: variables, functions,
Optional,Union - TypedDict, Callable, and what hints do not do
- match: structural pattern matching (3.10+)
- X | Y unions (3.10+)
- Quick reference
Type hints
A hint follows a colon after a name, and -> gives the type a function returns. Containers say what they hold: list[str] is a list of strings. Optional[X] means "an X or None", and Union[X, Y] means "an X or a Y".
Version: list[str] and dict[str, int] work from Python 3.9. Older code imports List and Dict from typing for the same thing.
TypedDict, Callable, and what hints do not do
TypedDict describes a dict with known keys and a type for each. Callable[[int], int] describes a function that takes one int and returns an int. Most important: Python does not check hints when the program runs. The last line below passes strings where numbers are expected and nothing stops it.
Concept: Hints are read by people, by editors, and by separate tools called type checkers, such as mypy and pyright. A type checker reads your code without running it and would report the last call above as an error.
match: structural pattern matching
The match statement compares a value against several shapes and unpacks the parts of the one that fits, in a single step. It needs Python 3.10 or newer, so this block is not run on this page.
X | Y unions
From Python 3.10, X | Y in a type hint means the same as Union[X, Y], and X | None the same as Optional[X]. It also works in isinstance. This block needs 3.10 or newer, so it is not run on this page.
Quick reference
| Feature | What it is for |
|---|---|
with, __enter__, __exit__ | Set-up and guaranteed clean-up around a block |
@contextmanager | A context manager written as one generator function |
suppress, closing, ExitStack | Ignore named errors, close an object, manage many contexts |
__repr__, __str__ | Text for developers and text for users |
__eq__, __hash__ | Equality by value; use in sets and as dict keys |
__lt__ with @total_ordering | Comparison, sorted, min, max |
__len__, __contains__, __getitem__, __iter__ | len(x), in, x[key], for |
__add__, __radd__, __mul__ | Arithmetic operators and sum() |
__bool__, __call__ | Truth value; calling an object like a function |
@property | Run code when one attribute is read or set |
Descriptor (__get__, __set__, __set_name__) | The same attribute logic reused on many fields |
__getattr__ | Fallback for attributes that are not found |
__slots__ | Fixed attributes, less memory per object |
@dataclass, field, __post_init__ | Data-holding classes without hand-written methods |
Enum | A fixed set of named values |
ABC, @abstractmethod | Force subclasses to write certain methods |
Protocol | Accept any object that has the right methods |
__init_subclass__ | Run code whenever a subclass is defined |
| Metaclass | Change how classes are built (rarely needed) |
| Class decorator | Change or record a class after it is built |
partial, reduce, singledispatch | Pre-fill arguments, fold a list, choose a body by type |
Counter, defaultdict, deque, namedtuple, ChainMap | Count, group, keep the last N, name tuple fields, layer dicts |
*rest, **dict, := | Unpack, spread and merge, assign inside an expression |
Type hints, Optional, Union, TypedDict, Callable | State expected types for readers and type checkers |
match (3.10+) | Branch on the shape of data and unpack it |
X | Y (3.10+) | Shorter way to write a union type |
That completes the advanced path. The next path is about everything a working project needs around the language, starting with Python Project Setup.