Python advanced, part 4 of 5
Python Async IO
A lot of programs spend most of their time waiting: for a web server, a database, a file on another machine. asyncio is the standard library module that lets one thread wait for many such things at the same time. This page builds it up step by step, using a small bank and slow "network calls" that are faked with asyncio.sleep, so every example runs without a network.
The idea
Part 1 of 4 · what async code is and how to start it
Before any of the tools make sense, you need three things: the problem async code solves, the picture of one loop serving many waiting jobs, and the two keywords async and await.
The problem: waiting in a queue
Rishikesh has an account in three banks and wants all three balances. Each bank takes a moment to answer. Ordinary code asks the first bank, waits, asks the second, waits, and so on, so the waits add up.
The program did no real work for those 0.6 seconds. It only waited. Work that is mostly waiting for something outside the program is called I/O-bound (I/O means input and output). The threads and processes page solved this with one thread per call. Async IO solves it with a single thread.
One thread, one event loop
Think of one waiter serving many tables. The waiter takes an order, hands it to the kitchen, and does not stand there until the food is ready. He moves on to the next table and comes back when the kitchen rings the bell.
| Restaurant | asyncio | In plain words |
|---|---|---|
| The waiter | The event loop | A loop that picks the next job that is ready and runs it for a while |
| A table being served | A task | One job the loop is looking after |
| "Your food is cooking, I will be back" | await | The point where a job says "I am waiting, run someone else" |
| The kitchen bell | The awaited thing finishes | The loop puts the job back in line and continues it after the await |
There is only one waiter, so only one job runs at any instant. Jobs take turns, and a turn ends only at an await. This is called cooperative multitasking: each job has to hand control back by itself.
Concept: Concurrent means several jobs are in progress over the same period. Parallel means several jobs run at the very same instant on different CPU cores. asyncio is concurrent, not parallel.
async def makes a coroutine
A function defined with async def is a coroutine function. Calling it does not run its body. It returns a coroutine object: a paused job that will run only when something awaits it or gives it to the event loop. This is the same idea as a generator, which also does nothing until you ask for a value.
Trap: If you write fetch_balance("City Bank") on a line by itself and forget await, nothing happens and nothing fails. Python only prints a warning later: RuntimeWarning: coroutine 'fetch_balance' was never awaited. When you see that message, look for a missing await.
asyncio.run and await
asyncio.run(main()) is the entry point. It starts an event loop, runs the coroutine main() until it finishes, closes the loop and returns the result. Inside a coroutine, await something means "pause me here until something is done, then give me its result".
Rules: await is allowed only inside an async def function. Call asyncio.run() once, at the top of the program, from ordinary code. Calling it from inside a running coroutine raises RuntimeError. Things you can put after await are called awaitables: coroutines, tasks and futures.
await alone is still one after another
Rewriting the first example with async and await does not make it faster. Each await finishes before the next line starts, so the three waits still add up. The gain is only that the loop could run other jobs during each wait, and here there are none.
Running things together
Part 2 of 4 · starting several jobs and collecting what they return
The speed comes from starting several coroutines before waiting for any of them. These chapters show the tools for that, and what to do when a job fails, takes too long or is no longer needed.
- asyncio.gather: run several and get all results
- asyncio.create_task: start a job in the background
- Errors inside gather: one failure should not hide the rest
- Timeouts: give up on a slow job
- asyncio.as_completed: handle results as they arrive
- Cancelling a task: stop a job from outside
Running together with gather
asyncio.gather(a, b, c) starts all the coroutines you pass in and waits until every one has finished. While one is paused at its await, the loop runs the others, so the total time is about the longest single wait, not the sum.
Concept: The results come back as a list in the same order as the arguments, not in the order the jobs finished. Village Bank answered first, but its balance is still last in the list.
Tasks: start now, collect later
A Task is a coroutine that has been handed to the event loop to run. asyncio.create_task(coro) makes one and returns at once. The task starts running the next time your code reaches an await, and you can await the task later to get its result.
gather wraps each coroutine in a task for you. Use create_task yourself when a job should run in the background while your coroutine carries on with something else.
Caution: Keep the task in a variable or a list until it is done. The loop itself holds only a weak reference to it, which means a task that nothing else points to can be removed by Python's memory cleanup before it finishes.
Errors inside gather
By default, the first exception raised in any job comes straight out of await asyncio.gather(...) and you lose the other results. With return_exceptions=True, gather never raises: a failed job puts its exception object in the result list, in its normal position.
Concept: In the default mode the other jobs are not stopped when one fails. They keep running in the background, and their results are simply thrown away.
Timeouts with wait_for
A network call can hang for a long time. asyncio.wait_for(job, timeout=seconds) waits for the job, but if the time runs out it cancels the job and raises asyncio.TimeoutError.
Version: From Python 3.11, asyncio.TimeoutError is the same class as the built-in TimeoutError. Writing except asyncio.TimeoutError works on every version.
Results as they arrive
gather makes you wait for the slowest job before you see anything. asyncio.as_completed(jobs) hands the jobs back one at a time in the order they finish, so you can show each answer as soon as it is ready.
Tip: The order is no longer the order you passed in, so each job must say who it is. That is why fetch_balance returns the bank name together with the balance here.
Cancelling a task
task.cancel() asks a task to stop. At the task's next await, Python raises asyncio.CancelledError inside it. The task may catch that to clean up, but it should then re-raise it with a bare raise so the cancel really happens.
Coordination
Part 3 of 4 · many tasks sharing limits, data and work
Once many tasks run together they start to share things: a server that allows only a few connections, a balance that two tasks both change, a list of jobs to hand out. asyncio has its own versions of the tools you met on the threads page, plus async forms of with and for.
- asyncio.Semaphore: only N at a time
- asyncio.Lock: races still exist between awaits
- asyncio.Queue: producers and consumers
- async with: set up and clean up with awaits
- async for: async generators and comprehensions
Limiting how many run at once
Starting a thousand requests at once can get you blocked by the server. A semaphore is a counter of free places. async with limit: takes a place, or waits if none is free, and gives the place back at the end of the block.
Five fetches of 0.1 seconds, two at a time, need three rounds: about 0.3 seconds. Without the semaphore they would all run together and finish in about 0.1.
Tip: Create semaphores, locks and queues inside a coroutine, as main() does here, not at the top of the file. On Python 3.9 and older, one created before the loop starts gets tied to a different loop and fails when used.
Races between awaits, and Lock
With one thread, two tasks can never run at the same instant. A race (two jobs spoiling each other's work on shared data) can still happen, because another task may run whenever yours pauses at an await. Below, both withdrawals check the balance, both pause, and both then take the money.
An asyncio.Lock lets only one task at a time into the block under async with lock:. The second task now waits at the door until the first has finished its check and its update.
Concept: Code between two awaits runs without interruption. So a check and an update with no await between them are safe without a lock. That is a real advantage over threads, where a switch can happen on any line.
Producer and consumer with Queue
An asyncio.Queue is a waiting line shared by tasks. A producer puts work in, and consumers take work out. await queue.get() pauses while the queue is empty, and await queue.join() pauses until every item taken has been marked with task_done().
Tip: asyncio.Queue(maxsize=10) makes put pause when the queue is full. That stops a fast producer from piling up more work than the consumers can handle.
async with
A normal with block runs set-up code on the way in and clean-up code on the way out. async with does the same when the set-up or clean-up itself has to wait, such as opening and closing a network connection. The object provides __aenter__ and __aexit__, both written with async def.
Tip: You already used async with on Semaphore and Lock. For your own code, contextlib.asynccontextmanager builds one from a single async function with a yield in the middle.
async for and async generators
An async generator is an async def function that contains yield. It can await between the values it hands out. You read it with async for, which pauses while the next value is on its way. Comprehensions accept async for too.
Caution: [await f(x) for x in items] awaits one item after another. It is short to write but it is not concurrent. Use asyncio.gather when the calls should run together.
Pitfalls and choosing
Part 4 of 4 · what goes wrong, and when to use something else
asyncio depends on every task giving control back quickly. These chapters show what happens when one does not, how to hand such work to threads or processes, and how to choose between the three ways of doing several things at once.
The biggest mistake: blocking the loop
The loop can switch tasks only at an await. A call that waits without await, such as time.sleep(), a normal file or network call, or a long calculation, holds the only thread. Every other task freezes until it returns. Below, a ticker should print every 0.1 seconds.
In the first run the ticker is stuck after tick 0 until the report is finished. In a real server that means every user waits because of one slow request.
Rule: Inside async def, anything slow must be awaited. If a library has no async version, or the work is heavy calculation, move it off the loop as the next chapter shows.
The fix: to_thread and run_in_executor
asyncio.to_thread(func, *args) (Python 3.9+) runs an ordinary blocking function in a separate thread and gives you something to await. The loop stays free while the thread waits. The older spelling is loop.run_in_executor(None, func, *args), where None means "use the default thread pool".
The ticker kept ticking while the report was being built, because the blocking time.sleep ran in another thread.
Mixing with threads and processes
Threads help with blocking I/O, but because of the GIL (the lock that lets only one thread run Python code at a time) they do not speed up heavy calculation. For that, give run_in_executor a ProcessPoolExecutor. The work then runs in other processes on other CPU cores, and the loop awaits the answers.
Going the other way, a normal thread that needs to start a coroutine on a running loop uses asyncio.run_coroutine_threadsafe(coro, loop). Most asyncio objects are not safe to touch from other threads, so do not call their methods directly from one.
asyncio, threading or multiprocessing
All three let a program do several things over the same period. They differ in who decides when to switch and in what kind of work they speed up.
| asyncio | threading | multiprocessing | |
|---|---|---|---|
| Runs on | One thread, one event loop | Many threads, one process | Many processes |
| Who switches jobs | Your code, at each await | The operating system, at any moment | The operating system, at any moment |
| Truly parallel | No | No for Python code (GIL) | Yes, one core per process |
| Good for | Very many network waits at once | Some blocking I/O calls, old libraries | Heavy calculation |
| Cost per job | Tiny: thousands of tasks are fine | Medium: hundreds of threads | High: a handful of processes |
| Shared data | Shared; races only across an await | Shared; races on any line | Not shared; data is copied between processes |
| Main catch | One blocking call freezes everything; needs async libraries | Locks and hard-to-find races | Slow start, arguments must be picklable |
A simple way to choose: lots of network waiting and async libraries available, use asyncio. A few blocking calls in an ordinary program, use threads. Work that keeps the CPU busy, use processes. They also combine, as the last two chapters showed.
Newer tools: TaskGroup and timeout
Python 3.11 added two tidier tools. asyncio.TaskGroup is an async with block that waits for all tasks started inside it and, if one fails, cancels the rest. asyncio.timeout(seconds) puts a time limit on a whole block. This snippet is not run on this page because it needs Python 3.11 or newer.
Tip: On 3.11 and later, prefer TaskGroup over gather for new code. With gather, the other jobs keep running after one fails. A task group stops them for you.
Libraries you will meet
The standard library gives you the loop, but most ordinary I/O libraries block. Real async programs use libraries written for asyncio. These are third-party packages, so they are listed here by name only.
| Library | What it is for |
|---|---|
aiohttp | HTTP client and web server built on asyncio |
httpx | HTTP client with both a normal and an async interface |
asyncpg | Fast async driver for PostgreSQL databases |
Caution: A blocking library such as requests called inside a coroutine blocks the loop exactly like time.sleep. Use an async library, or wrap the call in asyncio.to_thread.
Common mistakes
| Mistake | What you see | Fix |
|---|---|---|
Calling a coroutine function without await | Nothing runs; RuntimeWarning: coroutine ... was never awaited | Add await, or pass it to gather or create_task |
Using time.sleep or other blocking calls in a coroutine | All tasks freeze together | await asyncio.sleep(), an async library, or asyncio.to_thread |
| Awaiting jobs one by one and expecting speed | Total time is the sum of the waits | asyncio.gather or tasks |
Using await outside async def | SyntaxError | Put the code in a coroutine and start it with asyncio.run |
Calling asyncio.run inside a running loop | RuntimeError | Just await the coroutine |
| Not keeping a reference to a task | The task may vanish before it finishes | Store it in a variable, list or set |
Catching CancelledError and not re-raising | The task cannot be cancelled | Clean up, then raise |
Check, await, then update shared data | Wrong totals now and then | asyncio.Lock around the whole step |
| Starting thousands of requests at once | Server errors or a ban | asyncio.Semaphore |
Using threading.Lock or queue.Queue in coroutines | They block the loop | The asyncio versions |
Quick reference
| Write this | To do this |
|---|---|
async def f(): ... | Define a coroutine function |
asyncio.run(main()) | Start the event loop and run main to the end |
x = await job | Pause until job is done and take its result |
await asyncio.sleep(s) | Pause for s seconds without blocking the loop |
await asyncio.gather(a, b) | Run together, get a list of results in argument order |
gather(..., return_exceptions=True) | Get exceptions as results instead of raising the first one |
t = asyncio.create_task(coro) | Start a job in the background |
t.done(), t.result(), t.cancel() | Ask if a task finished, read its result, ask it to stop |
await asyncio.wait_for(job, timeout=s) | Give up after s seconds with asyncio.TimeoutError |
for d in asyncio.as_completed(jobs): await d | Take results in the order they finish |
asyncio.Semaphore(n) | Let at most n tasks into a block |
asyncio.Lock() | Let one task at a time into a block |
asyncio.Queue() | Pass work from producers to consumers |
async with x: | Enter and leave a block with awaited set-up and clean-up |
async for v in agen(): | Loop over values that arrive over time |
await asyncio.to_thread(func, arg) | Run blocking code in a thread (3.9+) |
await loop.run_in_executor(pool, func, arg) | Run code in a thread pool or process pool |
asyncio.TaskGroup(), asyncio.timeout(s) | Safer grouping and timeouts (3.11+) |
Next comes the last page of this path, a tour of the remaining advanced features: Python Power Features.