Learnings

Python advanced, part 4 of 5

Python Async IO

A lot of programs spend most of their time waiting: for a web server, a database, a file on another machine. asyncio is the standard library module that lets one thread wait for many such things at the same time. This page builds it up step by step, using a small bank and slow "network calls" that are faked with asyncio.sleep, so every example runs without a network.

The idea

Part 1 of 4 · what async code is and how to start it

Before any of the tools make sense, you need three things: the problem async code solves, the picture of one loop serving many waiting jobs, and the two keywords async and await.

The problem: waiting in a queue

Rishikesh has an account in three banks and wants all three balances. Each bank takes a moment to answer. Ordinary code asks the first bank, waits, asks the second, waits, and so on, so the waits add up.

The program did no real work for those 0.6 seconds. It only waited. Work that is mostly waiting for something outside the program is called I/O-bound (I/O means input and output). The threads and processes page solved this with one thread per call. Async IO solves it with a single thread.

One thread, one event loop

Think of one waiter serving many tables. The waiter takes an order, hands it to the kitchen, and does not stand there until the food is ready. He moves on to the next table and comes back when the kitchen rings the bell.

RestaurantasyncioIn plain words
The waiterThe event loopA loop that picks the next job that is ready and runs it for a while
A table being servedA taskOne job the loop is looking after
"Your food is cooking, I will be back"awaitThe point where a job says "I am waiting, run someone else"
The kitchen bellThe awaited thing finishesThe loop puts the job back in line and continues it after the await

There is only one waiter, so only one job runs at any instant. Jobs take turns, and a turn ends only at an await. This is called cooperative multitasking: each job has to hand control back by itself.

Concept: Concurrent means several jobs are in progress over the same period. Parallel means several jobs run at the very same instant on different CPU cores. asyncio is concurrent, not parallel.

async def makes a coroutine

A function defined with async def is a coroutine function. Calling it does not run its body. It returns a coroutine object: a paused job that will run only when something awaits it or gives it to the event loop. This is the same idea as a generator, which also does nothing until you ask for a value.

Trap: If you write fetch_balance("City Bank") on a line by itself and forget await, nothing happens and nothing fails. Python only prints a warning later: RuntimeWarning: coroutine 'fetch_balance' was never awaited. When you see that message, look for a missing await.

asyncio.run and await

asyncio.run(main()) is the entry point. It starts an event loop, runs the coroutine main() until it finishes, closes the loop and returns the result. Inside a coroutine, await something means "pause me here until something is done, then give me its result".

Rules: await is allowed only inside an async def function. Call asyncio.run() once, at the top of the program, from ordinary code. Calling it from inside a running coroutine raises RuntimeError. Things you can put after await are called awaitables: coroutines, tasks and futures.

await alone is still one after another

Rewriting the first example with async and await does not make it faster. Each await finishes before the next line starts, so the three waits still add up. The gain is only that the loop could run other jobs during each wait, and here there are none.

Running things together

Part 2 of 4 · starting several jobs and collecting what they return

The speed comes from starting several coroutines before waiting for any of them. These chapters show the tools for that, and what to do when a job fails, takes too long or is no longer needed.

Running together with gather

asyncio.gather(a, b, c) starts all the coroutines you pass in and waits until every one has finished. While one is paused at its await, the loop runs the others, so the total time is about the longest single wait, not the sum.

Concept: The results come back as a list in the same order as the arguments, not in the order the jobs finished. Village Bank answered first, but its balance is still last in the list.

Tasks: start now, collect later

A Task is a coroutine that has been handed to the event loop to run. asyncio.create_task(coro) makes one and returns at once. The task starts running the next time your code reaches an await, and you can await the task later to get its result.

gather wraps each coroutine in a task for you. Use create_task yourself when a job should run in the background while your coroutine carries on with something else.

Caution: Keep the task in a variable or a list until it is done. The loop itself holds only a weak reference to it, which means a task that nothing else points to can be removed by Python's memory cleanup before it finishes.

Errors inside gather

By default, the first exception raised in any job comes straight out of await asyncio.gather(...) and you lose the other results. With return_exceptions=True, gather never raises: a failed job puts its exception object in the result list, in its normal position.

Concept: In the default mode the other jobs are not stopped when one fails. They keep running in the background, and their results are simply thrown away.

Timeouts with wait_for

A network call can hang for a long time. asyncio.wait_for(job, timeout=seconds) waits for the job, but if the time runs out it cancels the job and raises asyncio.TimeoutError.

Version: From Python 3.11, asyncio.TimeoutError is the same class as the built-in TimeoutError. Writing except asyncio.TimeoutError works on every version.

Results as they arrive

gather makes you wait for the slowest job before you see anything. asyncio.as_completed(jobs) hands the jobs back one at a time in the order they finish, so you can show each answer as soon as it is ready.

Tip: The order is no longer the order you passed in, so each job must say who it is. That is why fetch_balance returns the bank name together with the balance here.

Cancelling a task

task.cancel() asks a task to stop. At the task's next await, Python raises asyncio.CancelledError inside it. The task may catch that to clean up, but it should then re-raise it with a bare raise so the cancel really happens.

Coordination

Part 3 of 4 · many tasks sharing limits, data and work

Once many tasks run together they start to share things: a server that allows only a few connections, a balance that two tasks both change, a list of jobs to hand out. asyncio has its own versions of the tools you met on the threads page, plus async forms of with and for.

Limiting how many run at once

Starting a thousand requests at once can get you blocked by the server. A semaphore is a counter of free places. async with limit: takes a place, or waits if none is free, and gives the place back at the end of the block.

Five fetches of 0.1 seconds, two at a time, need three rounds: about 0.3 seconds. Without the semaphore they would all run together and finish in about 0.1.

Tip: Create semaphores, locks and queues inside a coroutine, as main() does here, not at the top of the file. On Python 3.9 and older, one created before the loop starts gets tied to a different loop and fails when used.

Races between awaits, and Lock

With one thread, two tasks can never run at the same instant. A race (two jobs spoiling each other's work on shared data) can still happen, because another task may run whenever yours pauses at an await. Below, both withdrawals check the balance, both pause, and both then take the money.

An asyncio.Lock lets only one task at a time into the block under async with lock:. The second task now waits at the door until the first has finished its check and its update.

Concept: Code between two awaits runs without interruption. So a check and an update with no await between them are safe without a lock. That is a real advantage over threads, where a switch can happen on any line.

Producer and consumer with Queue

An asyncio.Queue is a waiting line shared by tasks. A producer puts work in, and consumers take work out. await queue.get() pauses while the queue is empty, and await queue.join() pauses until every item taken has been marked with task_done().

Tip: asyncio.Queue(maxsize=10) makes put pause when the queue is full. That stops a fast producer from piling up more work than the consumers can handle.

async with

A normal with block runs set-up code on the way in and clean-up code on the way out. async with does the same when the set-up or clean-up itself has to wait, such as opening and closing a network connection. The object provides __aenter__ and __aexit__, both written with async def.

Tip: You already used async with on Semaphore and Lock. For your own code, contextlib.asynccontextmanager builds one from a single async function with a yield in the middle.

async for and async generators

An async generator is an async def function that contains yield. It can await between the values it hands out. You read it with async for, which pauses while the next value is on its way. Comprehensions accept async for too.

Caution: [await f(x) for x in items] awaits one item after another. It is short to write but it is not concurrent. Use asyncio.gather when the calls should run together.

Pitfalls and choosing

Part 4 of 4 · what goes wrong, and when to use something else

asyncio depends on every task giving control back quickly. These chapters show what happens when one does not, how to hand such work to threads or processes, and how to choose between the three ways of doing several things at once.

The biggest mistake: blocking the loop

The loop can switch tasks only at an await. A call that waits without await, such as time.sleep(), a normal file or network call, or a long calculation, holds the only thread. Every other task freezes until it returns. Below, a ticker should print every 0.1 seconds.

In the first run the ticker is stuck after tick 0 until the report is finished. In a real server that means every user waits because of one slow request.

Rule: Inside async def, anything slow must be awaited. If a library has no async version, or the work is heavy calculation, move it off the loop as the next chapter shows.

The fix: to_thread and run_in_executor

asyncio.to_thread(func, *args) (Python 3.9+) runs an ordinary blocking function in a separate thread and gives you something to await. The loop stays free while the thread waits. The older spelling is loop.run_in_executor(None, func, *args), where None means "use the default thread pool".

The ticker kept ticking while the report was being built, because the blocking time.sleep ran in another thread.

Mixing with threads and processes

Threads help with blocking I/O, but because of the GIL (the lock that lets only one thread run Python code at a time) they do not speed up heavy calculation. For that, give run_in_executor a ProcessPoolExecutor. The work then runs in other processes on other CPU cores, and the loop awaits the answers.

Going the other way, a normal thread that needs to start a coroutine on a running loop uses asyncio.run_coroutine_threadsafe(coro, loop). Most asyncio objects are not safe to touch from other threads, so do not call their methods directly from one.

asyncio, threading or multiprocessing

All three let a program do several things over the same period. They differ in who decides when to switch and in what kind of work they speed up.

asynciothreadingmultiprocessing
Runs onOne thread, one event loopMany threads, one processMany processes
Who switches jobsYour code, at each awaitThe operating system, at any momentThe operating system, at any moment
Truly parallelNoNo for Python code (GIL)Yes, one core per process
Good forVery many network waits at onceSome blocking I/O calls, old librariesHeavy calculation
Cost per jobTiny: thousands of tasks are fineMedium: hundreds of threadsHigh: a handful of processes
Shared dataShared; races only across an awaitShared; races on any lineNot shared; data is copied between processes
Main catchOne blocking call freezes everything; needs async librariesLocks and hard-to-find racesSlow start, arguments must be picklable

A simple way to choose: lots of network waiting and async libraries available, use asyncio. A few blocking calls in an ordinary program, use threads. Work that keeps the CPU busy, use processes. They also combine, as the last two chapters showed.

Newer tools: TaskGroup and timeout

Python 3.11 added two tidier tools. asyncio.TaskGroup is an async with block that waits for all tasks started inside it and, if one fails, cancels the rest. asyncio.timeout(seconds) puts a time limit on a whole block. This snippet is not run on this page because it needs Python 3.11 or newer.

Tip: On 3.11 and later, prefer TaskGroup over gather for new code. With gather, the other jobs keep running after one fails. A task group stops them for you.

Libraries you will meet

The standard library gives you the loop, but most ordinary I/O libraries block. Real async programs use libraries written for asyncio. These are third-party packages, so they are listed here by name only.

LibraryWhat it is for
aiohttpHTTP client and web server built on asyncio
httpxHTTP client with both a normal and an async interface
asyncpgFast async driver for PostgreSQL databases

Caution: A blocking library such as requests called inside a coroutine blocks the loop exactly like time.sleep. Use an async library, or wrap the call in asyncio.to_thread.

Common mistakes

MistakeWhat you seeFix
Calling a coroutine function without awaitNothing runs; RuntimeWarning: coroutine ... was never awaitedAdd await, or pass it to gather or create_task
Using time.sleep or other blocking calls in a coroutineAll tasks freeze togetherawait asyncio.sleep(), an async library, or asyncio.to_thread
Awaiting jobs one by one and expecting speedTotal time is the sum of the waitsasyncio.gather or tasks
Using await outside async defSyntaxErrorPut the code in a coroutine and start it with asyncio.run
Calling asyncio.run inside a running loopRuntimeErrorJust await the coroutine
Not keeping a reference to a taskThe task may vanish before it finishesStore it in a variable, list or set
Catching CancelledError and not re-raisingThe task cannot be cancelledClean up, then raise
Check, await, then update shared dataWrong totals now and thenasyncio.Lock around the whole step
Starting thousands of requests at onceServer errors or a banasyncio.Semaphore
Using threading.Lock or queue.Queue in coroutinesThey block the loopThe asyncio versions

Quick reference

Write thisTo do this
async def f(): ...Define a coroutine function
asyncio.run(main())Start the event loop and run main to the end
x = await jobPause until job is done and take its result
await asyncio.sleep(s)Pause for s seconds without blocking the loop
await asyncio.gather(a, b)Run together, get a list of results in argument order
gather(..., return_exceptions=True)Get exceptions as results instead of raising the first one
t = asyncio.create_task(coro)Start a job in the background
t.done(), t.result(), t.cancel()Ask if a task finished, read its result, ask it to stop
await asyncio.wait_for(job, timeout=s)Give up after s seconds with asyncio.TimeoutError
for d in asyncio.as_completed(jobs): await dTake results in the order they finish
asyncio.Semaphore(n)Let at most n tasks into a block
asyncio.Lock()Let one task at a time into a block
asyncio.Queue()Pass work from producers to consumers
async with x:Enter and leave a block with awaited set-up and clean-up
async for v in agen():Loop over values that arrive over time
await asyncio.to_thread(func, arg)Run blocking code in a thread (3.9+)
await loop.run_in_executor(pool, func, arg)Run code in a thread pool or process pool
asyncio.TaskGroup(), asyncio.timeout(s)Safer grouping and timeouts (3.11+)

Next comes the last page of this path, a tour of the remaining advanced features: Python Power Features.