2.16 Concurrency: threads, processes, asyncio

What this is and why it exists

Concurrency makes waiting cheap. A script that fetches two hundred pages one after another spends nearly all its life doing nothing at all, and rearranging it so the waiting overlaps can turn ten minutes into twenty seconds. This topic sorts out which of Python's three tools fits which job, and it begins with the one fact that decides the answer — because reaching for the wrong tool produces code that is more complicated and exactly as slow.

The vocabulary

• I/O-bound — the program spends its time waiting for something else: a network, a disk, a database.
• CPU-bound — the program spends its time computing.
• Thread — a separate line of execution inside one process, sharing its memory.
• Process — a separate program with its own memory.
• The interpreter lock — the mechanism the glossary describes as assuring "that only one thread executes Python bytecode at a time".
• Coroutine — a function defined with async def, which can be paused at an await.
• Event loop — the scheduler that runs coroutines and resumes them when what they were waiting for is ready.
• Race condition — a bug whose outcome depends on which of two concurrent operations happened first.

The mental model

Start with the lock, because everything follows from it. The glossary defines it as "the mechanism used by the CPython interpreter to assure that only one thread executes Python bytecode at a time" — and then gives the exception that makes threads useful anyway: "some extension modules, either standard or third-party, are designed so as to release the GIL when doing computationally intensive tasks such as compression or hashing. Also, the GIL is always released when doing I/O."

Read that last sentence twice. It means threads do not help you compute faster, and they do help you wait faster — which is exactly the shape of most real work. Downloading two hundred files, calling an interface a thousand times, reading many files: all waiting, all improved by threads. Multiplying matrices in pure Python: not improved at all, because only one thread runs bytecode at a time.

So the decision is made by the bottleneck, and it is worth answering the question out loud before choosing. Waiting on the outside world, moderate scale? Threads, through the futures pool, which is the least intrusive change you can make to existing code. Waiting on the outside world at large scale, thousands of concurrent operations? Async, which handles that number in one thread where thousands of threads would not. Computing? Processes, which have their own interpreter each and therefore their own lock, at the cost of the data having to be sent to them and back. And if the computation is numerical, the honest first answer is usually not concurrency at all: NumPy's operations already run in optimised code that releases the lock, so vectorising is often faster than any arrangement of your loops.

Async deserves its own paragraph because the syntax makes it look harder than it is. A coroutine is a function that can pause; await is where it pauses; the event loop runs other coroutines while it is paused; and the documentation names asyncio.run as the way "to run the top-level entry point" of the whole thing. Concurrency comes from gather, which the reference describes as running "awaitable objects in the aws sequence concurrently" — awaiting one coroutine at a time in a loop is not concurrent at all, which is the mistake almost everybody makes on their first attempt. The other constraint is that async is contagious: an ordinary blocking call inside a coroutine stops the whole loop, so the libraries you use must be async-aware for any of it to help.

Then the shared thread that runs through all three: anything shared between concurrent workers can be seen half-changed. Two threads incrementing the same counter can both read the old value; two workers appending to the same file can interleave lines. The defences are, in order of preference: do not share — give each worker its own data and combine the results at the end; and where you must, protect the shared thing with a lock and keep the protected section as small as possible.

In code

Checked against the glossary and the asyncio reference.

import asyncio
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor


def fetch_one(url):
    """Blocking, I/O-bound: nearly all of this is waiting."""
    ...


# Threads: the smallest change to existing blocking code.
with ThreadPoolExecutor(max_workers=16) as pool:
    results = list(pool.map(fetch_one, urls))


def crunch(chunk):
    """CPU-bound: threads would not help, processes do."""
    return sum(x * x for x in chunk)


with ProcessPoolExecutor() as pool:
    totals = list(pool.map(crunch, chunks))


# Async: thousands of concurrent waits in one thread.
async def fetch_all(client, urls):
    tasks = [client.get(u) for u in urls]
    return await asyncio.gather(*tasks)


async def main(urls, client):
    # `client` is whichever async HTTP library you chose; the shape is the
    # same for all of them, and their own documentation names the class.
    pages = await fetch_all(client, urls)
    return len(pages)


asyncio.run(main(urls, client))

The important line is gather. Written the other way — a loop with an await inside it — each request would finish before the next began, and the program would be exactly as slow as the sequential version with more syntax. Concurrency in async comes from starting many things and awaiting them together.

What you should now be able to explain or do

State what the interpreter lock blocks and the one thing it always releases. Decide between threads, processes and async from a description of the bottleneck, and defend the choice. Say why async is the right tool at thousands of concurrent operations and threads at dozens. Spot the loop-with-await mistake and fix it with a gather. Say why vectorising is often the right answer for numerical work instead of any concurrency at all. Name the two defences against shared-state bugs, in order.

Check yourself