2.12 Decorators and closures

What this is and why it exists

A decorator wraps behaviour around a function without editing it: timing, caching, retries, registration. You will meet them constantly in the frameworks ahead — a web route, a test fixture, a tool an agent may call — and writing one yourself demystifies the @ permanently. It is also the clearest possible use of the closures from the functions topic, which is why it sits here.

The vocabulary

• First-class object — something that can be passed around like any value; in Python, functions are.
• Higher-order function — one that takes or returns a function.
• Closure — an inner function that remembers names from the scope it was defined in.
• Decorator — a callable that takes a function and returns a replacement for it.
• The at-sign syntax — sugar: writing the decorator above a definition is the same as reassigning the name to the decorated version.
• Wrapper — the replacement function the decorator returns.
• Introspection — code inspecting a function's name, docstring or signature at runtime.
• Memoisation — remembering results so a repeated call returns without recomputing.

The mental model

Three facts, and decorators follow from them. Functions are values, so one can be passed to another. A function can return a function. And an inner function keeps access to the names around it. Put those together and a decorator is a function taking a function and returning a new one that calls the original with something extra around it — and the at-sign is only a shorter way of saying "replace this name with the decorated version".

The wrapper is where the extra work lives: before the call, after it, around it, or instead of it. Timing measures before and after. Caching checks a store first and skips the call if the answer is there. Retrying calls in a loop. Registration does nothing to the call at all and merely records that this function exists — which is how tool registries and route tables work, and worth recognising because it looks like the decorator is doing nothing.

Then the trap, and it is invisible until something breaks a long way away. A wrapper is a different function, and it has the wrapper's name and docstring, not the original's. Every decorated function then reports the same name, help produces the wrapper's docstring, and anything that introspects — a test framework matching names, a documentation tool, a framework reading a signature — sees the wrong thing. The remedy is one line: functools.wraps, which the reference describes as updating "the wrapper function to look like the wrapped function" by copying its module, name, qualified name, annotations and docstring, and adding an attribute pointing back at the original. Apply it to every wrapper you write, always, and the problem never exists.

A decorator that takes arguments needs one more layer, and the shape is worth memorising rather than rediscovering: a function that takes the arguments and returns a decorator, which takes the function and returns a wrapper. Three levels — the outer for the settings, the middle for the function, the inner for the call. Once you have written it once it stops being confusing.

The standard library gives you the caching one already made. lru_cache is a "decorator to wrap a function with a memoizing callable that saves up to the maxsize most recent calls. It can save time when an expensive or I/O bound function is periodically called with the same arguments", and setting maxsize to None turns the eviction off entirely, so that the cache can, in the reference's words, "grow without bound" — which is what the simpler cache decorator does, described as a "simple lightweight unbounded function cache". Two cautions before you sprinkle it about: the arguments must be hashable, and the function must be pure, because caching something that reads a file or a clock means happily returning yesterday's answer forever.

In code

Checked against the functools reference.

import functools
import time


def timed(func):
    @functools.wraps(func)          # without this, every function is "wrapper"
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        try:
            return func(*args, **kwargs)
        finally:
            elapsed = time.perf_counter() - start
            print(f"{func.__name__} took {elapsed:.3f}s")
    return wrapper


def retry(attempts=3, wait=0.5):
    """A decorator with arguments: three layers, outermost holds the settings."""
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, attempts + 1):
                try:
                    return func(*args, **kwargs)
                except Exception:
                    if attempt == attempts:
                        raise
                    time.sleep(wait * attempt)
        return wrapper
    return decorator


@timed
@retry(attempts=3)
def fetch(url):
    """Fetch a record. Slow, occasionally fails."""
    ...


@functools.lru_cache(maxsize=128)
def expensive(n: int) -> int:
    return sum(i * i for i in range(n))

Two things to read. The timing wrapper puts its measurement in finally, so a failed call is still timed — the errors topic paying off here. And the retry backs off by multiplying the wait by the attempt number, which is the same idea as the backoff in any distributed system: retrying immediately makes a struggling thing worse.

Stacked decorators apply from the bottom up: fetch is wrapped by the retry first, then by the timer, so the timing covers all the attempts together. If you wanted to time each attempt separately you would swap the two lines — which is worth knowing, because the order is not decoration, it is behaviour.

What you should now be able to explain or do

State the three facts a decorator is built from. Write a decorator that times a function, including the detail that makes it time failures too. Say what functools.wraps copies and name three things that break without it. Write a decorator that takes arguments, and say what each of its three layers is for. Say when caching is safe and give two situations where it silently returns wrong answers. Read a stack of decorators and say which applies first.

Check yourself