11.8 Accelerators: GPU, NPU, FPGA

What this is and why it exists

A hardware specification sheet is a marketing document with numbers in it, and reading one well is a skill that saves real money. This topic is what the numbers mean for your model — how memory is arranged and why it decides performance more often than arithmetic does, what dedicated inference processors constrain, when reconfigurable hardware is worth its development cost, and the one simple model that tells you which optimisation will help before you attempt any.

The vocabulary

• Memory hierarchy — registers, caches and main memory, in decreasing speed and increasing size.
• Bandwidth — how many bytes per second can move between memory and the processor.
• Occupancy — how much of the accelerator's parallel capacity is in use.
• Arithmetic intensity — operations performed per byte moved.
• Compute-bound — limited by arithmetic; memory-bound — limited by data movement.
• Roofline — a plot showing which of the two limits applies.
• NPU — a processor built specifically for neural network inference.
• Fallback — an unsupported operation running somewhere slower.

The mental model

Start with the fact that reframes every specification sheet: moving data is expensive and arithmetic is cheap. An accelerator can perform an enormous number of operations per second, and it can only move a comparatively modest number of bytes per second from its main memory. The ratio between those two figures — both printed on the sheet — is the number that decides what your model will actually achieve, and most disappointing performance comes from memory movement rather than from arithmetic.

The hierarchy is the reason. Registers are immediate; the small on-chip memories are fast; the main memory on the card is far slower; and the host's memory across the connecting bus is slower again. An operation that reads its inputs from the fast levels runs at full speed, and one that streams from main memory runs at the speed of the memory.

Occupancy is the other half of the picture. An accelerator is many parallel units, and keeping them busy requires enough independent work. A small batch, a narrow layer or a kernel needing too much of the fast memory per unit leaves most of the chip idle. This is why a model can use a small percentage of a card's theoretical capability and be perfectly correct — and why the first thing to measure is utilisation, as the scaling topic said.

Roofline thinking makes this predictive rather than retrospective, and it is the most useful idea in the topic. For any operation, compute its arithmetic intensity: the number of operations divided by the number of bytes it must move. Compare that against the machine's ratio of peak operations to peak bandwidth. Below that ratio, the operation is memory-bound — it will finish moving data before it finishes computing, and no amount of extra arithmetic capability helps. Above it, the operation is compute-bound and faster arithmetic does help.

That single comparison tells you which optimisation will work. A memory-bound operation is improved by moving less data: fusing operations so intermediates are never written out, using lower precision so every value is fewer bytes, improving layout so reads are contiguous, or restructuring so data is reused while it is in fast memory. Making the arithmetic faster does nothing for it at all. A compute-bound operation is the opposite case, and it is the rarer one.

Applied to real models the answer is usually the same and worth memorising. Large matrix multiplications with big batches are compute-bound. Almost everything else is memory-bound — activations, normalisations, elementwise operations, depthwise convolutions, attention at modest sequence lengths, and generation one token at a time, which reads the entire model to produce a single token and is memory-bound by a wide margin. That last one is why the serving topic's whole discussion was about memory rather than arithmetic.

Which gives the rule for reading a specification sheet: divide the peak operations figure by the memory bandwidth figure, and you have the machine's balance point. A part with an enormous peak and modest bandwidth is built for large dense arithmetic and will disappoint on everything else.

And the trap the topic exists to name: comparing chips by peak operations alone. Peak figures assume perfect utilisation, the smallest precision the part supports, a specific operation, and frequently a sparsity assumption. Unreachable peaks are marketing. Two parts with the same headline figure and different memory bandwidth will perform very differently on the same model, and the sheet tells you which if you look at both numbers.

Dedicated inference processors — the accelerators in phones and embedded devices — are efficient and constrained, and the constraint is what to check. They implement a fixed set of operations, usually in a fixed precision, usually integer. An operation they do not support does not fail: it falls back to the general processor, and the fallback is much slower — often slower than not using the accelerator at all, because data now moves back and forth between them.

So the practical rule is to check the supported operation list before designing, exactly as the microcontroller topic said, and to verify after deployment that the model actually ran where you intended. The tooling reports which operations were accelerated and which fell back, and a single unsupported activation in the middle of a network can cost most of the benefit. It is also why the deployment target should be chosen before the architecture is finished rather than after.

Reconfigurable hardware is the specialist option, and the judgement is about when it is worth the development cost. It is hardware you configure into the shape of your computation, so the arithmetic is exactly what you need at exactly the precision you need, with the data flowing between stages without touching main memory. That gives very low and, importantly, very predictable latency, and good energy efficiency.

The cost is development: a different skill set, longer iteration, and a design that is expensive to change. So it pays for high-volume, fixed workloads with tight latency requirements — a fixed model, deployed at scale, where latency must be bounded rather than merely low, and where the model will not change every quarter. For a model still under development, it is the wrong choice, and knowing that is the judgement being taught.

Two closing habits. Measure your model's arithmetic intensity per layer rather than reasoning about the model as a whole, because the balance differs layer by layer and the bottleneck is usually one or two of them. And choose hardware from your workload's position on the roofline, not from a headline: if your work is memory-bound, buy bandwidth; if it is compute-bound, buy arithmetic; and if you do not know which, that is the measurement to make before the purchase rather than after.

What you should now be able to explain or do

Explain why data movement rather than arithmetic usually limits performance, and describe the memory hierarchy. Say what occupancy is and why utilisation is the first thing to measure. Compute arithmetic intensity and use the roofline comparison to decide whether an operation is memory- or compute-bound. Name the optimisations that help each case. Classify common model operations, including token-by-token generation. Read a specification sheet by dividing peak operations by bandwidth, and say why peak figures alone mislead. Check operator support for dedicated inference processors and verify what actually accelerated. State when reconfigurable hardware is worth its cost.

Check yourself