11.7 Model optimization for hardware

What this is and why it exists

An order of magnitude off your inference cost is available without retraining anything, from pruning, a portable export and a runtime built for your hardware. What makes it engineering rather than hope is the measurement: vendor figures describe somebody else's model on somebody else's input, and operation counts describe arithmetic that is frequently not the bottleneck. Measure on your hardware with your input sizes, and you will find memory movement blamed on computation more often than not.

The vocabulary

• Pruning — removing weights or structures from a trained network.
• Unstructured pruning — removing individual weights, leaving a sparse pattern.
• Structured pruning — removing whole channels, filters or layers.
• Sparsity — the fraction of weights that are zero.
• Portable format — a model representation independent of the training framework.
• Runtime — the engine that executes a model on particular hardware.
• Operator fusion — combining several operations into one to avoid intermediate writes.
• Throughput and latency — work per second, and time for one request.

The mental model

Pruning removes what the network is not using, and the two kinds differ in a way that decides whether you get a speedup at all.

Unstructured pruning removes individual weights — typically the smallest ones — and can remove a large fraction with little accuracy loss, which sounds excellent. The catch is that the result is a dense array with zeros in it, and ordinary hardware multiplies zeros exactly as fast as anything else. So the model has fewer meaningful weights, the same memory footprint, and the same speed. Unstructured pruning frequently shrinks a model without making it faster, and it only pays where you have hardware or a library that genuinely exploits sparsity, or where a compressed file size is the goal.

Structured pruning removes whole channels, filters or layers, so what remains is a smaller dense network. Every piece of hardware runs it faster because it is genuinely smaller. It costs more accuracy for the same reduction, and it usually needs a short fine-tune to recover — and it is the one to reach for when latency is the objective, which is nearly always.

The workable procedure is iterative rather than one-shot: prune a modest fraction, fine-tune briefly, measure, repeat, and stop when accuracy falls below your requirement. Removing thirty percent at a time and recovering beats removing eighty percent once and finding nothing recovers.

A portable export is what makes deployment independent of the training stack. Export the trained model to an interchange format and it can run under a runtime that has never heard of the framework you trained in. That decoupling matters more than it sounds: the training framework is chosen for research convenience, the runtime for the target hardware, and tying them together means the deployment target constrains the research and every framework upgrade is a deployment risk.

Two cautions from practice. Export can fail on unusual operations or on control flow that depends on the data, and the failure is at export time, which is the good case — so export early, before the architecture is settled, rather than discovering at the end that a layer will not travel. And verify numerically after export: run the same inputs through the original and the exported model and compare the outputs within a tolerance. A silently different result is entirely possible and is the worst outcome here, and the check takes minutes.

Vendor runtimes optimise a model for particular hardware, and each gives large speedups and constrains where the model can run. There is one for the major accelerator ecosystem, one for a large processor vendor's chips and integrated graphics, and one for a mobile and desktop platform's own accelerators. What they do is much the same. They fuse operators, combining a convolution with its normalisation and activation so intermediate results are never written to memory. They choose layouts and kernels suited to the chip. They quantise to the precisions the hardware runs fastest. And they plan memory across the whole graph rather than per operation.

The trade is portability: a model optimised for one vendor's runtime runs on that vendor's hardware, and the optimisation is not transferable. Where you control the hardware, that is a fine trade for a large gain; where you do not, a portable runtime that is somewhat slower everywhere is the better engineering.

Then the measurement, which is the point of the topic.

Do not trust operation counts. A count of multiply-accumulates measures arithmetic, and arithmetic is frequently not what limits you. Two models with the same count can differ by a large factor in real latency, because one moves far more data. Memory bandwidth routinely dominates where arithmetic was blamed — depthwise convolutions are the standard example, having very few operations and poor arithmetic intensity, so they are much slower relative to their operation count than the count suggests.

Do not trust vendor figures. They describe a particular model, a particular input size, a particular batch, a particular precision, on a particular part, frequently at the largest batch that flatters throughput. Your model at batch one is a different measurement.

So measure properly, on the target device, and the protocol matters. Warm up first, discarding the initial runs, because the first inference includes allocation and compilation. Use your real input sizes and your real batch size, which for interactive work is one. Run enough repetitions to see the spread and report a high percentile rather than a mean, because the tail is what users experience. Measure end to end, including preprocessing and data transfer, since moving data to an accelerator is frequently a large share of the total and never appears in a model benchmark. And measure latency and throughput separately — batching improves throughput and worsens latency, so a single number conceals which you improved.

On edge hardware, energy is frequently the binding constraint, and it is the measurement people omit. A device that meets its latency target and flattens the battery in two hours has failed. Measure energy per inference — with a power meter, or from the device's own counters where it has them — and note that the fastest configuration is not always the most efficient, since running at a lower clock for longer sometimes uses less total energy.

The habit that makes all of it useful: measure before optimising and after each change, on the target. Optimisation guided by intuition in this area is wrong often enough to be a poor use of time, and the profile usually points somewhere unexpected — at a preprocessing step, at a data transfer, at one unfused operation — which is exactly the information that makes the order of magnitude available.

What you should now be able to explain or do

Distinguish unstructured from structured pruning and say which delivers a speedup on ordinary hardware. Apply iterative pruning with fine-tuning and a stopping criterion. Export to a portable format, export early, and verify numerically afterwards. Say what vendor runtimes do and what portability they cost. Explain why operation counts mislead and name the standard example. Run a benchmark with warm-up, real sizes, a high percentile, end-to-end scope and separate latency and throughput. Measure energy per inference and say why the fastest setting may not be the most efficient.

Check yourself