11.7 Model optimization for hardware
You can cut inference cost by an order of magnitude.
Before:06. Deep Learning
Pruning, ONNX export and runtime stacks — TensorRT, OpenVINO, Core ML — can cut inference cost by an order of magnitude without retraining from scratch, with benchmarking of latency, throughput and energy as the evidence layer. It follows TinyML as the general optimisation toolkit. The scepticism is the skill: vendor numbers and FLOP counts are marketing until your own hardware has produced the measurement.
Work through these
Pruning: structured and unstructured
Removing weights or whole structures from a trained network, with the structured version giving speedups that hardware can actually use. Unstructured pruning often shrinks a model without making it faster.
ONNX export and runtime portability
Exporting to a portable model format so it can run outside the framework it was trained in. This is what makes deployment independent of the training stack.
TensorRT, OpenVINO, Core ML
Vendor runtimes that optimise a model for particular hardware, one for each major platform. Each can give large speedups and each constrains where the model runs.
Benchmarking latency, throughput and energy
Measuring response time, throughput and energy on the target device rather than estimating them. On edge hardware, energy is frequently the binding constraint.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.