11.6 Edge AI and TinyML

What this is and why it exists

Running a neural network on a microcontroller — kilobytes of memory, milliwatts of power, no operating system worth the name — is where embedded instincts become the differentiator. This topic is the workflow from a trained model to a device, the quantisation that makes it fit, and the rule that decides whether the project succeeds: read the budget before designing the model, because a model twenty percent too large is not slightly worse, it is not deployable.

The vocabulary

• Microcontroller — a small processor with memory measured in kilobytes and no operating system.
• Flash — the non-volatile storage holding the program and the weights.
• RAM — the working memory holding activations during inference.
• Arena — the fixed block of RAM the inference runtime is given.
• Post-training quantisation — reducing precision after training.
• Quantisation-aware training — training with the reduction simulated.
• Operator support — whether the runtime implements a layer at all.
• Duty cycle — the fraction of time the device is awake.

The mental model

Start from the budget, and it has three separate limits that must all be met.

Flash holds the program and the weights, and it is the limit people notice first. RAM holds the activations during inference, and it is the limit that surprises people: the working memory needed is the largest set of tensors alive at once, which for a convolutional network is usually an early layer's feature map, and it can dwarf the weights. Latency is how long one inference takes, which decides whether the device keeps up with its sensor — and on a small processor it is dominated by the number of multiply-accumulate operations, so it is estimable from the architecture before anything is trained.

There is a fourth in battery-powered devices: energy per inference, which is latency multiplied by power draw, and which combined with how often you run decides how long the device lasts. Frequently the largest saving is not in the model but in the duty cycle — a cheap always-on detector that wakes the real model rarely, exactly the staged arrangement the audio topic described.

Write those four numbers down before choosing an architecture. The design that follows is different: fewer channels, aggressive early downsampling to shrink the feature maps that dominate RAM, depthwise separable convolutions for their operation count, and a model measured in tens of kilobytes rather than tens of megabytes. A model designed first and squeezed later usually cannot be squeezed enough, and the work is wasted.

The workflow from laptop to device has four stages, and each can fail in its own way.

Train on ordinary hardware, in your usual framework, with the budget as a constraint on the architecture.

Convert to the compact interchange format for on-device inference — a flat file the small runtime can read without parsing overhead.

Quantise, which is what actually makes it fit, described below.

Deploy onto the device, where the model is usually compiled into the firmware as an array of bytes, and the runtime is a reduced version that assumes no operating system, no dynamic memory allocation and a fixed memory arena you size yourself.

Two failures particular to this path deserve naming. Operator support: the reduced runtime implements a subset of operations, and a layer it does not implement fails at conversion or at load rather than running slowly. Check the supported set before designing, not after. And the memory arena: you declare a fixed block and the runtime works inside it, so too small fails at initialisation and too large leaves nothing for your application — the size is found by measuring rather than guessing, and the tooling reports what was actually needed.

Quantisation is what makes the model fit, and it is more consequential here than in the server case. Full-precision weights are four bytes each; eight-bit integers are one, so the model is a quarter of the size and the arithmetic is integer arithmetic, which small processors do far faster than floating point — many have no floating-point unit at all, making this the difference between running and not.

Post-training quantisation converts an already-trained model, using a small sample of representative data to choose the mapping from real values to integers. It is fast, it needs no retraining, and it costs some accuracy. Provide genuinely representative calibration data — a few hundred examples resembling deployment — because calibrating on unrepresentative data produces ranges that clip real inputs, and that failure looks like a model that is merely bad.

Quantisation-aware training simulates the reduction during training, so the model learns weights that survive it. It gives better results and costs more effort: a training run, a modified pipeline, more moving parts. The rule is to try the cheap route first and measure, and reach for the trained route when the accuracy loss is more than you can accept — which happens more often at these sizes than at server scale, because a small model has less redundancy to give up.

Measure the quantised model on your own data at every step, for the reason the compression topic gave: degradation is uneven, and a general benchmark averages away exactly the capability you need.

The boards are where all of it becomes real, and there are three families you will meet. A widely available wireless-capable module is the usual starting point for connected projects, with enough memory for meaningful models and radio built in. The small development boards designed for machine learning bring sensors already attached, which removes the hardware work from a first project. And the broad family of general-purpose microcontrollers spans a wide range of capability, from parts that will barely hold a model to ones with accelerators for exactly this.

The practical advice is unglamorous and correct: choose a board you can actually obtain, with a supply you can rely on and a community that has already met the problems you will meet. A part that is superior on paper and unavailable in your market, or unsupported by the toolchain, is not a better choice.

And then the thing that makes this topic worth its place. Getting a model onto a device teaches, in one afternoon, what no amount of reading conveys: how quickly memory disappears, how much quantisation costs on your task, why operator support matters, and how different the engineering is when there is no room for anything to be approximate. Everyone who does it once reads model architectures differently afterwards — with the memory column in view — and that is exactly the instinct that makes hardware-adjacent work pay.

What you should now be able to explain or do

Name the three hard limits plus energy, and say which one surprises people and why. Design an architecture against the budget rather than squeezing afterwards, and use duty cycling. Carry out the four workflow stages and name the failure in each. Check operator support before designing and size the memory arena by measurement. Choose between post-training and quantisation-aware conversion, and supply representative calibration data. Measure on your own data at every step. Choose a board on availability and support.

Check yourself