EC-20.7 Sensor Fusion, and Machine Learning on the Device

What this is and why it exists

Two cheap sensors that fail differently can beat one expensive sensor.

That is the whole argument for fusion, and the mathematics that makes it precise is the same filtering that appears in navigation and control. It is worth meeting here because embedded devices are where most sensors actually live.

The second half of the topic is newer. A model quantised to eight-bit integers now runs on a part with a few hundred kilobytes of memory. That turns "where does the computing happen" from a given into a design decision, with an energy cost attached to each answer.

The vocabulary

• Bias — a constant offset in a sensor's reading.
• Drift — a slow change in that offset over time or temperature.
• Complementary filter — trusting one sensor at low frequency and another at high frequency.
• Kalman filter — a prediction from a model, corrected by a measurement, weighted by trust.
• State — the quantities the filter is estimating.
• Quantisation — representing model weights with fewer bits, typically eight.
• Pruning — removing weights that contribute little to the output.
• Inference — running a trained model to produce an answer.
• Drift, of a model — the gradual mismatch between training data and the world.

The mental model

Each sensing technology fails in its own way. An accelerometer is noisy but does not drift. A gyroscope is smooth but its offset wanders. A satellite receiver is absolutely referenced but slow and sometimes absent. The failures are largely independent, and that independence is what makes combination worthwhile.

The complementary filter exploits this in two lines of arithmetic. Trust one sensor below some frequency, trust the other above it, add the two. For a surprising number of real problems this is the whole answer, and reaching for something heavier before trying it wastes effort.

The Kalman filter is the same idea done properly. It keeps a model of the state and predicts what the state will be next. It then blends that prediction with the new measurement, weighted by trust. Reading it in that order makes the equations follow from the idea, rather than the other way round.

On the machine learning side, the constraint is memory and energy rather than accuracy. Quantisation to eight-bit integers and pruning of low-contribution weights together shrink a model by an order of magnitude. The accuracy loss is usually small. Whether that loss is acceptable is a question about the application, not about the model.

Then the placement decision. Running on the device costs memory and energy but no network. Running on a server costs a radio transmission and a round trip, which the previous topic showed to be the expensive operation. Running on a nearby box costs a second thing to maintain. The energy budget usually settles it, and the answer often surprises people.

Finally, honesty about data. A model trained on one installation frequently fails at the next, and nothing on the device will announce this. Deciding in advance what evidence would show the model has gone stale is part of shipping one. That is the same habit as defining a safe state before adding a watchdog.

What you should now be able to explain or do

• Say why two imperfect sensors can beat one good one, and when they cannot.
• Implement a complementary filter and say what its crossover frequency means.
• Describe the Kalman filter as prediction, measurement and weighting.
• Explain what quantisation and pruning cost and what they buy.
• Decide where inference should run for a stated device, with an energy argument.
• Name the evidence that would show a deployed model has drifted.

Check yourself