12.6 Small models and on-device AI

What this is and why it exists

A surprising share of real tasks do not need a frontier model, and capability per parameter keeps improving, so the share grows. Running on the device adds two arguments that hold regardless of capability — the data never leaves, and it works without a connection — and in a country where connectivity is uneven and data costs money, those are not abstractions. The mistake is anchoring on leaderboard position: the question is whether the small model clears your task's bar, not whether it beats the largest.

The vocabulary

• Small language model — one small enough to run on ordinary consumer hardware.
• Capability per parameter — how much a model can do for its size.
• On-device inference — running the model on the user's hardware.
• Browser accelerator interface — the web standard giving pages access to a device's accelerator.
• Offline-first — designed to work without a connection and synchronise later.
• Task bar — the level of capability your specific task actually requires.

The mental model

Capability per parameter has improved considerably, and the reasons are worth knowing because they suggest it continues. Better data, filtered harder rather than merely collected in greater volume. Training well past the compute-optimal point, since a smaller model that is cheaper forever is worth extra training. Distillation from larger models, which is the previous topic. And better recipes throughout.

The consequence is that a model small enough to run on a phone now does things that recently required a server, and the deployment question has changed as a result: it is no longer whether a small model is capable enough in general, but whether it clears the bar for your particular task.

Reframing the question is the point of this topic. "Which model is best" invites a leaderboard, and a leaderboard measures general capability on tasks that are not yours. "Does this model clear my bar" is answerable: take your evaluation set from the evaluation topic, run the small model against it, and compare the result with your requirement. That is a measurement rather than an opinion, it takes an afternoon, and it frequently comes back yes for tasks people assumed needed the largest available model — classification, extraction, routing, summarising short text, tidying structure, drafting from a template.

The tasks where small models still fall short are also predictable: long multi-step reasoning, wide world knowledge without retrieval, unusual languages and code beyond the routine. Notice that the first two are addressable by architecture rather than by model size — give a small model retrieval and it has the knowledge; give it a fixed workflow rather than an open agent loop and it does not need the long reasoning. A small model with good scaffolding beats a large model with none on a great many real tasks, and that is the design insight worth carrying.

On-device inference is now a genuine architectural option rather than a demonstration. Phones have dedicated inference processors, as the accelerator topic described, and quantised small models fit in their memory. On desktops, ordinary consumer hardware runs capable models comfortably. And in the browser, the accelerator interface gives a web page access to the device's graphics hardware for general computation — which is what makes in-browser models practical rather than a curiosity, and it removes the server from the picture entirely: the page loads, the model runs locally, and nothing is sent anywhere.

The engineering costs are real and should be stated. The model has to be delivered to the device, which is a substantial download the first time and needs caching and a story for what happens on a slow connection. Devices vary enormously, so you need a floor and a fallback for hardware below it. Battery and thermal limits mean sustained inference on a phone is constrained in ways a server is not. And updating a model on a million devices is a distribution problem rather than a deployment.

Then the two arguments that hold regardless of capability, and they are the reason this matters here.

Privacy is structural rather than promised. Data that never leaves the device has not been transmitted, logged, retained or subpoenaed. For personal messages, health information, photographs, financial records and anything a professional obligation covers, that is a different kind of assurance from a policy statement — and under the data-protection obligations from the production module, data you never collect needs no consent, no retention rule and no deletion path. On-device processing is the strongest form of minimisation available.

Offline operation is a real requirement, not a nicety. Connectivity in India is uneven — good in cities, patchy elsewhere, unreliable on the move, and metered in a way that makes constant round trips a cost to the user rather than to you. A feature that works on a train, in a basement, in a field, or during an outage is a materially better feature, and one that fails in those places will be judged by its worst experience. Design offline-first where the task allows: work locally, synchronise when a connection appears, and treat the network as an enhancement rather than a dependency.

Two further arguments are worth having in the list. Latency: no network round trip means a response can be immediate, which for anything interactive is quality rather than efficiency. And cost: inference on the user's hardware costs you nothing per request, which changes the economics of a free tier or a high-volume feature entirely.

The practical shape most systems end up with is a hybrid, and it is worth designing deliberately rather than arriving at. Run the small local model by default: it is fast, private, free and works offline. Escalate to a larger remote model when the task warrants it — which the local model can frequently judge — and be explicit with the user about when that happens, because it is the moment their data leaves the device and they are entitled to know.

And the closing caution about the leaderboard anchor. Positions change monthly, the numbers are contested for the reasons the next topic gives, and the gap between a small model and the largest is not the number that decides your design. Your task's bar is. Measure against that, re-measure when a new small model appears, and expect the answer to move in your favour over time rather than against it.

What you should now be able to explain or do

Give the reasons capability per parameter improved and why that trend suggests continuation. Reframe the question from which model is best to whether it clears your bar, and measure it. Name the tasks small models handle and those they do not, and say which of the latter are addressable by scaffolding. Say what makes on-device inference practical on phones, desktops and in the browser. Name the four engineering costs. Argue the privacy and offline cases in terms of minimisation and of real connectivity. Design a hybrid that is local by default and explicit when it escalates.

Check yourself