8.10 Quantization, distillation and local models

What this is and why it exists

Quantisation stores weights at reduced precision so a capable model fits on hardware you own; distillation trains a small model to reproduce a large one's behaviour. Between them, running a useful model locally is now ordinary rather than exotic — which matters for privacy, for cost at volume, and for working offline. The trap is trusting the size reduction without measuring quality on your task, because the damage is uneven and a model that looks fine on general tests can be quietly broken for the one thing you need.

The vocabulary

• Precision — how many bits each weight is stored in.
• Quantisation — converting weights to a lower-precision representation.
• Post-training quantisation — compressing an already-trained model.
• Calibration data — a sample used to decide how to map values during compression.
• Bit-width — the number of bits per weight, the direct memory-versus-quality dial.
• Distillation — training a small student model on a large teacher's outputs.
• Perplexity — how surprised a model is by text, a common but blunt quality proxy.
• Local serving — running the model on hardware you control.

The mental model

Weights are numbers, and you are choosing how precisely to store each one. Training precision keeps a wide range and fine gradations; compressed formats keep far fewer distinct values and map each weight to the nearest. The memory saving is close to the ratio of bit-widths, which is the whole appeal: a model that would not load at training precision loads comfortably at a quarter of it, and memory is almost always the binding constraint on local hardware.

The quality cost is not uniform, and understanding why explains the whole practice. Some weights matter far more than others, and a few large-magnitude values dominate a layer's behaviour; compressing everything at the same resolution spends precision on weights that do not need it and destroys the few that do. So the good methods are all variations on spending precision where it matters: compress in small groups with their own scaling factors, use a small sample of representative text to find which weights are sensitive and treat them differently, and keep the layers known to be fragile at higher precision.

The named methods differ mostly in how they decide that, and the practical consequences are: one family reconstructs each layer's output using calibration data, one identifies the weights that matter most by watching activations, and one is a container format designed for running on ordinary processors with a range of bit-widths and mixed-precision variants inside a single file. Which is available depends on your serving stack more than on any quality argument.

Bit-width is the dial, and the shape of the trade is the useful knowledge. Coming down from training precision to half of it is essentially free. Coming down further is where judgement starts: at moderate compression the loss is small and frequently invisible; at aggressive compression it becomes real; and past a point the model degrades sharply rather than gradually. The practical rule is that a more compressed larger model usually beats a less compressed smaller one at the same memory budget — but only within the range before the sharp degradation, and where that point sits varies by model and by task.

Which is why the measurement rule is the point of this topic. Published averages tell you what happened on somebody else's evaluation. They do not tell you what happened to the capability you depend on, and degradation is uneven: compression tends to damage the long tail first — rare knowledge, precise arithmetic, structured output formats, less-represented languages, long-range consistency — while leaving common conversational fluency intact. So a compressed model can chat impeccably and have become worse at emitting valid JSON, or at your domain's terminology, or at anything in a language other than the dominant one in its training data.

The check is not elaborate. Take your own evaluation set from the evaluation topic, run it against the uncompressed model and each compression level, and compare. Include the specific things you rely on — schema conformance, tool-call correctness, arithmetic, the languages your users write in — because those are exactly what a general benchmark averages away. A general quality proxy such as perplexity is a screen, not evidence: it can barely move while a capability you need has collapsed.

Distillation is the other route to small, and it works differently. Rather than compressing a model, train a small one to reproduce a large one's behaviour — on its outputs, and often on its full output distribution, which carries more information than the single chosen token. The result is a model that is genuinely small rather than a compressed large one, so it is faster in a way compression cannot match.

Its shape suits narrow tasks particularly well: a small model taught to do one thing the way a large model does it can match it closely on that thing while being far cheaper, and it will not have the general capability the teacher had. That is a feature when the task is fixed. Practically, this is often the best-value move available — use a large model to generate a training set for your specific task, train or fine-tune a small model on it, and serve the small one, which is the same hybrid the encoder topic recommended. Check the generated labels by hand on a sample first, since you are inheriting the teacher's mistakes as ground truth.

On running one locally, the tooling is now genuinely undemanding: an efficient inference engine written to run on ordinary processors as well as accelerators, a command-line tool that fetches and serves models with one command, and desktop applications for people who would rather not use a terminal. All of them speak a request format compatible with the common hosted interfaces, which means your application code frequently does not change between a hosted model and a local one — the endpoint changes.

And running one is the fastest way to understand what local deployment involves, which is the recommendation this topic ends on. Half an hour with a model on your own machine teaches the memory constraint, the speed, the quality difference and the operational reality more concretely than any amount of reading. It also makes the privacy argument concrete: text that never leaves your machine has not left your machine, and for regulated, confidential or personal data that is not a preference but a requirement.

What you should now be able to explain or do

Say what quantisation changes and why the memory saving tracks the bit-width. Explain why compressing everything equally is wasteful and what the good methods do instead. Describe the shape of the quality trade across bit-widths and state the larger-model rule with its caveat. Say what degrades first under compression and design a check using your own evaluation set. Explain why a general quality proxy is a screen rather than evidence. Describe distillation, how it differs from compression, and where it suits. Run a model locally and say what that makes concrete.

Check yourself