8.10 Quantization, distillation and local models
You can run a capable model on hardware you own.
Before:07. Natural Language ProcessingUnlocks:09. Agentic AI12. Frontier Topics
Quantization compresses model weights to fewer bits so capable models run on hardware you own, and distillation shrinks capability into smaller models outright; Ollama and llama.cpp make local serving practical. It sits here as the ownership topic — models on your machine, your data staying home. The measurement rule holds: quality loss from quantization is task-dependent and sometimes severe, so the size drop is only half the story until your own evaluation tells the rest.
Work through these
GPTQ, AWQ, GGUF and bit-widths
Several methods for storing weights at reduced precision, and the file formats associated with each. The bit-width chosen is a direct trade between memory and quality.
Quality loss measurement after quantization
Compression costs quality, and the only honest way to know how much is to measure on your own task. Published averages will not tell you about your use.
Distilling a large model into a small one
Training a small model to imitate a large one, which produces something far cheaper to run at some cost in capability. It suits narrow tasks particularly well.
Ollama, llama.cpp, LM Studio
The tools that run models locally on ordinary hardware, which is what makes private and offline use possible. Running one is the fastest way to understand what local deployment involves.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.