8.9 Fine-tuning: LoRA, QLoRA, PEFT

What this is and why it exists

Fine-tuning used to mean updating every parameter, which meant hardware most people do not have. Adapter methods changed that: train a small number of additional parameters alongside a frozen model and the work fits on one accelerator. The mechanics are approachable. The decision framework matters more, and it comes down to one sentence you should be able to defend — fine-tune for form, retrieve for facts — because getting that backwards is the most expensive mistake in this area.

The vocabulary

• Full fine-tuning — updating every parameter of the model.
• Parameter-efficient fine-tuning — training a small added set while the rest stays frozen.
• Update matrices — the small trainable matrices an adapter method adds.
• Rank — how much capacity those matrices have, written r.
• Target modules — which parts of the network the adapters are attached to.
• Quantised training — training adapters on top of heavily compressed frozen weights.
• Adapter — the trained result, small enough to store and swap per task.
• Catastrophic forgetting — losing general capability while learning a narrow task.

The mental model

Three ways to change a model's behaviour, and they do different things.

Prompting changes the instruction. It is instant, free to iterate, and adjusts form and framing. Its limits are the tokens it costs on every request and the fact that it cannot install anything the model does not know.

Retrieval supplies facts at request time. The facts stay in a store you can update, correct, cite and remove, and the model reads them fresh on every request.

Fine-tuning changes the weights. It teaches form, format, tone, domain vocabulary, and behaviour — how to respond, not what is true. It costs a training run and a dataset, and it produces something you then have to serve and version.

So the decision framework. Fine-tune when the model can do the task but not in the shape you need — a house style it will not hold across a long document, an output format that keeps drifting despite examples, a domain register, a decision procedure with many implicit conventions, or when a long prompt is doing the same instructing on every request and moving it into the weights removes that cost. Retrieve when the answer depends on facts, especially ones that change or need citing. Prompt when either would work, because it is the cheapest thing to try and the fastest to undo.

The trap is fine-tuning to inject knowledge, and it fails in a specific way worth understanding. Facts trained into weights cannot be cited, cannot be updated without another training run, cannot be removed on request, and — the damaging part — get blended. The model learned the statistical shape of your documents rather than a lookup table, so it produces confident text in the right register that mixes facts from different documents into something that was never true. That is worse than a wrong answer, because it is a plausible one with your organisation's voice. Facts belong in retrieval where they can be updated and cited.

Now the mechanics, which are simpler than their reputation. The adapter approach represents the weight updates with two smaller matrices through low-rank decomposition, which "can be trained to adapt to the new data while keeping the overall number of changes low. The original weight matrix remains frozen and doesn't receive any further adjustments. To produce the final results, both the original and the adapted weights are combined."

Two settings do most of the work. The rank is "the rank of the update matrices, expressed in int. Lower rank results in smaller update matrices with fewer trainable parameters" — so it is a capacity dial: too low and the adapter cannot represent what you are teaching, too high and it overfits a small dataset and costs more. Start low and raise it only if the training loss says the adapter is the limit. A scaling factor, conventionally alpha, controls how strongly the adapter's contribution is weighted. And target modules names "the modules (for example, attention blocks) to apply the LoRA update matrices" — the documentation notes that in principle the method applies to any subset of weight matrices, but "for simplicity and further parameter efficiency, in Transformer models LoRA is typically applied to attention blocks only."

Because the base model is untouched, the trained adapter is small, storable and swappable: one base model in memory with several task adapters is a genuinely different deployment shape from several fine-tuned models.

Quantised adapter training is what brought this within reach of one consumer accelerator. Hold the frozen base weights at heavily reduced precision so they occupy a fraction of the memory, and train the adapters at higher precision on top. The frozen weights are never updated, so their compression costs less than it would in ordinary training, and the memory saving is what makes the difference between fitting and not fitting. It is slower per step than uncompressed training and it is the technique that opened this work to individuals rather than institutions.

Then the part that decides whether it works: the dataset is the project. Everything else here is a configuration file.

Assemble examples that show exactly the behaviour you want, in the exact format you want it, including the awkward cases and at least some examples of declining. Quality dominates quantity by a wide margin — a few hundred carefully written examples routinely beat many thousands of scraped ones, because the model learns whatever is consistent in the data, including its mistakes and its accidental patterns. Hold out a test set from the start, split by time or by source rather than at random. And check for the leakage the classical module warned about, since a duplicated example across the split makes the result meaningless.

And measure before and after, on the same held-out set, which is what makes this a result rather than an anecdote. Record the base model's performance first. Then the tuned one. Then two things people skip: a general-capability check, because narrow tuning can cost broad ability — the model that now writes perfect summaries in your format and has become worse at everything else is a real and common outcome — and a comparison against the cheaper alternatives you skipped, meaning a good prompt with few-shot examples, and retrieval. If a better prompt gets ninety percent of the benefit for none of the operational cost, that is the answer, and finding out afterwards is expensive.

What you should now be able to explain or do

Distinguish prompting, retrieval and fine-tuning by what each changes. State the framework in one sentence and defend it. Explain exactly how knowledge fine-tuning fails, including blending. Describe the low-rank decomposition and what stays frozen. Set rank, scaling and target modules with a reason for each. Say what quantised adapter training makes possible and what it costs. Build a dataset with quality over quantity, held-out splits and declining examples. Measure before and after, including a general-capability check and a comparison against the cheaper alternatives.

Check yourself