Comparison
Full Fine-Tuning vs PEFTUpdate every weight, or a carefully chosen few.
Full fine-tuning updates every parameter in the model. Parameter-efficient fine-tuning (PEFT) freezes the base and trains a small set of added parameters instead. The gap between them is narrower than most teams expect, and the operational difference is large.
The short answer
Reach for PEFT first. It is cheaper, faster to iterate on, and produces adapters you can swap per customer or per task. Escalate to full fine-tuning when PEFT has plateaued below your target and you have ruled out data quality as the cause — which, in practice, is where most plateaus actually come from.
Side by side
Full Fine-Tuning and PEFT (LoRA family), dimension by dimension.
| Dimension | Full Fine-Tuning | PEFT (LoRA family) |
|---|---|---|
| Parameters updated | All of them | A small added subset |
| GPU memory | High — weights, gradients and optimiser state | Much lower |
| Artefact size | A full model checkpoint | A small adapter file |
| Serving many variants | A full copy per variant | One base, many hot-swappable adapters |
| Catastrophic forgetting | Higher risk | Lower — base weights are frozen |
| Ceiling on adaptation | Highest | High, but bounded by adapter capacity |
| Iteration speed | Slow and expensive per run | Fast enough to run many experiments |
Choose Full Fine-Tuning when
- You are adapting to a domain genuinely distant from the pre-training distribution.
- PEFT has plateaued and you have confirmed the data is not the bottleneck.
- You are training a new base model for others to build on.
- You have the compute budget and the evaluation discipline to justify it.
Choose PEFT (LoRA family) when
- You are adapting tone, format or task behaviour rather than core capability.
- You need per-customer or per-task variants without per-variant hosting cost.
- You want to run many experiments before committing to one recipe.
- You want to preserve the base model’s general ability alongside the new skill.
FAQ
Common questions.
Try it on your own data.That settles it.
The only comparison that decides this is the one run against your own evaluation set.