Does QLoRA produce a worse model than LoRA?

Usually the gap is small, but it is task-dependent. The honest answer is to run both on your own eval set — the difference that matters is the one on your data, not a benchmark average.

Can I merge a QLoRA adapter into the base model?

Yes, but you merge into a de-quantised copy of the base, not into the 4-bit weights. Budget the memory for that step.

Which rank should I pick?

Rank is a separate axis from LoRA vs QLoRA. Start low, raise it only if training loss plateaus above where you need it — higher rank costs memory and rarely fixes a data problem.

Comparison

LoRA vs QLoRASame adapter maths. Very different memory bills.

LoRA and QLoRA both freeze the base model and train a small set of low-rank adapter matrices. The difference is what precision the frozen base is held at during training — and that single choice decides which GPU you need.

All comparisons

The short answer

Start with QLoRA if the model does not comfortably fit on your GPU in 16-bit. Use LoRA when it does — you get slightly faster steps and one less moving part. Adapter quality between the two is close enough that memory, not accuracy, is usually the deciding factor.

Side by side

LoRA and QLoRA, dimension by dimension.

LoRA compared with QLoRA
Dimension	LoRA	QLoRA
Base model precision	16-bit (bf16/fp16), frozen	4-bit NF4 quantised, frozen
Trained parameters	Low-rank adapters only	Low-rank adapters only
Relative GPU memory	Baseline	Substantially lower — this is the whole point
Training step speed	Faster per step	Slower per step from de-quantisation overhead
Typical use	Model already fits in VRAM	Large model on a single smaller GPU
Merging adapters	Merge straight back into 16-bit weights	Merge into a de-quantised copy of the base
Main risk	Runs out of memory on big models	Quantisation error on sensitive tasks

Choose LoRA when

The base model fits in your GPU memory at 16-bit with headroom for activations.
You want the shortest wall-clock training time per epoch.
You plan to merge adapters and ship a single 16-bit checkpoint.
Your task is precision-sensitive and you would rather not add quantisation error.

Choose QLoRA when

You are fine-tuning a large model on one GPU rather than a cluster.
Memory is the binding constraint and you can trade step time for it.
You are iterating on many experiments and want each to be cheap.
You intend to serve quantised anyway, so training quantised matches deployment.

FAQ

Common questions.

Try it on your own data.That settles it.

The only comparison that decides this is the one run against your own evaluation set.

Read the quick start Talk to us

Dimension

LoRA

QLoRA

Base model precision

16-bit (bf16/fp16), frozen

4-bit NF4 quantised, frozen

Trained parameters

Low-rank adapters only

Relative GPU memory

Baseline

Substantially lower — this is the whole point

Training step speed

Faster per step

Slower per step from de-quantisation overhead

Typical use

Model already fits in VRAM

Large model on a single smaller GPU

Merging adapters

Merge straight back into 16-bit weights

Merge into a de-quantised copy of the base

Main risk

Runs out of memory on big models

Quantisation error on sensitive tasks