LoRA & QLoRA

Train a small adapter instead of the whole model. It is how most fine-tunes are done, and it fits on one GPU.

PyTorch
PEFT

How LoRA works

LoRA freezes the base model's weights W and learns a small update next to each chosen layer, written as two thin matrices: ΔW = B·A. With rank r = 16, that is well under 1% of the parameters of a 7B model, so training needs far less memory, and the result is an adapter file of tens of megabytes instead of a new copy of the model.

How QLoRA differs

QLoRA loads the frozen base model in 4 bits (NF4) and trains the same adapter on top. Memory drops by about three quarters, so a 7B or 8B model fits on a single 16 GB GPU, at a small cost in speed. It is Langtrain's default.

Use it

load_in_4bit=True with method="qlora" gives QLoRA; load_in_4bit=False with method="lora" gives plain LoRA.

python

1from langtrain import FastLanguageModel
2
3model, tokenizer = FastLanguageModel.from_pretrained(
4    "meta-llama/Llama-3.1-8B-Instruct",
5    load_in_4bit=True,
6)
7model = FastLanguageModel.get_peft_model(
8    model,
9    r=16,
10    lora_alpha=32,
11    lora_dropout=0.05,
12    method="qlora",
13)

Settings

r (rank) - 16 by default. 8 is often enough for style and format; 32 to 64 for harder tasks with plenty of data
lora_alpha - scales the update; twice r is a good rule
lora_dropout - 0.05 by default; raise it if the model overfits a small dataset
target_modules - which layers get adapters; by default all attention and MLP layers
use_rslora=True - scales by alpha / √r, which keeps high ranks stable

Other adapters

The same call takes method="dora" (DoRA: splits each weight into size and direction, often a little better than LoRA at the same rank), "ia3" (even fewer parameters) and "prefix" (learned prompt vectors). For our own recipe, see AdaptiveRank in the Python SDK.

Merge and save

Merging folds the adapter into the base weights, giving a standard model that any tool can run. Without merging you save only the adapter, which you can load on top of the same base model later.

python

1# Standard weights for llama.cpp, vLLM, Ollama
2FastLanguageModel.save_pretrained(model, tokenizer, "./my-model", merge_before_save=True)
3
4# Adapter only
5FastLanguageModel.save_pretrained(model, tokenizer, "./my-adapter")

How LoRA works

Use it

load_in_4bit=True with method="qlora" gives QLoRA; load_in_4bit=False with method="lora" gives plain LoRA.

python

1from langtrain import FastLanguageModel
2
3model, tokenizer = FastLanguageModel.from_pretrained(
4    "meta-llama/Llama-3.1-8B-Instruct",
5    load_in_4bit=True,
6)
7model = FastLanguageModel.get_peft_model(
8    model,
9    r=16,
10    lora_alpha=32,
11    lora_dropout=0.05,
12    method="qlora",
13)

Settings

r (rank) - 16 by default. 8 is often enough for style and format; 32 to 64 for harder tasks with plenty of data
lora_alpha - scales the update; twice r is a good rule
lora_dropout - 0.05 by default; raise it if the model overfits a small dataset
target_modules - which layers get adapters; by default all attention and MLP layers
use_rslora=True - scales by alpha / √r, which keeps high ranks stable

Merge and save

Merging folds the adapter into the base weights, giving a standard model that any tool can run. Without merging you save only the adapter, which you can load on top of the same base model later.

python

1# Standard weights for llama.cpp, vLLM, Ollama
2FastLanguageModel.save_pretrained(model, tokenizer, "./my-model", merge_before_save=True)
3
4# Adapter only
5FastLanguageModel.save_pretrained(model, tokenizer, "./my-adapter")