Fine-tuning

Fine-tuning continues training an open model on your examples so it answers the way you need, without a long prompt.

What it changes

Fine-tuning is good at behaviour: format, tone, following your rules, doing one task reliably. It is a poor way to add facts that change often, like prices or stock; give the model those at question time instead, by retrieval or in the prompt.

Methods

Choose one with method= in get_peft_model:

qlora (default), lora - a small adapter on a frozen model. See LoRA & QLoRA
dora, ia3, prefix - other adapter types
dpo, orpo, simpo, kto - learn from preferred answers. See Preference tuning

Adapter methods train on example conversations, which is supervised fine-tuning.

A typical path

1. Write a few hundred example conversations and keep 10% back2. Train a QLoRA adapter on an instruct model3. Read its answers on the held-back examples (Evaluation)4. Fix the data where it goes wrong, and train again5. If tone or judgement still needs work, collect preferred and rejected answers and run DPO6. Merge and take the weights with you

In code

The whole loop, on your own GPU.

python

1from datasets import load_dataset
2from langtrain import FastLanguageModel
3
4model, tokenizer = FastLanguageModel.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", load_in_4bit=True)
5model = FastLanguageModel.get_peft_model(model, r=16, method="qlora")
6
7data = load_dataset("json", data_files="train.jsonl", split="train").train_test_split(test_size=0.1)
8FastLanguageModel.train(model, tokenizer, data["train"], output_dir="./output")
9
10FastLanguageModel.save_pretrained(model, tokenizer, "./my-model", merge_before_save=True)