Preference tuning
When you can't write the perfect answer but can say which of two is better, preference tuning moves the model toward the one your team would pick.
When to use it
Run it after SFT, once the model already does the task. Preference tuning then sharpens tone, safety, length and judgement: shorter answers, refusing what it should refuse, not inventing facts.
Pick a method
- dpo - learns from pairs of a better and a worse answer. The usual first choice
- orpo - also learns from pairs, and folds SFT in, so it can start from a base model
- simpo - pairs again, without a reference model, so it uses less memory
- kto - learns from single answers marked good or bad. Use it when you have thumbs up / down instead of pairs
Data for DPO, ORPO and SimPO
Each line is a prompt with the answer you prefer and the one you don't.
python
1{"prompt": "Summarise our refund policy.", "chosen": "Full refund within 30 days, no questions asked.", "rejected": "Refunds may or may not be possible depending on various factors."}
Data for KTO
Each line is a prompt, one answer, and whether it was good (true) or bad (false).
python
1{"prompt": "Summarise our refund policy.", "completion": "Full refund within 30 days, no questions asked.", "label": true}
Train
Choose the method with method=. beta sets how far the model may move from where it started: lower moves further. Start from your SFT model.
python
1from datasets import load_dataset2from langtrain import FastLanguageModel34model, tokenizer = FastLanguageModel.from_pretrained("./my-sft-model", load_in_4bit=True)5model = FastLanguageModel.get_peft_model(model, r=16, method="dpo")67pairs = load_dataset("json", data_files="preferences.jsonl", split="train")8FastLanguageModel.train(9 model, tokenizer, pairs,10 output_dir="./output-dpo",11 num_train_epochs=1,12 beta=0.1,13)
In the dashboard
The dashboard's Human Feedback page collects preferences for you: rank the model's answers and each choice is saved as a preferred and a rejected answer, ready for DPO.