Evaluation
Check a fine-tuned model on examples it has not seen before you ship it.
Keep some examples back
Before training, set aside a slice of your data (around one in ten examples) and don't train on it. Those held-out examples show whether the model learned the task or only memorised the training set.Pick examples that look like what real users will ask, including the hard and unusual ones.
Generate answers for the held-out examples
Run the trained model on each held-out prompt and read the answers next to the ones you expected. For most assistants, a careful read of a few dozen answers tells you more than a single score.
python
1from langtune import FastLanguageModel23FastLanguageModel.for_inference(model, tokenizer)45for example in held_out[:20]:6 answer = FastLanguageModel.generate(model, tokenizer, example["prompt"])7 print(example["prompt"], "→", answer)
Compare with the base model
Load the base model you started from and ask it the same prompts. If the fine-tuned answers aren't clearly better on your task, add or clean up training data before training again.
python
1base, base_tokenizer = FastLanguageModel.from_pretrained("meta-llama/Llama-3.1-8B")2FastLanguageModel.for_inference(base, base_tokenizer)34for example in held_out[:20]:5 print("base: ", FastLanguageModel.generate(base, base_tokenizer, example["prompt"]))6 print("tuned:", FastLanguageModel.generate(model, tokenizer, example["prompt"]))
Automatic scores
If your task has one right answer (a label, a number, a short extraction), count how often the model gets it exactly right on the held-out set. For open-ended text, standard metrics such as ROUGE or BERTScore from open-source libraries can help rank runs, but they are no substitute for reading the answers.