Langvision·Open-source library
Teach a modelto see your world.
Langvision is an open-source Python library for fine-tuning vision-language models on your own images and text: answering questions about photos, describing them, or reasoning about what they show.
from langvision import FastVisionModel
from PIL import Image
model, processor = FastVisionModel.from_pretrained(
"llava-hf/llava-1.5-7b-hf",
load_in_4bit=True,
)
# LoRA on the language side; the vision encoder stays frozen
model = FastVisionModel.get_peft_model(model, r=16, method="qlora")
FastVisionModel.train(model, processor, dataset, output_dir="./output")
answer = FastVisionModel.generate(
model, processor,
images=Image.open("invoice.jpg"),
prompt="What is the total amount due?",
)What you get
The Langtune approach, for images.
Popular open models
Works with LLaVA 1.5, Qwen-VL, InternVL, CogVLM and Phi-3 Vision, loaded from Hugging Face.
Runs locally
Fine-tune on your own GPU. Images stay on your machine unless a Langtrain API key is set.
Cloud when you need it
Need a bigger GPU? Start the run on Langtrain's GPUs from the dashboard, then take the weights back.
Simple data format
One JSON line per image: the image path and the conversation about it.
Export
Save the trained model as ONNX or safetensors, to run it where you like.
MIT licensed
Read the code, change it, and use it commercially.
Your data
Pictures and what to say about them.
Each line pairs an image with the questions and answers you want the model to learn. Start with a few hundred good examples.
{"image": "photos/0001.jpg",
"conversations": [
{"from": "human", "value": "What's wrong with this part?"},
{"from": "assistant", "value": "A hairline crack near the left mount."}
]}The Langtrain family
Four ways in. One model you own.
Use whichever fits how you work. They share one account and one API, so a run started in the cloud from any of them shows up in all of them.
- Langtrain StudioDesktop appManage datasets and fine-tuning runs from a native app for Windows and Linux.
- Langtrain SDKPython & TypeScriptUpload data, fine-tune and follow runs from code, with the lt command line.
- LangtuneOpen-source libraryFine-tune open text models like Llama, Mistral and Qwen on your own GPU.
- You're hereLangvisionOpen-source libraryFine-tune vision-language models like LLaVA and Qwen-VL on your own images.
Fine-tune a vision modelon your own images.
Install it and train on your first dataset. Langvision is also included in langtrain-ai[vision].