Run your model anywhere

A fine-tuned model is a set of standard weights. Train it on your own GPU or on Langtrain, then run it offline, on your own servers, or anywhere open models run.

Your own GPU
Hugging Face
Offline
vLLM

Train on your own GPU

Langtune trains on your machine, so your data is never uploaded. Save with merge_before_save=True to get full model weights in a folder, ready for the tools below.

python

1pip install "langtrain-ai[train]" langtune
2
3from langtune import FastLanguageModel
4
5model, tokenizer = FastLanguageModel.from_pretrained("meta-llama/Llama-3.1-8B")
6model = FastLanguageModel.get_peft_model(model, r=16, method="qlora")
7FastLanguageModel.train(model, tokenizer, dataset, output_dir="./output")
8
9# Merge the adapter into the base model and save standard weights
10FastLanguageModel.save_pretrained(model, tokenizer, "./my-assistant", merge_before_save=True)

Push a cloud run to your Hugging Face repo

For runs trained on Langtrain's GPUs, push the merged weights to a Hugging Face repository you own. The repo is private by default. Download it from there to run the model anywhere.

python

1curl -X POST https://api.langtrain.xyz/api/v1/training/jobs/$JOB_ID/export -H "X-API-Key: $LANGTRAIN_API_KEY" -H "Content-Type: application/json" -d '{"repo_id": "you/my-assistant"}'

Run it offline with llama.cpp

llama.cpp is an open-source runtime that works on a laptop or server with no internet connection. Convert the weights with its converter, then run the file.

python

1# In a llama.cpp checkout
2python convert_hf_to_gguf.py ./my-assistant --outfile my-assistant.gguf
3
4llama-cli -m my-assistant.gguf -p "Hello"

Serve it on your servers with vLLM

vLLM is an open-source server that exposes an OpenAI-compatible API, so apps written for the OpenAI SDK can point at your own server instead.

python

1pip install vllm
2vllm serve ./my-assistant
3
4# Then, from any OpenAI SDK client:
5# base_url = "http://your-server:8000/v1"

What is not available yet

Direct downloads from Langtrain (GGUF, ONNX or safetensors files) and a hosted endpoint for your fine-tuned model are not available yet. Use the Hugging Face export above to get your weights.

Train on your own GPU

Langtune trains on your machine, so your data is never uploaded. Save with merge_before_save=True to get full model weights in a folder, ready for the tools below.

python

1pip install "langtrain-ai[train]" langtune
2
3from langtune import FastLanguageModel
4
5model, tokenizer = FastLanguageModel.from_pretrained("meta-llama/Llama-3.1-8B")
6model = FastLanguageModel.get_peft_model(model, r=16, method="qlora")
7FastLanguageModel.train(model, tokenizer, dataset, output_dir="./output")
8
9# Merge the adapter into the base model and save standard weights
10FastLanguageModel.save_pretrained(model, tokenizer, "./my-assistant", merge_before_save=True)

Push a cloud run to your Hugging Face repo

For runs trained on Langtrain's GPUs, push the merged weights to a Hugging Face repository you own. The repo is private by default. Download it from there to run the model anywhere.

python

1curl -X POST https://api.langtrain.xyz/api/v1/training/jobs/$JOB_ID/export -H "X-API-Key: $LANGTRAIN_API_KEY" -H "Content-Type: application/json" -d '{"repo_id": "you/my-assistant"}'

Run it offline with llama.cpp

llama.cpp is an open-source runtime that works on a laptop or server with no internet connection. Convert the weights with its converter, then run the file.

python

1# In a llama.cpp checkout
2python convert_hf_to_gguf.py ./my-assistant --outfile my-assistant.gguf
3
4llama-cli -m my-assistant.gguf -p "Hello"