Research
The techniquesbehind the libraries.
What our training code does beyond standard LoRA, described by how it works. The code is open: read it in langtrain-ai on GitHub.
In langtrain-ai
Three techniques, all in the open.
AdaptiveRank
langtrain.training.adaptive_rank
A LoRA trainer that starts adapters from the base weights' top singular vectors (SpectralLoRA) instead of random noise, and batches similar examples together. It runs on your own GPU.
- SpectralLoRA initialisation from an SVD of the base weights
- Micro-batches clustered by content (TF-IDF k-means)
- 4-bit loading by default, like QLoRA
- In progress: growing or shrinking each layer's rank as it trains
TurboQuant KV
langtrain.training.kv_quant
Compression of the attention key/value cache at inference time, based on Google Research's TurboQuant: 3-bit polar quantisation of KV heads with a 1-bit Johnson-Lindenstrauss residual correction.
- PolarQuant: 3-bit polar-coordinate quantisation
- QJL: 1-bit residual correction
- Applied at inference; no retraining
- Suggested for long sequences (over 2,048 tokens)
DatasetIntelligence
langtrain.intelligence
Looks at a dataset before training and reports the kind of task, its size and domain, and suggests a base model and training setup.
- Runs on your machine
- Detects the task type and domain
- Suggests a model, method and rank
- Printable summary before you train
Try them onyour own data.
pip install "langtrain-ai[train]"
AdaptiveRank and DatasetIntelligence ship in the Python SDK.