Local fine-tuning, LoRA adapters, quantization, and Ollama

Ollama is primarily an inference and model-packaging runtime. Train adapters with tools such as Hugging Face, Unsloth, or MLX, then import compatible Safetensors or GGUF artifacts.

Adapter workflow

select a licensed base model and exact revision;
create clean train/validation/test datasets;
train LoRA/QLoRA with recorded hyperparameters and seeds;
evaluate the adapter and base model independently;
export a supported adapter format;
import with the same base model;
optionally quantize and re-evaluate;
package provenance, license, model card, and rollback artifact.

FROM <exact-base-model>
ADAPTER /models/claim-adapter
PARAMETER num_ctx 8192
SYSTEM You extract claim facts into the supplied schema.

ollama create claims-extractor -f Modelfile
ollama run claims-extractor

The base in FROM must match training; otherwise output may be erratic.

Quantization

Quantization reduces memory and may improve speed at a quality cost. Benchmark each quantization on the real task, long context, tools, and JSON conformance. Do not infer quality from file size or a generic leaderboard.

Reproducibility

Record base hash, dataset hash, code/container, library versions, GPU type, precision, epochs, learning rate, rank/alpha, prompt template, and evaluation results. Scan model artifacts like other supply-chain dependencies.

Serving

Pin the packaged model, pre-pull it, warm deliberately, bound concurrency, and monitor load time, tokens/second, queue time, memory, context truncation, and fallbacks.

Feynman check

LoRA is a small removable lesson attached to a large textbook. Quantization prints the textbook smaller; some fine detail may disappear.