Tokens, transformers, context, and uncertainty
A language model receives tokens and predicts a probability distribution for the next token. Repeating that operation produces text, JSON, tool calls, code, or multimodal descriptions. It does not query a hidden database of guaranteed facts.
Tokens and context
Tokens are pieces of text, not words. Input instructions, retrieved documents, tool schemas, history, images, and output all consume context and money. Context capacity is not memory quality: filling a large window can reduce signal, increase latency, and expose unnecessary data.
Parameters and inference
Model parameters contain learned statistical patterns. Inference uses fixed weights to produce an answer. Fine-tuning changes some weights. RAG changes the input context without changing weights. Tool calling lets the model request deterministic work from software.
Sampling
Sampling settings alter output diversity but do not repair missing evidence. Current provider families increasingly manage sampling and reasoning internally. Do not copy temperature defaults between providers. Establish a task-specific baseline and evaluate it.
Hallucination
“Hallucination” means unsupported or fabricated content. Reduce it with:
- authoritative retrieval and citations;
- constrained structured output;
- tools for current facts and calculations;
- explicit abstention when evidence is insufficient;
- claim-level groundedness tests;
- authorization and deterministic validation outside the model.
Reasoning
More reasoning effort can improve difficult tasks while raising latency and cost. It cannot make an impossible or under-specified task valid. Route by task: cheap models for bounded extraction; balanced models for most interactions; flagship reasoning for complex, high-value decisions with strong evaluation.
Feynman check
The model is a brilliant sentence completer with learned patterns. Explain why that can create both an excellent analyst and a confident mistake.