Why LLMs Hallucinate
LLMs are next-token prediction machines. They are trained to produce statistically likely text, not verified facts. When the model encounters a query outside its confident knowledge range, it generates plausible-sounding text based on patterns — even if the specific facts are invented. The model has no internal fact-checking mechanism.
Hallucinations are more common with: rare topics (few training examples), specific numbers (dates, statistics, prices), citations (paper titles, URLs), and technical details (API signatures, function names). The model fills knowledge gaps with pattern-matching rather than admitting uncertainty.
Reducing Hallucinations
Four practical strategies work in production:
RAG (Retrieval Augmented Generation) grounds the model in your actual documents. Instead of relying on training knowledge, the model cites retrieved text. This is the most effective approach for domain-specific applications.
Temperature reduction makes the model more conservative. Setting temperature to 0.0-0.3 reduces creative variation and makes the model stick closer to high-confidence predictions.
Explicit uncertainty instructions in the system prompt ("Say 'I don't know' if you're not confident") help. Claude and GPT-4 both respond well to explicit instructions to flag uncertainty rather than guessing.
Verification pipelines use a second model or rule-based system to fact-check the first model's output. Expensive but effective for high-stakes applications (medical, legal, financial).