1.2 AI Failure Modes: Hallucination, Stale Knowledge, Confabulation

AI errors are not slip-ups that will disappear with the next model generation, they are a structural property of probabilistic systems. If you can tell the three central failure modes apart, you will spot them faster in daily work and know which countermeasure applies to each.

Why errors are structural, not bugs

(Contributed by Prof. Inwah Kim)

Unlike traditional software that operates on deterministic logic and explicit code, Large Language Models (LLMs) function on a probabilistic architecture. They do not retrieve verified facts from a relational database, nor do they possess human-like reasoning. Instead, LLMs generate text dynamically by predicting the most statistically probable next token based on pattern recognition from massive training datasets. Consequently, AI errors are structural — inherent to the core mathematical design of the system — rather than traditional programming bugs or software glitches.

Four Pillars of Structural Vulnerability

• Probability vs. Veracity — The model optimizes for statistical likelihood, not objective correctness. While the most probable continuation of a sentence often aligns with reality, it offers no mathematical guarantee of accuracy.
• Absence of Internal Verification — LLMs lack an intrinsic mechanism to cross-reference statements against external databases or to self-audit their outputs. Without integration into external retrieval-augmented generation (RAG) tools, the model cannot differentiate between a plausible-sounding fabrication and empirical truth.
• Statistical Patterning vs. Knowledge Storage — During the training phase, the architecture maps statistical relationships between semantic concepts rather than indexing a structured knowledge base. The output is a fluid reconstruction of observed linguistic patterns, not a retrieval of stored factual records.
• Optimization for Linguistic Fluency — The underlying training objective rewards coherence, syntax, and stylistic naturalness. Because confidence and articulateness are merely products of this fluency optimization, they function as deceptive indicators that frequently mask factual inaccuracies.

Key Takeaway for Practitioners: Because these errors are structural, they cannot be completely "patched" out of existence like traditional software bugs. Managing AI failure modes requires systemic guardrails, such as external retrieval tools and human-in-the-loop validation.

The three failure modes defined

(Contributed by Prof. Inwah Kim)

In Short:

• Hallucination breaks reality by creating fake facts.
• Stale Knowledge breaks time by using expired facts.
• Confabulation breaks context by injecting unfounded assumptions.

Where each failure mode comes from

(Contributed by Prof. Inwah Kim)

1. Hallucination
   • The Cause: Plausibility Optimization + Tokenization
   • The AI is trained to sound natural and fluent, not necessarily to tell the truth (plausibility optimization). Because it builds words piece-by-piece using math codes (tokenization), it sometimes pieces together real-sounding tokens to invent fake facts, fake URLs, or fake book titles. It cares that the sentence looksgood, not that it is real.
2. Stale Knowledge
   • The Cause: Training Data & Knowledge Cutoff
   • This is a timeline problem. Once an AI finishes its training, its memory is frozen at that exact date (knowledge cutoff). It has no live internet connection to see today's world. If a company gets a new CEO or a law changes after that cutoff date, the AI will confidently give you the old information.
3. Confabulation
   • The Cause: Plausibility Optimization + Missing Context
   • This happens when your prompt has missing information or gaps. Because the AI hates leaving a job half-done (plausibility optimization), it hates saying "I don't know." Instead, it uses its statistical guessing power to fill in the blanks, making up logical-sounding details or motives to give you a complete story.

In Short:

• Stale Knowledge happens because the AI's clock stopped.
• Hallucination happens because the AI values smooth writing over real facts.
• Confabulation happens because the AI makes wild guesses to fill in your missing details.

Real-world cases and their implications

Three documented cases show the range. In the US proceedings Mata v. Avianca (2023), a lawyer filed a brief containing six precedents invented by ChatGPT — complete with invented case numbers. The court imposed sanctions; the case became the textbook example of why legal citations from LLMs are worthless without verification. In 2024, a Canadian tribunal ordered Air Canada to pay damages because the airline's chatbot had promised a customer a refund policy that did not exist — the company was held liable for its bot's statement (Moffatt v. Air Canada, 2024). And in 2025, the consultancy Deloitte had to partially refund fees to the Australian government after a report for the Australian Department of Employment was found to contain AI-generated, non-existent references — including a fabricated court quotation.

For student life, the second failure mode is often the most relevant: ask for "current figures on banking regulation" for a term paper and you may receive the state of two years ago — correctly phrased, confidently presented, outdated. And anyone who has references generated risks the Avianca mistake in miniature: sources that never existed, or real authors with invented titles.

Dealing with hallucination through output validation

(Contributed by Prof. Inwah Kim)

Since hallucination is a structural property of how LLMs generate text, we cannot eliminate it at the model level alone. Instead, we manage it through how we handle, verify, and constrain the output after it is generated.

1. Require sources and grounding — Ask the model to cite where a claim comes from, then independently confirm that the source actually exists and says what the model claims.
2. External fact-checking — Cross-check factual claims — especially names, dates, statistics, and citations — against authoritative external sources rather than accepting the AI's output at face value.
3. Retrieval-Augmented Generation (RAG) — Connect the model to real documents or live search so it generates answers grounded in retrieved evidence, rather than relying purely on internal, unverifiable pattern completion.
4. Cross-verification — Rephrase the question or query multiple models/approaches. Inconsistent answers across attempts are a warning sign of hallucination.
5. Prompting for calibrated uncertainty — Explicitly instruct the model to say "I don't know" when evidence is insufficient, rather than defaulting to a confident-sounding guess.
6. Requesting specificity — Ask for precise, checkable details (exact quotes, page numbers, links) rather than vague summaries — specific claims are easier to verify or falsify than general ones.
7. Human-in-the-loop review — In high-stakes domains (law, medicine, finance, academia), a human must review and validate AI output before it is treated as final.

Key Takeaway for Practitioners: AI output should be treated as a draft requiring verification, not a final answer.No matter how fluent or confident the model's response sounds, fluency is not evidence of truth. This is the single most important mindset for managing hallucination at the output stage — the burden of verification always remains with the human user.

A peek into the research

Research on these problems runs on three fronts, and two of them have already appeared above: retrieval-augmented generation couples the model to an external, current knowledge base (Lewis et al., 2020), and grounding approaches oblige models to tie claims to retrievable evidence — both reduce hallucination and stale knowledge, neither eliminates them. The third front makes progress measurable in the first place: benchmarks such as TruthfulQA (Lin et al., 2022) systematically test how prone models are to reproducing popular falsehoods — and thereby turn "the model got better" from a marketing claim into a checkable statement.

[CONCLUSION] The question is never whether a model makes errors, but which error class is likely right now — and whether your verification process catches it. Chapter 1.3 introduces the second line of defense: prompts that make errors less likely in the first place.