Why AI Confidently Makes Things Up: From Root Causes to Practical Engineering Strategies

Anyone who’s been confidently burned by an AI knows that moment of total disbelief — it cites an API that doesn’t exist, makes up a reference that looks completely legit, and says it like it’s the most natural thing in the world.

If you think this is just the model “not being smart enough,” you’re underestimating the problem. LLM hallucination isn’t some intelligence gap or engineering bug — it’s a systemic risk the architecture carries by design. You can’t fix it by tuning model parameters; you need system-level guardrails to set the bottom line. Here, we’ll break down how hallucinations actually happen, then look at how guardrails can minimize the damage — and finally, I’ll use two real-world war stories to show why AOSP actually isn’t fazed by this side effect and why it’s a safer environment for AI agents.

1. The Architecture: It Optimizes for Probability, Not Truth

At its core, an LLM does next-token prediction — it predicts the next word. What it learns from massive text corpora is the statistical patterns of human language, not a lookup table of facts.

This creates a fundamental problem: high-frequency patterns like spelling and grammar improve reliably as models scale, but low-frequency, arbitrary facts — like someone’s birthday — can’t be derived from language patterns alone. OpenAI’s 2025 paper Why Language Models Hallucinate is blunt about this: when the model hits a knowledge gap, it sacrifices accuracy to keep the sentence fluent, picking the most “contextually plausible” token to fill the slot. Its knowledge isn’t stored like a SQL database — it’s distributed across billions of neural weights. When you ask a question, it doesn’t look anything up. It reconstructs a probabilistic shadow of a plausible answer through fuzzy association.

The Evaluation Pipeline Rewards Guessing

There’s a more critical reason the model won’t say “I don’t know”: the training and evaluation pipeline itself punishes honesty.

Think of a multiple-choice exam that only counts right vs. wrong — guessing right earns points, leaving blanks earns zero. The rational strategy is obvious: guess even when you don’t know. LLMs are trained exactly this way. When scoring only measures correctness, the model learns to fabricate rather than admit uncertainty. OpenAI’s SimpleQA benchmark illustrates this well: an older model that almost never abstained (o4-mini) had a 75% error rate; a model willing to say “I don’t know” on roughly half the questions dropped to 26%. The difference isn’t intelligence — it’s whether the model is allowed to acknowledge its own ignorance (source: OpenAI SimpleQA benchmark; see also Nature 2026).

Training Side Effects: Long-Tail Loss and the People-Pleaser Personality

To compress massive datasets into finite parameters, models automatically trade off: they memorize high-frequency general patterns and drop low-frequency long-tail details. When you push for an obscure specific fact, the model simply doesn’t have that memory — it fills in a fake answer from pattern extrapolation.

Another side effect comes from alignment training (RLHF). Anthropic’s research Towards Understanding Sycophancy in Language Models found that reward models and human annotators both tend to score “compliant, well-formatted, plausible-sounding” answers higher — while an honest “I don’t know” actually gets penalized. This effectively gives hallucination a green light, breeding a people-pleaser personality: the model learns to satisfy you rather than be honest with you.

2. Error Amplification: One Wrong Step, Then Doubling Down

An LLM’s decoding is an autoregressive chain that can’t stop and restart: each step takes its own previous output as input. This creates exposure bias — if the first step drifts even slightly, errors compound like dominoes. That’s why hallucinations tend to cluster in the second half of long outputs — you’ve probably heard this called the snowball effect, where errors just keep piling up.

Zhang and Press’s ICML 2024 paper How Language Model Hallucinations Can Snowball revealed a counterintuitive phenomenon: models will actively generate more wrong supporting arguments to stay “consistent” with their initial incorrect answer. But here’s the thing — when those errors are isolated and asked about separately, ChatGPT recognizes 67% of its own mistakes, and GPT-4 catches 87%. The model isn’t lacking knowledge. It’s held hostage by its own prior context, trapped in a hole it dug itself.

Attention Drift: More Context Isn’t Always Better

There’s another amplifying factor. Stanford’s Lost in the Middle found that when critical information is buried in the middle of a long context window, model accuracy drops by over 20 percentage points compared to placing it at the beginning or end. In scenarios with 20-30 documents stuffed into context, performance was actually worse than providing no documents at all.

More context isn’t always better. Stuff in too much, and the model actually loses the signal.

3. Engineering Countermeasures

After all of the above, the conclusion should be clear: hallucination can’t be solved by trusting the model to self-regulate. It takes deliberately designed system architecture to prevent or correct the problem. The industry has converged on three approaches, all sharing the same principle — don’t let the model brute-force it on weights alone.

Semantic Entropy: Why Logits Aren’t Enough

The third approach deserves a closer look because there’s a common misconception. Many assume you can use a single token’s output probability (logit) as a confidence score and block anything below a threshold. But single-token logit calibration is poor — models are frequently “very confident and very wrong.” Logits alone can’t catch hallucinations.

The more robust approach in current research is Semantic Entropy: sample the same question multiple times and cluster the answers by meaning. If the model says something different each time (semantic divergence), it’s guessing, and the system should raise a red flag. This method was published in Nature and significantly outperforms baselines at detecting hallucination, at the cost of roughly 10x compute.

However, probabilistic detection has an inherent ceiling. What I want to talk about next is a harder-edged approach — deterministic verification.

4. An Engineer’s Perspective: AOSP Is a Built-In Hallucination Safety Net

As someone who has worked on AOSP / Android framework for years, the more I study these mechanisms, the more familiar they feel: the three countermeasures above already have direct counterparts in AOSP’s workflow.

The first line of defense is the RAG counterpart: make the AI agent work against the real source tree — look up APIs by reading the actual code, not guessing from training data. But grounding alone isn’t enough. The two cases below show how the other two defenses catch the two types of hallucination AI agents are most prone to.

Case 1: Copying a Pattern, Forgetting the Scope — Caught by Code Review

In a HAL-layer authentication polling service, the outer loop used std::unique_lock + wait_for for periodic status checks. The inner loop needed the same sleep functionality, so someone copied the outer loop’s pattern verbatim:

while (!exit) {
    std::unique_lock<std::mutex> autolock(mMutex);  // LOCKED
    mCv.wait_for(autolock, 200ms);
    // when wait_for returns, mutex is re-acquired

    if (condition) {
        for (int i = 0; i < N; i++) {
            std::unique_lock<std::mutex> slock(mMutex);  // DEADLOCK
            // autolock still holds mMutex
            // slock tries to lock the same non-recursive mutex
            // → waits for itself to unlock → waits forever
            mCv.wait_for(slock, 200ms);
        }
    }
}

std::mutex is non-recursive. The outer autolock is still holding the mutex when the inner slock tries to lock the same one — deterministic deadlock. The program silently hangs with zero error output.

This bug was caught in the first round of Gerrit code review.

Tell an AI agent to “add a periodic check to the inner loop,” and the first thing it does is find the closest pattern in context and copy it — that’s literally how next-token prediction works. The code looks “symmetric” and reads “reasonably,” but the semantics are wrong. Gerrit review, acting as a human checkpoint, cut the snowball off before the code ever ran — the AOSP version of a Chain of Thought defense.

Case 2: Updated One Side, Forgot the Other — Caught by Boot-Time Validation

During an update to a system-level privileged app, an engineer modified the privapp-permissions allowlist (removed a permission) but forgot that the prebuilt APK’s manifest still declared it:

privapp-permissions.xml:  removed WRITE_SECURE_SETTINGS  ← updated
prebuilt APK manifest:    still has WRITE_SECURE_SETTINGS ← not synced
→ mismatch → IllegalStateException → boot loop

Android’s PermissionManagerService performs a strict comparison of these two lists at systemReady(). A mismatch triggers an IllegalStateException — the system won’t boot. Not a warning, not a degradation — a hard block.

When an AI agent modifies permissions, its context window might only contain the allowlist XML — not the prebuilt APK manifest sitting in a different directory. This is what a knowledge gap looks like in practice: the model doesn’t have the complete state. But PermissionManagerService doesn’t care what you know or don’t know — inconsistent means no boot. Semantic entropy can only estimate the model “might be guessing.” Deterministic verification tells you straight: you’re wrong. Harder than any logit threshold.

Three Defenses, All Essential

With these three layers, AI agents can safely handle the work that genuinely should be automated — boilerplate, HAL templates, test generation, log analysis, large-scale refactoring migrations. Even when the agent gets it wrong, the system catches it.

My principle in one line: don’t let AI guess — make it tell the truth in front of the compiler and the test suite.

A Word of Honesty

I need to add a boundary so this post doesn’t read too optimistic.

Most of the research cited above comes from OpenAI, Anthropic, and top conferences — sources that carry an inherent narrative bias toward “hallucination is manageable and engineering can solve it.” Engineering measures can dramatically reduce hallucination, but they can’t eliminate it entirely.

So the real moat isn’t “trusting that AI won’t make mistakes.” It’s designing a system that catches mistakes when they happen. Every commit we make in AOSP follows the same logic — nobody assumes their code is bug-free, but with the compiler, CI, CTS, and code review all in place, you can enjoy the benefits AI agents bring with a lot more confidence.

References

• OpenAI, Why Language Models Hallucinate (2025); Nature version (2026)
• Zhang & Press, How Language Model Hallucinations Can Snowball, ICML 2024
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts, 2023
• Anthropic, Towards Understanding Sycophancy in Language Models, 2023
• Farquhar et al., Detecting hallucinations in large language models using semantic entropy, Nature, 2024