How to Prevent AI Agent Hallucinations

"The agent made something up" is not one bug. An operations agent that drafts a client update, files a ticket, or pulls a number for a report can invent a fact in five different places: how it reads the request, which tool it picks, what it remembers from last week, what a connected system actually returned, or how it phrases the answer once it has one. Each of those has a different fix. A prompt rewrite will not repair a misread field.

A hallucination, here, means the agent states something its own inputs do not support. A deal stage that never changed. A ticket that was never closed. A number pulled from memory instead of the CRM field. That is wider than a chatbot inventing a quote in one reply, because an agent also plans, picks tools, holds state, and sometimes hands work to another step. Each of those is a separate place to go wrong.

What does it mean when an agent hallucinates?

Treating "the agent made something up" as one bug is why so many fixes miss. A prompt rewrite can correct a misread goal and do nothing for a misread field, where the agent parses a tool response in the right shape and still misunderstands what the field means. Naming which of the five places produced a specific wrong answer comes before you pick a fix. If you skip that, you will edit the prompt, ship the same wrong number next week, and conclude the agent cannot be trusted.

A hallucination can start in five places, and the fixes are not the same.

A 2025 survey of agent hallucinations (arXiv:2509.18970) groups the failures into five types and eighteen triggering causes.

• Reasoning. The agent misreads the goal or builds the wrong plan, so "flag overdue invoices" becomes "flag all invoices."
• Execution. The agent picks the wrong tool, or calls the right tool with the wrong parameters because it misread the tool's documentation.
• Perception. The agent misreads what a connected system returned, such as a status field that was reopened an hour after the value it quoted.
• Memory. The agent restates last week's number because this week's update has not reached the context it is holding.
• Communication. One step passes a wrong fact to the next, and the error compounds instead of getting caught at the handoff.

An agent that reports a ticket closed while the support list still shows it open is usually a perception or memory failure. A perception failure needs a clearer field mapping, so the agent reads the current status rather than a cached label. A memory failure needs a shorter refresh on the context it pulls, so last week's close cannot outlive this week's reopen. Treating both as "the prompt was vague" is why the same wrong answer comes back. The eighteen causes inside the survey are the finer split of these five. You do not need all eighteen on day one. You need to know which of the five you are looking at before you change anything.

Why do agents fill the gap instead of stopping?

Kalai, Nachum, Vempala, and Zhang (OpenAI and Georgia Tech, arXiv:2509.04664, 2025) argue that standard training and evaluation reward a guess over an admission of uncertainty, the way a multiple-choice test rewards a guess over a blank. They show that the error rate of generating a specific fact is mathematically at least twice the error rate of simply classifying an answer as right or wrong, because producing a fact from scratch is harder than checking one you were given. The gap does not close just because the model gets larger. The incentive is in how most benchmarks are scored. A wrong answer and a blank answer were graded the same way, so the model learned to always answer. In production, those two outcomes do not cost the same.

That habit survives a model upgrade because it was not caused by one training run. It was caused by the score. If a blank and a wrong answer were worth the same points while the model was being evaluated, producing something was always the higher-scoring move. The model carries the habit into your workflow, where a wrong renewal date in a client update costs a relationship and a declined answer costs a review. Those are not the same cost, and the benchmark never had to price them that way.

Their scoring rule is the one to copy into the prompt and the tool contract.

Their example is a birthday. For an arbitrary fact like a person's birthday, there are 364 wrong answers for every correct one, so a model trained to always produce an answer will be wrong most of the time on facts it was never told. Their scoring rule: answer only above a stated confidence threshold, penalize a confident wrong answer, and score "I don't know" as zero rather than as a failure equal to a wrong answer. Build that rule into the prompt and the tool contract. It only works if a decline counts as a correct outcome in your review, not as the agent falling short.

Apply the same rule to the facts an operations agent is tempted to invent: a renewal date, a contract term, a number it was never given. The agent declines instead of guessing. The decline is a finished output, with the same standing as a drafted sentence, and the review treats it as correct when the source really did not contain the fact. If your review marks every decline as a miss the agent should have "tried harder" on, you have recreated the benchmark. The agent will learn, from the way you edit it, that a guess is safer than a stop.

How often does this happen when an agent uses tools?

ToolFailBench tested 19 models on real tool-use tasks and split the errors instead of blending them. "The agent is 90% accurate" hides which 10% is failing. The best model in that run, Grok-4.3, reached an 86.33% Clean Tool-Use Rate, which means about 1 in 7 tool-required tasks still failed at the top of the field. Output fabrication, invented structured data with no tool result behind it, stayed under 0.13% for the strongest models and reached 1.61% for the weakest, more than a 12 times spread on the failure most likely to land in a client report.

A blended accuracy number is the wrong dashboard for this. A model can be rarely fabricating and still skipping tools, or calling tools it does not need, and both of those will be inside the same "90%." ToolFailBench's split is useful because the fix depends on the mode. You do not tighten a prompt to cure a model that never called the tool. You do not add a tool to cure a model that called the right one and then ignored the result.

The errors cluster in four modes: skip, ignore, fabricate, or overreach.

The same benchmark separates four modes.

• Tool-skip. The agent never calls a tool it needed and answers from an assumption.
• Result-ignore. It calls the right tool, gets a real answer, and then writes something else.
• Output fabrication. It invents structured data with no tool result at all, which is the mode most likely to reach a client report without anyone noticing the missing source.
• Unnecessary tool use. It calls a tool on a question that needed a direct answer, adding a delay and a new place to fail.

The weakest models called tools when none were needed on nearly every control task. That is its own unreliability. The same kind of question does not behave the same way twice. You cannot write a runbook for an agent that sometimes reaches for a system and sometimes does not, on a question that never required the system. The cost is not only a wrong fact. The cost is that you cannot tell, from the outside, whether this run is the one that checked the CRM field or the one that improvised. A clean fabrication rate and an unstable tool habit are both reasons to keep the output in review.

Does grounding the agent in your own data reduce this?

Yes, and the size of the effect depends on how the grounding is built. A 2025 comparison (arXiv:2504.05324) measured three retrieval methods on the same hallucination-detection tasks. Sparse keyword retrieval produced a 12 to 22% hallucination rate. Dense semantic retrieval produced 10 to 20%. A hybrid of both, with query expansion, brought the rate to 4.00%, roughly a three to five times reduction against either method alone, on the same model and the same evaluation set.

Connecting the agent to the CRM and connecting it well are different setups. Both can be described as "grounded." Only one of them reliably finds the specific CRM entry a question is actually about. A connection that returns a nearby entry, or returns nothing and lets the model continue, is still a place for a guess. The rates above are the size of that difference on one evaluation set, not a promise about your CRM.

Hybrid retrieval beat either method alone because each misses what the other finds.

Keyword search finds the entry with the exact term. Semantic search finds the entry that means the same thing in different words. An agent grounded in only one misses whatever the other would have caught, and a missed entry is the gap a model fills with a guess instead of "I could not find that." The same principle applies to a weekly report: ground it in a specific CRM field and a ticket status, not in "summarize what happened this week." The narrower and better indexed the source, the less room the model has to fill.

Query expansion is the third piece of the hybrid result, not a decoration. A person asks about "the renewal," and the CRM field is named "contract end date." Keyword search on the person's words misses. Semantic search may still miss if the index was built on a different vocabulary. Expansion rewrites the question toward the words the source actually uses, and then both retrieval methods run. If you only turn on one method because it is the one your vendor ships, you should expect to land nearer the 10 to 22% band than the 4.00% result. Say that out loud when someone proposes "we will just connect the CRM" as the whole plan.

What three guardrails should you build first?

First, ground the agent in a narrow, verified source, and measure the error rate before you trust it. The baseline belongs in how you measure agent performance. A source that returns the wrong entry sometimes is still better than no source, but only if you know the failure rate instead of inferring it from how confident the prose sounds.

Second, give the agent an explicit decline. It should be able to say "needs review" instead of stating a fact it cannot point to. Left unstated, the default is to attempt an answer. A decline option it was never told about is not one it will use. Write the decline into the prompt and into the tool contract on purpose, using the same logic as the confidence threshold above. "Needs review" has to be a real output, not a sentence you hope the model will invent when it feels unsure.

Third, spot-check a sample every week. A rare failure can look near zero for weeks and still be real. A five-output sample will not catch every rare mode. A standing weekly check is what catches it before a client does. The failure you are hunting may be the 0.13% fabrication mode, which will not show up in every batch, or the tool-skip mode, which shows up as a fluent answer with no tool call behind it. A week with five clean outputs is not a retirement of the check.

Five outputs a week is a check you can run before a client sees the draft.

A check you can run this week: pick five outputs at random from the last batch. For each one, trace every factual claim to the deal ID, the ticket number, or the field value it should cite. Log how many of the five hold up, and note which of the five failure types the miss belongs to. A skipped tool and a fabricated output need different fixes. Below four of five clean, stop trusting that agent unattended and go back to the source, not the wording. The same sample scales once you know it works on a small batch, which is covered in testing an agent before it ships.

Write the log as five rows, not as a vibe. The row names the output, the claim, the CRM entry or the ticket or the field you expected to find, whether you found it, and which of the five types the miss belongs to if you did not. A row that says "sounded right" is not a row. A row that says "quoted a close date, no field contains that date, fabrication" tells you the next edit. Four of five clean is the bar for leaving the agent unattended on that task. Three of five clean is a reason to pull it back to review, even if the writing is the best it has been.

When should an agent say it does not know?

Any time it cannot point to the specific CRM entry, ticket, or field behind a claim. That sounds obvious until you write the prompt, because the default is to answer. Give "insufficient data" the same weight as any other output, and treat it as a correct response in review. An agent that never declines is not more capable. It is guessing more often, and you cannot tell which answers are which.

The refusal has to be built, not assumed. A model asked a question will attempt an answer unless you have given it an output called "insufficient data" and told it that output is allowed. In review, do not send that output back with a note that says "try again." Send it back only when you can point at the CRM entry, the ticket, or the field the agent should have used. If you cannot point at one either, the decline was the correct result, and the next step is to put the fact in the source, not to lean on the model.

If every output reads equally confident, that confidence is not a signal. A grounded agent should occasionally decline, flag a gap, or ask for a missing input. An unbroken run of confident answers usually means the decline path was never wired in. Watch for that tell in the weekly sample. Five confident paragraphs, none of which cite a field, are a worse sign than four confident paragraphs and one "needs review."

What does this mean for the agent you are building this week?

Start with the source, not the prompt. Pick the one system the first task depends on, ground the agent there with hybrid retrieval if the stack supports it, and add the decline before the first output goes out unreviewed. Run the five-line check for the first month. An agent that cannot clear it is not ready for an unattended report, however clean the writing is.

None of the three guardrails require a bigger model. A narrower source, an explicit decline, and a weekly sample are cheaper than most of what gets tried first, and they aim at the failure points above. The weekly owner report and the owner brief are where a decline should show up so it is visible without opening the logs. Bring the task and the baseline you want to measure. Get your free Systems Plan, and build the guardrails into the first version.

How we know this.

Kalai, Nachum, Vempala, and Zhang, 2025 (arXiv:2509.04664), supply the training incentive, the generative-versus-classification bound, and the confidence-threshold rule. The 2025 survey (arXiv:2509.18970) supplies the five types and eighteen causes. ToolFailBench (arXiv:2607.04686), 19 models, supplies the 86.33% Clean Tool-Use Rate and the fabrication figures. The hybrid-retrieval comparison (arXiv:2504.05324) supplies the 12 to 22%, 10 to 20%, and 4.00% rates. The five-line check is a method you can run, not a client result.