← All articles
Governance • Field note

Your AI agent's explanation is not a record of what it did

Researchers keep finding that the step-by-step reasoning AI models write out has little to do with how they reached the answer. If your agent's explanation is your audit trail, what you have is a story.

David Soden • 5 min read • 11 November 2026
A hand holding a magnifying glass over printed financial statements on a wooden desk
An audit reads the documents. It doesn't take the summary on trust.

What happened

Quanta Magazine ran a long piece by John Pavlus asking whether AI reasoning is right for the wrong reasons. The models are getting real results. They won gold at the 2025 International Mathematical Olympiad, and one solved a famous open math problem this year. The question is what the "thinking" they print along the way has to do with any of it.

So far, less than it looks. Subbarao Kambhampati's lab at Arizona State swapped correct reasoning traces for wrong or irrelevant ones, and performance on a formal reasoning task didn't drop. A Northeastern and UC Berkeley study found that 30 to 60 percent of the thinking steps in open-source reasoning models had minimal effect on the answer to benchmark math questions, and removing half of them barely changed the score. Melanie Mitchell of the Santa Fe Institute put the stakes plainly: "You want the right answer for the right reason, so you can trust these things."

The labs dispute some of this. Sébastien Bubeck of OpenAI told Quanta the older critiques reflected quirks of obsolete models. But the article also notes that OpenAI, Google DeepMind and Anthropic publish rewritten summaries of the reasoning rather than the raw trace, so outsiders can't settle it either way.

Why this matters to your business

Plenty of support teams already treat the agent's explanation as the justification. An agent denies a warranty claim and writes "The purchase date is outside the 12-month window, so this claim is not eligible." A supervisor reads it, it sounds right, they move on. Three weeks later the customer disputes it and someone pastes that sentence into the reply.

If the research holds, that sentence may be a plausible account written next to the decision rather than the cause of it. The agent could have the policy right and the date wrong. It could have reached the right answer for a reason it never stated. Nobody can tell from the prose, and the prose is all you kept.

A man holding a printed page talks with a support agent wearing a headset at her desk
A reviewer who only sees the agent's explanation is grading how well it writes.

That gap gets expensive when someone outside the team asks why. A chargeback analyst, a regulator or a lawyer wants to know which record the decision was based on, what that record said, which rule applied and who approved it. "Here is what the model said" answers none of those. The same problem shows up in quality review. If your QA team scores agents by reading their reasoning, a well-written wrong answer passes and a terse right one gets flagged.

Why this is a CX-Builder use case

The fix is to move the "why" out of the model's prose and into things the system records as facts. The agent still reads the ticket and decides. But it looks up the order through a tool, and the tool call and the response get logged. It reaches the warranty policy through retrieval, and the passage it used comes back with a document reference. Its decision comes out in a fixed shape: the outcome plus the policy clause and record fields it relied on, such as purchase date and order value.

Then a plain rule, with no model involved, checks the claim against the evidence. Does the purchase date in the record the agent actually pulled fall outside the window? If the decision and the data disagree, the case goes to a person. Denials can go to a person regardless, with the record and the cited clause on screen instead of a paragraph of reasoning. The explanation the customer sees is written from the checked fields, so it can't say something the record doesn't support.

A row of blue office binders on a shelf, each labeled with dates and names
The record is what the system looked up and what a person signed, kept where you can find it.

What this looks like if you build it

It's one agentflow. A tool node calls the order system and a retriever searches the policy documents, returning source references with each passage. Next, a structured output step forces the decision into a schema with the cited fields. The condition node after it compares those fields to the tool response, and a human input node holds mismatches and denials for review. The execution trace keeps every tool call and response, which is the part an auditor will ask for.

For testing, run a fixed set of past cases with known correct outcomes through the flow and score the decision and the cited fields. Ignore how convincing the reasoning reads. That test keeps working whichever model you swap in next quarter.

The takeaway

Pull ten denied or escalated cases from last month where an AI agent was involved. For each one, try to answer from your logs alone, without reading the model's explanation: what record did it look at, what did that record say, which rule applied, and who signed off. Every question you can't answer is a place where the explanation is standing in for evidence. Close those gaps before the next audit, or before the agent handles anything above your refund limit.

All articles Install CX-Builder View on GitHub