Why AI Fails Without Anyone Noticing (and How to Catch It)

What happened

January 2024. A man asks a delivery company’s chatbot about a missing parcel. The bot can’t find it, can’t connect him to a human, can’t give him a phone number. So he starts poking at it. A few messages later the chatbot is swearing, and then it writes him a poem calling its own company the worst delivery firm in the world.

To the servers running that chatbot, every one of those replies was a success. Fast, healthy, status code all good. The system was up. The answer was wrong.

That is the whole problem in one story. Traditional software fails loudly. If the database is down, the chain snaps, the server throws the famous 500, a red light flashes and an engineer’s phone buzzes. A language model doesn’t work like that. It predicts the next word, then the next, based on probabilities. Ask it the same question twice and you get two different answers. When it goes wrong, nothing snaps. The server sends back 200 OK, which only means “I got your request, here’s a reply.” It’s a handshake. It says nothing about whether the reply is true.

Think of a patient in an emergency room with perfect vitals who doesn’t know what year it is. If you only read the heart monitor, you send them home.

Then there’s Jake Moffatt. In November 2022 his grandmother died, and Air Canada’s website chatbot told him to book at full price and claim the bereavement discount within 90 days. The real policy said no refunds after travel. Air Canada argued the chatbot was a separate legal entity responsible for its own actions. The tribunal said no: the chatbot is part of your website, and you’re responsible for your website. Air Canada was ordered to pay 812 Canadian dollars. Small money, big precedent. “The AI said it” is not a defence.

Where the control failed

The control that failed is the dashboard itself. It watches the machine, not the answer. AI observability is the discipline built to close that gap: continuously checking and recording what an AI actually did, live. What it was asked, what it looked at, which steps it took, what it cost, and whether the answer was any good. The old gauges tell you the plane came down. Observability is the flight recorder that tells you why the pilot flew into the storm.

Two things made this urgent. First, scale: according to Stanford’s AI Index, 88% of organisations used AI in 2025, and documented AI incidents rose from 233 to 362. Real users try to break your bot on day one. A Chevrolet dealership chatbot agreed to sell a 2024 Tahoe for one dollar, and the dealership pulled the bot.

Second, agents. An agent makes a plan, uses tools and loops until it thinks it’s done. One sentence from you, seven steps by the machine. When the final report says your biggest competitor is a bakery in Paris, old monitoring only knows the last step returned 200 OK. Observability records each step as a span and the whole journey as a trace, so you can find the exact step where the logic broke.

Then comes the part that surprised me most: who grades the answers? A bank’s chatbot might handle a hundred thousand conversations a day, so the industry uses AI to grade AI. The research says these judges have very human flaws. Swap the order of two answers and, in the MT-Bench study, GPT-4 kept the same verdict only about 65% of the time. Two older judges preferred padded answers over 90% of the time. GPT-4 shows a significant preference for text that looks like its own writing. And on JudgeBench, with hard questions that have objectively right answers, GPT-4o with a basic prompt picked correctly about half the time. A coin flip.

The field moves fast. With the same judging prompt, GPT-4o gets about 57% and the best reasoning judge, o3-mini, about 81%. Better, but not solved: it still gets about one hard case in five wrong. The grader still needs grading.

The evidence an auditor should ask for

In audit there’s a basic test called vouching. You take something recorded and trace it back to its source document. If you can’t find the source, the number doesn’t get signed off. AI observability is vouching for AI: every answer traced back to what it was based on.

So I vouched the research for this video. I spent about two days with Gemini Notebook doing deep research. It was good: fluent, confident, mostly right. Not all of it vouched.

1. Exhibit one: the $15,000 runaway agent that appears in almost every write-up. I traced it to a single practitioner blog. No company, no date, nothing I could check. It might be true. I can’t sign it off.
2. Exhibit two: two detailed case studies, a $45,000 weekend loop and a fintech firm whose AI judge picked the wrong model. No sources at all, and they read exactly like real case studies.
3. Exhibit three: “AI judges prefer longer answers over 90% of the time.” True for two older judges, not for the latest models. The number survived; the context didn’t.

My research pipeline returned 200 OK. Everything looked green, and some answers were wrong. That’s a verification problem.

The evidence matters more now that the law is no longer waiting. The EU AI Act’s transparency rules started applying in August 2026, and for high-risk systems like hiring and credit scoring, automatic event logging and post-market monitoring apply from December 2027. “We don’t know, it’s a black box” is the wrong answer to a regulator. One trade-off to plan for: masking names and numbers in logs protects the customer, but may also mask why the AI failed.

The test to run this week

If you work in risk, compliance or audit, or you’re just the person who’ll get blamed, take three questions to your AI team:

1. Pick any answer your AI gave a customer last month. Can you trace it to the exact document it came from?
2. Who graded the grader? Show me how your AI judge was tested for position and length bias.
3. Where’s the kill switch? What stops a looping agent, at what spend, and who gets the call?

If they can answer all three, you have AI observability. If they show you a green dashboard, you have a dashboard.

I’ve put the cases, the signals and every number with its source into a free field guide: AI Fails Silently: The AI Observability Field Guide.