What happened
An AI agent was given a hard coding problem five times and failed all five. Then it got the exact same problem again, same instructions, same words: five out of five. No new training, no better prompt. Just one setting: effort.
I’ve spent over 13 years in audit, risk and data analytics. In audit we always ask: how much testing is enough? Too little and you miss the problem. Too much and you burn a week on something that needed an hour. AI has the same question built in, and it’s called effort.
Anthropic engineer Thariq Shihipar published a deep dive on the Claude Code blog, testing effort levels on Opus 5.5 and Fable 5.1.
Think of effort like a time budget from your boss. Told to take 12 hours, you double-check, explore options and polish. Told you have one hour, you hand over a good first version. Effort tells the AI roughly how much thinking you want it to spend. The AI always tries to do the task properly, but at higher effort it checks its own work more, tests more edge cases and makes more decisions on its own. Hold on to that last part.
His first test was simple. He asked Claude to build a personal fitness and workout tracker app, with no details. Low effort took one and a half minutes and produced a workout log and a simple graph. Medium took four minutes, high took 11, and max took 67 minutes. The max version was far more detailed and even had a heat map.
So max effort wins? Not so fast. Every extra minute, the AI was making choices for you: features, design, what a fitness tracker even means to you. If you wanted a quick base to shape yourself, the 90 second version might be the better result.
Then he did something smarter. He had Claude interview him about the app, which produced a proper written spec, then gave that spec to Claude at every effort level. The results were much more similar: similar designs, similar code, small differences. At max effort, Claude mostly spent its extra time simplifying things. A clear spec reduces the guessing that effort would otherwise fill in. In audit terms, a clear audit program matters more than how many hours you throw at fieldwork.
Where it held, and where it failed
For problems where the AI either succeeds or completely fails, Thariq used Terminal-Bench 3.0, a set of seriously hard tasks like building an 8-bit game console in hardware code or writing a mathematical proof.
One task asks the AI to build an HTML sanitizer: a filter that blocks every way a hacker could sneak malicious code into a web page. At low effort, Fable 5.1 passed once in five tries, about two minutes each, writing the filter in one go and testing it on a single page. At higher effort, five out of five. In one high effort run of about 33 minutes, the AI:
- Reviewed its own first draft the way an attacker would.
- Read the source code of the tool it relied on, looking for bugs.
- Ran a standard security test suite.
- Wrote its own tool to throw random junk at the filter to see if anything broke.
That’s not an AI working longer. That’s an AI behaving like a careful auditor.
The pattern kept repeating. On a database crash bug, low effort (about a minute) edited code without reproducing the crash: zero out of five. Extra high effort (about 11 minutes) reproduced the crash first, built a randomized test, and checked that its tests actually caught half finished fixes: four out of five. On a math solver, low effort wrote it in one pass and stopped. It even warned it might be slow on big problems, but never checked: zero out of five. High effort compared its answers against a separate brute force solver, timed bigger problems and fixed crashes: five out of five. On a biology data analysis, high effort tried two ways of preparing the data, noticed the answers disagreed, and dug into why: zero to four out of five.
The difference wasn’t intelligence. It was verification.
Now the uncomfortable part. Thariq sorted every failure across 370 attempts by why the AI failed. Going from low to max effort on Fable 5.1, passes went from 140 to 214. Bugs its own tests missed dropped from 40 to 14. Median tokens per attempt went from about 73,000 to 222,000, roughly three times the cost.
And one failure type went up. Picking the wrong reading of the task nearly doubled, from 25 to 47. Higher effort means the AI makes more decisions on its own. If the task is ambiguous, it doesn’t stop and ask you. It picks an interpretation and then polishes the wrong answer beautifully. More effort fixes missing edge cases. It does not fix a wrong approach. In audit terms, a bigger sample size won’t save you if you’re testing the wrong control.
The evidence an auditor should ask for
Which fields benefit most? Moving from low to high effort on Fable 5.1:
- Hardware: 34 percent to 75 percent, more than doubled
- Security: 64 percent to 87 percent
- Machine learning: 54 percent to 73 percent
- General software: 43 percent to 56 percent
- Operations (rulebook style business processes): only 12 percent to 22 percent
The more hidden edge cases a field has, the more effort pays off. When the rules are the hard part, grinding doesn’t help much. You need the right understanding up front.
The rule of thumb from the article:
- Low: brainstorming, sketching, quick changes, when you want to stay in the loop
- Medium: normal work, like building a new feature
- High: when checking matters, like fixing a bug in an old, messy code base
- Max: when you want the AI to run fully on its own on something hard, like building and testing a full app or hunting for security holes
One honest note. These numbers come from Anthropic’s own internal runs, five attempts per task, with some production safety features turned off for Fable 5.1, so they won’t exactly match public leaderboards. Treat them as strong evidence of a pattern, not exact promises.
The test to run this week
This is the loop Thariq actually uses:
- Give Claude Code a spec and ask it to interview you about what’s missing.
- Build on low effort: fast and cheap.
- Review it yourself. Did it get the idea right? Iterate on low.
- Verify and test on high effort.
You make the judgment calls; the AI does the heavy checking. Low effort for direction, high effort for proof.
The jump from zero to five out of five wasn’t magic. It was the AI testing its own work before calling it done. But effort is a budget, not a free upgrade. Spend it on verification and edge cases, and keep the judgment calls for yourself.
If you’re in Claude Code, type this and try one task at low and at high:
/effortCodex users can do the same, and note what surprised you.