The Compounding Error Rule
Buy maximum accuracy wherever steps loop — error compounds, price does not.
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 70%
The rule converts model choice from a price comparison into an arithmetic one. Two models that look close on a single call diverge violently when their output feeds their own next input. A model that is right ninety percent of the time, run a hundred times recursively, is almost never right end to end; a model that is right ninety-nine point nine percent of the time still lands in the eighties or nineties. The gap is not ten percent, it is two orders of magnitude of error. So the decision rule is: measure loop depth first, then buy accuracy for anything involving judgment or chained reasoning, and reserve cost optimisation for repetitive work where a single failure is cheap and recoverable.
Origin
Extracted from Naval. Naval sets out the rule while explaining why he stays cost-insensitive on matters of judgment but would optimise hard on highly repetitive tasks.
Core principles
- 01Errors do not add across steps, they compound.
- 02A hundredfold reduction in error rate is worth far more than a hundredfold reduction in price.
- 03Cost sensitivity belongs on repetitive tasks, not on matters of judgment.
- 04Leverage amplifies both correctness and mistakes.
- 05The right model is a function of loop depth, not of vendor loyalty.
How to run it
- 1
Classify the task by loop depth
Separate one-shot calls from recursive agent runs where each output becomes the next input. Only the second category compounds.
Pro tip Log the actual step count from a real run rather than guessing from the prompt design.
- 2
Estimate per-step accuracy, not benchmark scores
Get a per-step success rate for each candidate model on your own task. Headline benchmarks rarely reflect your loop.
Pro tip A small hand-labelled set of your real steps beats a large public benchmark here.
Watch out Models can look identical on a single call and differ by a hundredfold in error rate.
- 3
Compound before you compare
Raise the per-step rate to the power of the step count for each model, then compare the end-to-end success rates side by side with the price.
Pro tip Express the result as expected reruns per completed task — that converts accuracy directly into cost.
- 4
Route judgment work to maximum intelligence
Where the task involves judgment, leverage, or a long chain, buy the most capable model. Being right more often outweighs the token bill when your time is the scarce input.
Pro tip Price the alternative in your own hours, not in tokens.
- 5
Cost-optimise only the repetitive tail
For high-volume, low-judgment, easily-verified work, drop to the cheapest model that clears your eval bar, and drive the unit cost down deliberately.
Pro tip Elastic fleets that spin up and down are where the large cost reductions actually come from.
Watch out Do not migrate a task to a cheaper model without re-running the compounding maths for that task's loop depth.
- 6
Keep an eval harness on the routing decision
Measure the routing rule continuously so a model upgrade or regression changes the routing rather than silently degrading output.
Pro tip Multiple eval levels, including cross-modal checks, catch failures a single scorer misses.
In the wild
Naval works the arithmetic live on the panel: take one model that is right 99.9 percent of the time and another right 90 percent of the time. Run each recursively a hundred times. The ninety percent model ends up right only a small fraction of the time end to end, while the 99.9 percent model still lands in the eighties or nineties. The point he draws out is that the difference is not ten percentage points, it is a hundredfold lower error, and errors compound.
→ A per-call price gap that looked decisive becomes irrelevant next to the end-to-end success gap.
A team running a dedicated agent for every user applies the rule by splitting traffic. Long, judgment-heavy planning steps stay on the frontier model, while the repetitive retrieval and formatting steps that make up most of the volume are pushed to cheaper models behind an eval harness. That split is what makes a per-user agent economically viable rather than a hundred-dollar-a-month liability.
→ Per-user cost falls by well over an order of magnitude without the long agent runs degrading.
Common mistakes
Comparing models on a single call
Parity on one prompt says nothing about a hundred-step loop; the compounding maths is where the real gap appears.
Cost-optimising judgment work
Saving tokens on decisions that carry leverage trades a small, visible cost for a large, invisible one.
Assuming the routing stays correct
Model quality and price both move quickly, so a routing rule set once and never re-measured drifts into being wrong on both axes.
Is it for you?
Best for
Anyone routing work between frontier and cheaper models, especially where agents loop recursively over many steps.
Not ideal for
Single-shot, low-stakes, high-volume tasks where a cheap model's error rate is genuinely tolerable.
From the transcript
“if they're recursively looping and you run them a hundred times, the one that was right 90% of the time is not gonna be right…”
“Errors can accumulate. Errors compound. That's the point. Errors compound.”
“I will pay for more intelligence, but and I'm not that cost sensitive on matters of judgment”
From the episode
Live in the Future