The Data-and-Verification Test
Score any domain on data volume and verification speed to predict where AI wins.
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 5
- Confidence
- 80%
Naval reduces the question of where AI will excel to two variables. First, how much data exists for the task, because the models learn from the corpus that already exists. Second, how cheaply and objectively the output can be verified, because training only compounds when a machine can grade the result without a human. Coding scores maximum on both, since code has to compile and execute and pre-written tests answer the did-it-work question automatically. Mathematics and self-driving score similarly. Creative writing scores high on data but collapses on verification, because nobody can algorithmically decide what is good, and hiring cheap graders only buys you their taste. Brand-new fields score zero on data. Plotting a task on these two axes tells you whether to hand it to an agent, keep it human, or invest in building the verification loop yourself.
Origin
Extracted from Naval, where Naval works through why coding agents improved so fast in late 2025 and generalises the answer into a test for any domain.
Core principles
- 01Models get good where training signal is abundant, not where the work is prestigious.
- 02A domain is learnable at speed only if a machine can grade the output without a human in the loop.
- 03Coding and mathematics fell first because both have huge corpora and instant, objective verification.
- 04Where verification requires taste, the bottleneck is the grader, not the model.
- 05Brand-new fields have no corpus, so they stay human territory by default.
- 06The durable advantage is owning a high-taste feedback loop nobody else can copy.
How to run it
- 1
State the task, not the job
Break the work into the smallest concrete unit that could be handed to an agent. Jobs are bundles of tasks that score very differently on both axes.
Pro tip Write the task as a sentence that ends in a checkable output, e.g. 'produce code that passes these tests'.
- 2
Score the data axis
Ask how much of this exact task already exists in public text, code or recordings that a model could have trained on. Abundant corpora mean the model has seen the pattern thousands of times.
Watch out A field that feels mature to you may still be data-poor if the work was never written down.
- 3
Score the verification axis
Ask whether a machine can decide, unaided, whether an output is right. Compilation, test suites, proofs and simulator outcomes are cheap verifiers; human opinion is not.
Pro tip If your grading step requires a person to say 'this one is better', you are on the hard side of the axis.
Watch out Paying low-cost human raters does not solve verification, it just imports their taste ceiling.
- 4
Place the task in the quadrant
High data plus easy verification means agents will dominate quickly. Low data or hard verification means humans and creativity still hold the ground.
- 5
Act on the placement
Automate the high-scoring tasks immediately, keep your attention on the low-scoring ones, and treat a missing verifier as a build opportunity rather than a permanent barrier.
Pro tip The scarce asset in taste-graded domains is a high-quality feedback loop, so build one before competitors do.
Watch out Re-run the test periodically; domains move as new data and new verifiers appear.
In the wild
Naval explains that coding turned out to be unusually easy to train models on because there is an enormous public corpus of code and because correctness is mechanically checkable. The code has to compile and execute, and pre-written tests can confirm whether the model did the thing it was supposed to do. That closed loop lets training run at scale without humans grading each attempt, which is why coding agents improved dramatically in a short window while other domains lagged.
→ Coding agents crossed from assistant to autonomous operator, and Naval moved from not having seriously coded in decades to shipping working apps.
Naval points out that models can output infinite essays, but nobody can say algorithmically which ones are good. Running a closed training loop requires immediate automated grading, and creative writing has none. Even hiring people call-centre style to label outputs good or bad caps the result at those raters' taste. The data is abundant; the grader is the bottleneck.
→ Creative and taste-driven work remains a human advantage until someone builds a credible high-taste feedback loop.
A finance operator wants to know whether to automate contract compliance review. Data axis: high, since decades of contracts and rulings are written down. Verification axis: mixed, because clause extraction is checkable against a rule list but materiality judgements are not. The test says to automate extraction and flagging now, keep the materiality call human, and invest in building a labelled rule-check set that turns part of the judgement into a machine-gradable step.
→ A split workflow where agents handle the verifiable half and human attention concentrates on the taste-graded half.
Common mistakes
Scoring the job instead of the task
Whole roles bundle tasks that sit in different quadrants, which produces a mushy answer. Score each task separately and the picture resolves.
Mistaking cheap human raters for verification
Hiring people to label outputs good or bad does not create an automated loop, and the result is only as good as their taste. Real verification runs without a human in it.
Treating the placement as permanent
A domain moves as new corpora accumulate and as someone builds a verifier for it. Re-run the test rather than defending a one-time verdict.
Is it for you?
Best for
Operators, investors and creators deciding where to deploy agents, where to keep humans, and what to build a career or company around.
Not ideal for
Anyone who needs a precise timeline or capability forecast for a named model, rather than a directional read on a domain.
From the transcript
“So in domains where you have a lot of data and you have good verification, self-driving is another one of those, these models do extremely…”
“In areas where you don't have a lot of data, which are kind of brand new fields, the models are not gonna do well, and…”
“You need high-taste feedback loops to improve these models, and those are harder to develop than they look.”
From the episode
A Return to Code