The Feedback Source Test
Trust the judgement of people paid by reality, discount the people graded by their peers.
- Difficulty
- Easy
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 72%
The test asks one question about any claim: where does this person's feedback come from? A journalist writing to impress other journalists, or a restaurateur cooking to impress other restaurateurs, is optimising for a signal generated inside the profession, and that signal drifts from what is practical and high quality even as the prizes accumulate. A scientist tested by nature, or an entrepreneur tested by whether strangers part with money and time, is graded by something that does not care about their standing. This also explains the standing asymmetry between pessimists and optimists: pessimism is a social act performed for other people and is rewarded inside academia and media, while entrepreneurs are rewarded for optimism because they get paid only when something actually works.
Origin
Naval Ravikant sets out this rule while explaining why intellectuals are systematically pessimistic and entrepreneurs systematically optimistic, in conversation with Brett Hall about rational optimism.
Core principles
- 01Any profession whose feedback comes from its own members drifts away from reality.
- 02Nature and free markets are harsher and more honest graders than colleagues.
- 03Incentives explain the systematic optimism of builders and the systematic pessimism of commentators.
- 04Accolades from an elite circle are evidence about the circle, not about the work.
- 05A pessimistic philosophy tends to become a pessimistic psychology.
How to run it
- 1
Name the claim and the claimant
Isolate the specific forecast, opinion or piece of advice you are being asked to weight, and who is making it.
- 2
Trace the feedback loop
Work out who decides whether this person is doing well: their colleagues and prize committees, or customers, users and physical reality.
Pro tip Follow the funding — where the money comes from usually reveals who the real audience is.
- 3
Identify the incentive on belief
Ask what this person gains by holding this position. Explaining how dangerous a problem is can be the route to funding; claiming a problem is solvable can sound naive in the same room.
Watch out Incentive analysis explains a bias, it does not by itself refute the claim — you still have to check the argument.
- 4
Weight accordingly
Downgrade claims produced inside a closed professional feedback loop and upgrade those from people whose livelihood depends on being right in the world.
Pro tip Ask for the claimant's last five predictions and how they resolved.
- 5
Rewire your own loop
Apply the same test to yourself. Make sure some part of your work is judged by nature or by a market, not only by people who do the same job as you.
Pro tip Shipping something small that strangers can pay for or reject beats another round of peer applause.
In the wild
A journalist writing articles designed to impress other journalists, and a restaurateur running a restaurant designed to impress other foodies and restaurateurs, both accumulate accolades and prizes inside their circles. Neither output ends up especially practical or high quality, because the audience being satisfied is not the audience being served. The same work judged by readers who choose what to read, or diners who choose where to spend, would be pulled back toward reality.
→ Prestige inside a profession turns out to be a poor proxy for quality once you check who is doing the grading.
Standing in front of an audience with a frown and explaining all the ways civilisation will come to ruin reads as the intellectually serious position, and for an academic it doubles as the case for further funding to study the problem. Saying the problem is solvable through collaboration and knowledge sounds naive in that room. Meanwhile the historical timelines on which the world was supposed to end have been quite wrong, and the entrepreneurs graded by nature and free markets have been the ones consistently rewarded for building the way out.
→ The feedback-source test explains a persistent forecasting bias that credentials alone would never reveal.
Common mistakes
Reading incentive as proof of falsehood
A peer-graded claimant can still be right, and a market-graded one can still be wrong. The test adjusts your prior, it does not replace examining the argument itself.
Confusing prizes with reality checks
Awards, citations and elite acclaim are generated inside the same closed loop you are trying to test, so they cannot be used as independent evidence of quality.
Assuming every builder's optimism is calibrated
Entrepreneurs are incentivised toward optimism just as academics are toward pessimism; the difference is that the market eventually settles the bet. Treat an unsettled optimistic claim as an open bet, not as a verdict.
Is it for you?
Best for
Choosing whose forecasts, advice or criticism to act on, and designing your own work so it gets graded by reality rather than by your peer group.
Not ideal for
Domains where no market or natural test exists and expert peer review genuinely is the only available check.
From the transcript
“in general professions in which you get your feedback from other members of that profession tend to get corrupted”
“the people who are operating in the real world and are getting paid for it tend to be optimist the people who are operating ivory…”
“if you're a pessimist you get your feedback from other people it's a social act you're convincing other people of your pessimism”
From the episode
The Beginning of Infinity, Part 1