Error-Theory-First Experimentation
Spend most of the effort theorising the errors before trusting any measurement
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 71%
Deutsch's claim is that experimentation is far harder than people realise, and that almost all of the effort required to do a scientific experiment is forming theories about the errors: explanations of what errors could occur, and then either forestalling them or measuring them. The output is not a single number but a body of understanding about the ways the apparatus can lie. He describes visiting the cellar of the Cavendish laboratory, where a team was making a single atom perform quantum computations, and finding the wall covered with graphs of the errors, so that the whole experiment could be regarded as an experiment about the errors involved in making a qubit. Skip that stage and you get random results, which in practice means you get the results you were hoping for, and they do not replicate.
Origin
Extracted from Naval. Deutsch draws the method from watching serious experimentalists work, notably a qubit experiment in the cellar of the Cavendish laboratory, and contrasts it with the room-temperature superconductor episode.
Core principles
- 01Mistakes are everywhere and there is no limit to how many you can make.
- 02Almost all of the effort in a real experiment goes into theories about the errors.
- 03An experiment is best understood as an experiment about its own errors.
- 04Wanting a result is the mechanism by which you get it spuriously.
- 05Authority and sincerity say nothing about whether an observation is sound.
How to run it
- 1
Declare the result you are hoping for
Write down, before you start, what outcome you want. Naming it turns an invisible bias into a known error source you can design against.
- 2
Write explanations of the possible errors
Do not list error bars, list mechanisms. For each one, explain how the apparatus, the environment or the procedure could produce a wrong reading.
Pro tip Treat this as the main scientific work of the project rather than as preparation for it.
Watch out An error you have not explained cannot be forestalled or measured, so it will silently pass into the result.
- 3
Forestall or measure each error
For every mechanism, decide whether the design will eliminate it or whether you will quantify it and carry it through. Every mechanism gets one of the two.
- 4
Put the errors on the wall
Keep the error data as visible and as central as the headline result. In the Cavendish cellar the wall of error graphs was the experiment.
Pro tip If the error record is harder to find than the result, the priorities have already inverted.
- 5
Run the bad-experiment thought test
Ask what a deliberately sloppy version of this experiment would return. If the sloppy version would give you the answer you want, your controls are not yet doing any work.
- 6
Refuse authority as evidence
When a claim is challenged and the defence appeals to who made the observation, treat that as a signal that the error analysis is missing, not as reassurance.
In the wild
Deutsch was taken into the cellar of the Cavendish laboratory, past what he describes as Frankenstein-like apparatuses, to see an experiment on a single atom being made to perform quantum computations on a single qubit. What struck him was the wall: it was covered with graphs of the errors. The whole experiment could be regarded as an experiment about the errors that happen when you try to make a qubit. Had they simply set up the experiment as it would later be described in the paper, without the improvements that removed those errors, they could have got random results.
→ The error analysis, not the headline measurement, was the substance of the work, and it was what made the result trustworthy.
Deutsch points to the recent room-temperature superconductor episode as the failure mode in the wild. When you want to believe something and drop the skepticism you should have around the measurement, you can get almost any result you want, in a non-replicable way. The mechanism is not fraud; it is the ordinary consequence of running an experiment without first building theories of how it could mislead you.
→ Results appear, get announced, and then fail to replicate, because the experiment was never an experiment about its own errors.
Common mistakes
Treating error analysis as a write-up chore
If errors are only characterised when the paper is being drafted, the design never had the chance to forestall them and the result may already be contaminated.
Defending a result with credentials
Pointing to the observer's rank or honesty answers a question nobody asked. Mistakes are universal and independent of sincerity.
Running the experiment you already believe
A poorly controlled experiment tends to return the hoped-for answer rather than a random one, which makes wanting the result the single most dangerous input.
Is it for you?
Best for
Researchers, lab scientists and analysts running measurements where a plausible-looking wrong answer is easy to obtain.
Not ideal for
Rough exploratory checks where the cost of being wrong is trivial and speed matters more than validity.
From the transcript
“required to do a scientific experiment is forming theories about the errors forming explanations of what errors there could be and then forestalling them or…”
“the wall was covered with graphs of the errors so you could regard their whole experiment as an experiment about the errors”
“if you want to believe something and then you drop all of the skepticism that you should have around the measurement then you can get…”
From the episode
The Deutsch Files II