Apples-to-Apples Chain Benchmark
Measure every chain on one real transaction type under production conditions.
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 80%
Published throughput numbers are usually marketing artifacts: each team picks the transaction type and the environment that flatter it, typically bare transfers on a testnet. This method removes both degrees of freedom. First, fix one workload that every chain genuinely runs in production, such as an automated market maker swap, which is how most on-chain trading actually happens. Second, fix the environment to production, so the result reflects real block times, real cost accounting, and real contention. Where a chain's gas model fully describes the cost of a transaction, capacity can be derived analytically from the gas limit and the block time; where the gas model does not tell you enough, the workload has to be run empirically until the chain stops keeping up. The output is one comparable figure per chain, which should always be reported next to the decentralization that chain gave up to reach it.
Origin
Extracted from Naval. Haseeb Qureshi describes the study his team at Dragonfly ran after seeing repeated throughput claims that could not be reconciled, choosing an automated market maker trade as the common workload across chains.
Core principles
- 01A vendor's benchmark measures the vendor's marketing, not the system.
- 02Comparability comes from fixing the workload, not from trusting the reporter.
- 03Pick the transaction people actually send, not the one that scores best.
- 04Test where the system really runs; a devnet number is a laboratory number.
- 05Derive analytically where the cost model allows it, measure empirically where it does not.
- 06A performance number without its decentralization cost is only half a result.
How to run it
- 1
Choose a workload every system actually runs
Select one transaction type that is both common in production and supported everywhere. An automated market maker swap qualifies because most on-chain trading takes that form, so the benchmark measures work people really do.
Pro tip Pick the workload before you look at any results, so the choice cannot be reverse-engineered from which chain it favours.
Watch out Simple transfers are the default benchmark precisely because they are the least informative; they exercise almost none of the execution path.
- 2
Fix the environment to production
Run the comparison against the live network rather than a testnet or devnet. Isolated environments produce enormous numbers that do not reflect real contention, real block propagation, or real validator sets.
Watch out Any claim sourced from a devnet should be treated as an upper bound with no evidential value for production capacity.
- 3
Derive capacity where the cost model is complete
For chains whose gas accounting fully prices a transaction, compute throughput from the gas limit and the block time. This is cheap, transparent, and reproducible by anyone who disagrees with the result.
Pro tip Publish the arithmetic alongside the number so critics can attack the model rather than the conclusion.
- 4
Measure empirically where it is not
Some chains price transactions in ways that do not predict real behaviour. There you must submit the workload against the live system and observe the sustained rate rather than trusting the cost model.
Watch out Peak burst rates are not sustained rates. Record what the chain holds under continuous load, not its best block.
- 5
Report the trade-off alongside the number
A throughput figure is meaningless without the decentralization it cost. Pair each result with hardware requirements and node counts so readers can see which chains bought speed and how much they paid.
Pro tip State explicitly when the decentralization axis has not been measured, so the comparison is not read as a full ranking.
Watch out Without this column, the benchmark will be quoted as a leaderboard and will reward exactly the trade-off it was built to expose.
In the wild
Rather than accepting published throughput claims, the team fixed a single workload used across the industry and measured how many of those trades each chain sustained. Ethereum landed at roughly ten trades per second, Celo about twenty-five, Avalanche about thirty, Polygon about fifty, and Binance Smart Chain about two hundred. Solana, which claims thousands to tens of thousands of transactions per second, came in at about two hundred and eighty. The exercise did not measure the decentralization axis, which was flagged as the missing half of the comparison.
→ Chain performance claims collapsed onto one comparable scale, and the gap between advertised and measured throughput became visible.
Common mistakes
Benchmarking bare transfers
Transfers are the cheapest possible transaction and exercise almost none of a chain's execution machinery. They produce large, comparable-looking numbers that say nothing about how the system behaves under the work users actually generate.
Accepting devnet results as capacity
A benchmark run in an isolated environment removes contention, adversarial load, and real validator geography. The resulting figure is an upper bound of a laboratory, not a description of production.
Publishing speed without the decentralization column
Reporting throughput alone turns the benchmark into a leaderboard that rewards centralization. The number only becomes decision-useful once the reader can see what verifiability was traded away to earn it.
Is it for you?
Best for
Investors and engineers who need a defensible performance comparison across systems whose vendors all claim to be fastest.
Not ideal for
Systems with no shared workload in common, where no single transaction type is representative of all of them.
From the transcript
“usually when people benchmark blockchains, they tend to choose benchmarks that make them look good, that produce big numbers”
“they tend to benchmark transfers that are done in a testnet or done in a devnet environment, where you can always get crazy, gigantic numbers…”
“Most of these you can just simulate by looking at the gas limit and looking at the block time”
From the episode
Vitalik: Ethereum, Part 1