Explainer · · 6 min read
Benchmarks lie. Here is how to read them anyway
A model topping a leaderboard tells you less than you think. What the numbers measure, what they miss, and when to trust them.
By Manish
Every model launch arrives with a chart, and the launching lab always wins its own chart. That is not fraud. It is selection. You publish the benchmarks you do well on, and every lab does it, including the ones you trust.
Benchmarks are still the best public evidence we have about model capability. You just have to know what they measure, what they cannot measure, and when a difference is real. This is the working guide.
What a benchmark score actually is
A benchmark is a fixed set of questions with known answers. The score is the percentage the model got right. That is genuinely useful and considerably narrower than the headline suggests.
- It measures performance on those questions, not on your work.
- It usually measures one run. Ask again with a different random seed and the number moves.
- It measures the raw model, not the product wrapped around it, and you use the product.
- It measures the model on the day it was tested, under whatever settings the tester chose.
None of that makes benchmarks useless. It makes them a specific kind of evidence, in the way a lab test is evidence about a patient without being the patient.
Contamination: the one that invalidates everything
If benchmark questions appeared in the training data, the model is recalling rather than reasoning, and the score means nothing at all. This is not a hypothetical. Public benchmarks sit on the open web, which is where training data comes from.
The defences are held-out test sets that are never published, private evaluations run by third parties, and freshly written questions released after a model's training cutoff. When you see a model do dramatically better on an old benchmark than on a structurally similar new one, contamination is the first explanation to reach for.
There is now statistical work on detecting this directly, estimating whether specific items were memorised. It is worth knowing that when a lab publishes contamination analysis alongside results, that is a meaningful signal about how seriously they take their own numbers.
Saturation: when the benchmark stops working
Once every serious model scores above ninety-five percent, the benchmark has stopped discriminating. The remaining few points are noise, ambiguous questions, and occasionally errors in the answer key rather than genuine capability differences.
This is why benchmarks get retired and replaced. A test that everything passes tells you only that you are looking at the wrong test. When you see a leaderboard where the top ten sit within two points, that leaderboard has finished being informative and nobody has taken it down yet.
Prompt sensitivity: the underrated one
Rephrase a benchmark question and rankings reorder. Research on evaluation variance has found shifts large enough to change which model leads, from changes as small as reordering multiple-choice options or adjusting the instruction wording.
This has an uncomfortable implication. A published ranking reflects both the models and the specific prompt formatting the evaluator chose. Two honest labs running the same benchmark can reach different orderings without either doing anything wrong.
Treat a two-point gap as a tie. Treat a twenty-point gap as a finding. The middle is a judgement call, not a fact.
The benchmarks that actually matter to you
Find the one closest to your real work and ignore the rest. General averages blend a dozen unrelated capabilities into a single number that describes nobody's use case.
- Writing code: software engineering benchmarks scored on genuine merged pull requests, not on isolated function generation.
- Long documents: long-context retrieval scores specifically, and pay attention to whether facts were placed mid-document, because that is where models degrade.
- Reasoning: benchmarks with held-out sets released after training cutoffs, since these are the most contamination-resistant.
- Multilingual work: per-language results, because aggregate multilingual scores hide enormous variation between languages.
- Agents: repeated-run reliability rather than best-of-n, because you experience the average run.
That last distinction catches people. A model reported as solving a task usually means it solved it at least once across several attempts. If you need it to work every time, best-of-n is measuring the wrong thing entirely.
What benchmarks cannot tell you
Quite a lot, and it is mostly the part that decides whether you are happy with a tool.
They cannot tell you about latency, which for interactive work matters more than a few points of accuracy. They cannot tell you about cost, and a model that is two percent better for ten times the price is not better for most purposes. They cannot tell you about refusal behaviour, tone, or how it fails, and how a model fails is often more important than how often.
They also cannot tell you about the product. Rate limits, uptime, the quality of the API, whether the interface is bearable. Benchmarks measure the engine and you are buying the car.
Build your own evaluation, it takes an hour
Save ten real inputs from your actual work, along with outputs you were satisfied with. That is your benchmark. It is more informative for your purposes than any public leaderboard, because it tests the thing you actually need.
Run it whenever you consider switching models, and whenever your provider ships an update. This catches the silent regressions that forced upgrades introduce, which is the failure mode that costs teams the most and gets discussed the least.
The hour this takes is the highest-return hour available in AI engineering, and it is the one almost everybody skips.
The position we hold to
Benchmarks are facts about tests. What they mean for your work is judgement. We keep those two things visibly separate across this site, publishing benchmark results untouched by our opinion of a tool, and publishing our opinion separately where you can argue with it.
Blending them lets one launder the other, and a directory that does that is just an advertisement with numbers on it.
Who ran the test changes what it means
Provenance is the fastest quality filter available, and it takes seconds to check.
- Self-reported by the lab that built the model. Useful, and selected. Assume the unflattering results exist and were not published.
- Independent third party with a published methodology. The strongest evidence available, and the rarest.
- Community leaderboards from human preference votes. Good signal on which outputs people like, weak signal on correctness, and gameable by response style.
- A blog post with a chart and no methodology. Not evidence at all, whatever the numbers say.
When comparing two models, check whether the same party tested both under the same conditions. Comparing a self-reported number to an independently measured one is not a comparison, and this happens constantly in coverage that should know better.
Arena scores measure something real and specific
Human preference leaderboards, where people vote blind between two responses, get used as a general capability ranking. They are not that, and understanding what they actually measure makes them useful.
They measure which response a person prefers on a quick read. That correlates with quality and it also correlates with length, confident tone, formatting and friendliness. A model that writes longer, better-formatted answers scores higher without being more correct.
This makes arena rankings genuinely informative for consumer chat products, where being pleasant to read is a real feature, and misleading for anything where correctness is what matters and the voter cannot verify it.
The questions to ask of any chart
Five questions catch nearly everything, and they take a minute.
- Who ran it, and did they build one of the models?
- Is the test set public? If so, contamination is possible and should be addressed.
- How many runs, and is there any variance reported? A single run with no error bars is one sample.
- What settings were used, and were they identical across models? Temperature and prompt formatting move results.
- What is missing? A chart with four models when six are relevant has told you something by omission.
That last one is the most productive. Benchmark charts are as informative for what they leave out as for what they show, and the omission is rarely accidental.
See how the top AI models compare
Our model rankings put the scores, prices and strengths side by side.