Why Do AI Leaderboards Disagree With Each Other?

AI leaderboards disagree because they are not measuring the same thing. One board averages scores across a fixed set of benchmarks. Another counts anonymous votes from people comparing two chatbot replies. A third times how fast a model solves a task against how well it solves it. Ask why do AI leaderboards disagree and the honest answer is that they were built to answer different questions, so a model can lead one board and sit twentieth on another in the same week. Anyone keeping up with the flow of AI releases runs into this constantly. Gemini 3.7 Flash, which Google released in mid-August 2026, is a clean recent example, and walking through it shows what to check before trusting any single chart.

Why AI Leaderboards Disagree: Three Things a Board Can Measure

Most disagreements trace back to which of three setups a chart is drawing from. Knowing the difference between an eval and a benchmark helps here too.

1. A composite benchmark index. It aggregates scores from a fixed list of evaluations into one number. The Artificial Analysis Intelligence Index v4.1.1, for example, combines nine evaluations, from Humanity's Last Exam and GPQA Diamond to agentic tasks like Terminal-Bench and GDPval-AA v2.

2. A crowd-preference leaderboard. Anonymous users compare two model outputs and vote for the one they prefer, producing an Elo-style rating that rewards tone and confidence as much as correctness.

3. A single narrow task or tradeoff chart. One coding benchmark, one cost-per-task figure, or a chart plotting intelligence against speed. It tells you exactly one thing well and nothing about anything else.

A model can top the second kind of chart and rank outside the top ten on the first without anyone involved being dishonest. They graphed different axes.

The Worked Example: Gemini 3.7 Flash's Numbers Depend on Which Chart You Read

Google's own model card for Gemini 3.7 Flash reports domain wins: 65.3% on DeepSWE v1.1, 43.6% on FrontierCode 1.1, 97.0% on GDM-MRCR v2 at 128k tokens, and comparisons against Claude Sonnet 5, GPT-5.6 Terra, and Muse Spark 1.2. That is one honest way to present the release: here is where the new model beats named competitors on named tasks.

Artificial Analysis covered the same release differently. Its Intelligence Index writeup put Gemini 3.7 Flash (high) at 56 points, a 4-point gain over Gemini 3.6 Flash, and framed the headline as reaching the Intelligence-versus-Time-per-Task Pareto frontier, meaning the best speed-for-intelligence tradeoff at its tier, not an overall rank. Its model comparison page places that same 56 at position 20 of 186 models tracked overall. Both numbers are accurate. They answer different questions: one about tradeoffs, one about rank.

There is a second layer here. Google's model card states plainly that this release used improved evaluations and that results are, in its own words, not directly comparable with previous Gemini model cards. That is a vendor admitting its own baseline moved. If a company's current chart is not safely comparable to its own last chart, a chart comparing that model to a competitor's deserves the same scrutiny.

Four AI Benchmark Red Flags to Check Before You Trust a Score

These same details point to four checkable red flags, and every one of them shows up somewhere in the Gemini 3.7 Flash coverage above.

1. The comparison baseline is cherry-picked or outdated. A model card chooses which competitors it lines up against and how recent their versions are. Gemini 3.7 Flash's card names three current models, which is reasonable, but the fact that the vendor's own numbers are not comparable release to release is a reminder to check whether a rival's score reflects its current shipped version or an older snapshot that flatters the new release.

2. The eval set is private or not independently reproducible. GDPval-AA v2 and AA-Briefcase, two evaluations feeding the Intelligence Index, are Artificial Analysis's own proprietary agentic tests. Publishing a methodology page is good practice, but it is not the same as an outside lab rerunning the exact task set. A private eval can be well built and still be a single source's word, not a verified result.

3. Pass@k or best-of-N is not disclosed next to the headline number. Buried in the same coverage, Gemini 3.7 Flash's AA-AnalystAgent score is reported as 60% at pass@5, meaning five attempts were allowed. The flagship Intelligence Index figure that gets quoted carries no sampling label at all. A score with no attempt count attached could be a first try or the best of ten.

4. Benchmark contamination cannot be ruled out. Even an honestly run benchmark can be compromised if pieces of it leaked into a model's training data first. Researchers working on this problem describe the core issue directly: test samples or close variants leaking into training data inflate reported performance, and it is genuinely hard to catch from outside the lab, as this paper on decontamination methods lays out. Because labs rarely publish full training corpora, nobody outside the building can rule this out for certain.

How to Read Any AI Model Benchmark in About a Minute

Are AI benchmarks reliable? For what they were built to measure, generally yes. As a shortcut for which model to use, only if you check the fine print first:

Worth checking separately: the difference between a preview and a GA release, since a benchmark run against a preview build is not a promise about what ships generally. Once the version, sampling method, and source are pinned down, a model's system card is usually the fastest place to find all three in one document.

None of this means ignore leaderboards. It means treat a rank as a hypothesis to test on your own task, not a verdict to repeat. Spotting an inflated benchmark claim and reconciling why two leaderboards disagree are the same habit: read past the chart to the method underneath it.

Questions

Are AI benchmarks reliable?

They are reliable for the narrow thing each one measures. A composite index reflects a fixed evaluation set, a crowd leaderboard reflects voter preference, and a single task chart reflects that task alone. None predicts performance on a task it never tested, which is why cross-referencing two or three sources beats trusting any one rank.

Why does LMArena rank models differently than the Artificial Analysis Intelligence Index?

They run on different inputs. LMArena's Elo ratings come from anonymous voters picking a preferred response, which rewards tone as much as accuracy. The Intelligence Index aggregates scores from a fixed set of standardized evaluations instead. Two models can be tied on one and far apart on the other because the boards are not measuring the same thing.

What is benchmark contamination and why does it matter?

It happens when questions or answers from a test set end up in a model's training data, so the model has effectively seen the exam beforehand. A contaminated model can score well through memorization rather than reasoning, and since labs rarely publish full training corpora, it is hard for anyone outside to confirm one way or the other.

What does pass@k mean on a benchmark chart?

It means the model got k attempts at a task, and the score counts a success if any attempt got it right. A pass@5 score is a meaningfully easier bar than pass@1 on the same task, which is why the sampling method needs to sit next to the number, not buried in a methodology page nobody reads.

Which AI leaderboard should I trust?

None of them exclusively. Cross-reference at least two boards built on different methods, one composite and one preference-based, check the methodology page for sampling details, and where the decision matters, run the model on tasks that look like your own work.