How to Read a Vendor's AI Benchmark Table

A vendor's benchmark table is a marketing document that happens to contain real numbers. It is not fabricated, and it is not neutral. The useful skill is not deciding whether to believe it, but knowing which four things to check so you can tell how much of the result will survive contact with your workload.

Knowing how to read vendor AI benchmarks comes down to four checks: the comparison set, the harness, the variant, and the rows the vendor could have omitted and did not.

1. Who is in the comparison, and who is missing

The first signal is the guest list. A vendor picks which competitors appear beside it, and that choice is made after the numbers are known.

Look for the obvious absence. If the strongest model in a category is not in the table, the omission is the finding. A comparison against three models from eighteen months ago is a comparison designed to be won.

The inverse is also informative. When DeepSeek published the model card for V4.1-Flash-1017 in October 2026, the table benchmarked it against K3, GLM-5.3, GPT-5.6 Sol and Opus 5, and and the card included rows where Opus 5 came out ahead: ProgramBench at 37.0 against 29.1, NL2Repo-Bench at 75.3 against 68.2. A vendor that prints the rows it loses is giving you a reason to weight the rest more heavily. We covered that release in detail separately.

2. Which harness produced the number

The same model on the same benchmark can differ by several points depending on scaffolding. Agentic benchmarks are the worst for this, because the harness is doing real work: retry logic, tool definitions, how failures are handled, how many turns are allowed.

So the question is whether the vendor ran its competitors through the same harness it used for itself, or copied their self-reported numbers from their own cards. Both are common. Only one is a comparison.

Things that quietly change a score:

• Tools on or off. A figure reported "without tools" and one reported with them are different measurements. Cards usually say which, in small type.

• Turn limits. More attempts, higher scores, on anything agentic.

• Prompt format. A model evaluated with its own preferred format against competitors evaluated with a generic one is not a fair fight.

• Pass@k. Best of several attempts is a different claim from first attempt.

3. Which variant was measured

Vendors ship families. Check that the row refers to the checkpoint you can actually call, at the settings you would use.

The usual mismatches are a benchmark run on a larger sibling than the one being announced, a reasoning mode you pay more for, or a quantisation you would not deploy. None of these are dishonest when disclosed. All of them make the number non-transferable if you miss the footnote.

4. What the benchmark actually measures

A benchmark name is a compression of a long methodology, and the compression loses the part you need.

Terminal-Bench scores completing tasks in a terminal, which exercises tool use and multi-step recovery rather than prose quality. A reasoning exam measures hard single-shot problems and says little about whether a model follows a formatting instruction across forty calls. Neither is a proxy for "good at my job".

Ask what task in your product the benchmark resembles. Often the honest answer is none of them, and the table is therefore a weak signal about your use case, however rigorous it is. Our primer on what an AI benchmark is covers the categories, and benchmark contamination covers the failure where test items have leaked into training data, which inflates scores without anyone intending it.

A quick way to read any vendor table

Work down this list before you quote a number to anyone:

1. Which competitors are present, and which obvious one is absent.

2. Did the vendor run everyone, or import self-reported figures.

3. Exact model variant, and the settings in the footnote.

4. Tools on or off, turn limits, pass@k.

5. The size of the gap. Half a point is noise. Eleven points is a claim.

6. Any row where the vendor loses, which tells you how curated the rest is.

What to do with a number you like

Treat it as a reason to test, never as a result. A benchmark is evidence that a model is worth an afternoon of your attention, and that is genuinely valuable, because there are too many releases to test them all and the table is a filter.

The afternoon itself looks like twenty prompts from your own production traffic, run through both candidates, graded blind. That measurement answers the only question the vendor cannot: whether it is better at your work. We walk through the method in how to evaluate a new AI model release before switching, and the same discipline applies to reading a model's system card, where the safety sections reward the same scepticism.

FAQ

Are vendor AI benchmarks trustworthy?

The numbers are usually real and the selection is not neutral. Vendors rarely publish figures they cannot support, because competitors reproduce them. What they control is which benchmarks appear, which competitors appear, and which harness ran them.

Why do independent benchmarks give different results?

Mostly harness and prompt differences, plus genuine variance between runs. A gap of a point or two between a vendor's figure and an independent one is normal. A gap of ten points suggests the two were measuring different things.

What is a meaningful improvement on a benchmark?

It depends on the benchmark's spread, but as a working rule, under one point is noise, one to three points is a real but modest point release, and anything into double digits is a claim worth testing immediately because it will either be obvious in your own harness or not there at all.

Should I ignore benchmarks entirely?

No. They are a cheap filter in a field that ships faster than anyone can evaluate. Use them to decide what to test, and never to decide what to deploy.