What Is an AI Eval? Your Test, Not Their Test
An eval is a repeatable test of whether a model does your job correctly. Benchmarks are somebody else's test. Here is how to build your first one.
An AI eval is a repeatable test that checks whether a model does your specific job correctly. You collect real inputs, define what a correct output looks like, run the model, and score the results. Do that again after every prompt change, model upgrade, or provider switch, and you have replaced "it feels better now" with a number.
The distinction that matters: a benchmark is someone else's test on someone else's data. An eval is your test on your data. Only one of them tells you whether your product works.
Why benchmark scores do not answer your question
A model that scores well on a public exam-style benchmark has demonstrated something real, but it has not demonstrated that it can read your customers' support emails and route them correctly. Those are different tasks with different failure modes, and the gap between them is where most AI products quietly break.
There is a second problem. Public benchmarks are public, which means their questions leak into training data over time, and scores drift upward without capability moving with them. We wrote about how to read those claims critically in what is an AI benchmark and how to spot an inflated AI benchmark claim. Your eval has neither problem, because nobody outside your company has ever seen it.
Benchmark | Eval | |
|---|---|---|
Whose data | Public, shared | Yours, private |
What it measures | General capability | Your task, done your way |
Contamination risk | High and rising | None |
Tells you when to ship | No | Yes |
Who builds it | A lab or research group | You, in an afternoon |
What an eval is made of
Three parts, and none of them require a framework to start.
Cases. Real inputs from your actual usage, not invented examples. Twenty is enough to be useful. Pull them from logs, from support tickets, from the requests users actually made. Include the ugly ones: the empty input, the input in the wrong language, the one with a typo in the customer name, the one where the correct answer is "I do not know."
A definition of correct. This is the hard part and the part worth spending your time on. For classification it is easy: the right label or the wrong one. For open-ended generation you need a rubric, which usually means three or four checks per case rather than one holistic judgement. Does the response cite a real order number from the input? Does it avoid promising a refund? Is it under 120 words?
A scorer. Exact match where possible. A regular expression or a JSON schema check where the shape is known. A second model acting as judge where the output is genuinely open-ended, though a model judge needs its own spot check against human ratings before you trust it.
Build your first eval in an afternoon
Collect twenty real inputs. Sort them so roughly five are cases you know the model handles badly today. An eval where everything passes teaches you nothing.
Write the expected outcome for each. Not the exact expected text, which is too brittle, but the properties the answer must have. Write these before you look at what the model currently does, or you will unconsciously grade to the current behaviour.
Run the model and score by hand, once. Yes, by hand. Doing twenty by hand teaches you what your rubric is actually missing, and it takes about forty minutes.
Automate the scoring. Now that the rubric survived contact with real outputs, encode it. Most checks turn out to be string containment, a schema validation, or a number comparison.
Record the baseline and never delete it. The number only means something as a series.
Both major providers ship tooling for this if you would rather not build the harness yourself. Anthropic documents an approach to developing test cases, and OpenAI has an evals guide covering graders and scoring. The tooling is a convenience. The rubric is the work.
What good looks like after a month
An eval earns its keep when it stops you from shipping something. That is the actual success criterion, and it usually happens the first time a prompt change that obviously improved one case quietly broke three others.
A few habits that make the difference:
Run it on every prompt edit, not weekly. The whole value is catching regressions at the moment you cause them, while you still remember what you changed.
Keep failures in the set forever. Every production bug becomes a new case. Over six months this is how the set becomes genuinely representative, and it costs nothing to maintain.
Track the score per category, not just the total. A stable 85 percent can hide one category collapsing while another improves.
Do not tune the prompt until the eval passes. If you iterate against the eval hard enough you have simply overfitted to twenty cases. Hold back five you never look at.
Where evals fit with everything else
An eval is a quality gate, not a substitute for judgement about which model to use at all. Sequence it: shortlist candidate models on capability and cost, then run your eval on the shortlist, then decide. How to test a new AI model before you switch covers that sequence, and if the work in question is code rather than text, benchmarking AI coding agents on your codebase applies the same idea to a harder-to-score output.
It also helps to understand what you are measuring. Model outputs vary between runs by design, so a single-run eval score has noise in it. Run each case three times and take the worst result if your product cannot tolerate variance, or the median if it can. How AI models work explains where that variation comes from.
Frequently asked questions
How many cases does an eval need?
Twenty gets you signal. A hundred gets you confidence. Beyond a few hundred you are mostly paying for compute, unless your task has many distinct categories, in which case aim for twenty per category rather than a bigger pile overall.
Can I use a model to grade another model's output?
Yes, and for open-ended text it is often the only practical option. Validate it first: grade thirty cases by hand, have the judge grade the same thirty, and check they agree. If agreement is poor, the rubric is ambiguous, not the judge.
What is the difference between an eval and a unit test?
A unit test asserts one exact outcome and fails on any deviation. An eval scores a distribution of acceptable outcomes and reports a rate. Treat eval results as a percentage that should not drop, not as a pass or fail on any individual case.
Do I need an eval if I am just using a chatbot manually?
No. Evals matter when a model is doing a job repeatedly and unsupervised. If a human reads every output before it goes anywhere, that human is the eval.
How did this land?
About the author
Senior Editor, AI & Product
Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.
