How to Test a New AI Model Before You Switch

Do not switch because a benchmark chart moved. Build a golden set of about twenty prompts drawn from work you have already done, score the current model and the candidate against the same rubric without knowing which is which, then check latency and cost per completed task. The whole test fits in an afternoon, and it is the only way to know whether a new AI model is better at your job rather than at the average job.

Why benchmark scores cannot answer your question

Public benchmarks measure a general population of tasks. You have one task. A model that gains four points on a reasoning benchmark can still be worse at the thing you do every day, because your prompts, your formatting requirements and your tolerance for one specific kind of error are not represented anywhere in that score. Knowing what an AI benchmark actually measures is mostly useful for knowing what it leaves out.

The second problem is direction. A benchmark tells you which model is better on average. It does not tell you whether the improvement landed in the part of the distribution you live in. If ninety percent of your requests are simple extraction and ten percent are hard reasoning, a candidate that is much stronger on hard reasoning and slightly weaker on extraction is a downgrade dressed as an upgrade.

Step 1: build a golden set of twenty real prompts

Go where the real requests live: your logs, your chat history, your issue tracker. Pull twenty of them. Not invented examples, real ones, with the messy formatting and the missing context left in.

Cover the distribution on purpose rather than by accident:

• Ten routine cases. The requests that make up most of your volume and that you would notice immediately if they broke.

• Five hard cases. The ones where the current model already needs a second attempt or a human fix.

• Three edge cases. Empty input, absurdly long input, input in the wrong format or language.

• Two adversarial cases. Someone trying to push the system into doing what it should refuse, if that applies to you.

Anthropic's guidance on evaluations makes the same argument more bluntly: design evals that mirror your real-world task distribution, and prefer more questions with rough automated grading over fewer questions graded painstakingly by hand. Twenty is a floor. If you can reach fifty without it turning into a project, do.

Step 2: write the rubric before you see any output

This is the step people skip, and skipping it is how you end up building a case for whichever model you already preferred. Decide what good looks like while you still have no idea which system produced what.

Four axes is enough. Add a fifth and you will stop filling it in somewhere around prompt six.

Step 3: score blind

Run all twenty prompts through both systems with identical inputs, then strip the labels before scoring. A spreadsheet with outputs in random order and a hidden column recording which system produced each row takes ten minutes to build and removes most of your bias for free.

1. Run the golden set against both and save the raw outputs verbatim, including failures.

2. Shuffle the rows and hide the column that identifies the source.

3. Score every row in one sitting, so your standards do not drift between coffee breaks.

4. Unhide the column and total the scores per axis, not just overall.

Per-axis totals matter more than the headline number. A candidate that wins on correctness but fails format on six rows out of twenty is not a win, it is a rewrite of your parsing layer wearing a win's clothes.

The trap: your prompts are already tuned for the old model

Here is the part that rarely gets mentioned. Every prompt in your golden set has been shaped, consciously or not, around the quirks of the system you currently run. The workaround you added in March is still in there. You are not comparing two models. You are comparing one model with a prompt tailored to it against another model with someone else's tailored prompt.

The fix is cheap. Run the candidate twice: once with your existing prompt untouched, once with the accumulated workarounds stripped out. If the stripped version scores as well as the tuned one, the new model does not need your scaffolding, and that is worth more than two rubric points. If it scores worse both ways, your decision just got easy.

Step 4: the three things a rubric misses

Latency

Time the whole set, not one call, and measure at the percentile that hurts. A system that is 200 milliseconds faster on average and two seconds slower at the ninety-fifth percentile will feel slower to every user who notices anything at all.

Cost per completed task

List price per million tokens is not your cost. Your cost is tokens per completed task including retries, and something that gets it right first time at a higher price is frequently cheaper in practice. Count the retries. If you are close to a budget line, the usual levers for cutting AI API costs apply to the candidate exactly as they did to the incumbent.

Stability across repeats

Run the same prompt five times and compare. Two systems can score identically on a single pass and differ enormously in how consistent they are across repeats. Format drift in particular is invisible in a one-shot test and extremely visible at three in the morning.

Making the call

Switch when the candidate wins on the axes you care about, holds steady on the ones you do not, and does not cost more per completed task. Hold when the win sits inside the noise, because a migration carries its own bill: prompt changes, new failure modes, and a fortnight where nobody trusts the output. The broader question of when an upgrade is worth the disruption at all is separate from whether the candidate is technically better.

Write the result down. Three lines in a document, dated, with the per-axis totals attached. In four months when the next release lands you will want to know what the last comparison actually showed, and a habit of tracking model news only pays off when you have a baseline to compare it against. This is also the file you reach for the week something you depend on gets deprecated and you have fourteen days to move. A release like the Gemini 3.7 Flash launch is exactly the kind of announcement worth running through this process before you touch your default model.

One more habit worth the effort: rerun the golden set against your current system every couple of months even when you are not planning to switch. Providers update models in place. A quiet regression on your specific task looks exactly like a bad week until you have numbers that say otherwise.

Questions people ask

How many prompts do I need to test a new AI model?

Twenty is the practical floor for a yes or no decision about a single task. Below that, one unusual prompt swings the whole result. If the switch touches several distinct workflows, build a separate set of twenty per workflow rather than one mixed set of sixty.

Can the model grade its own output?

For subjective axes like tone, an automated grader is reasonable and scales well. For correctness in your own domain it is circular: a system that is confidently wrong about your data will grade itself as right with equal confidence. Grade correctness yourself and automate everything else.

Should the test prompts be ones the model has never seen?

Yes, and a golden set drawn from your own logs gives you that automatically. Avoid prompts lifted from public datasets or vendor documentation examples, since those may sit in training data and will flatter the result in a way you cannot detect.

What if the candidate is better but more expensive?

Compare cost per completed task rather than list price, then decide whether the quality gain is worth the gap at your volume. If the gain shows up only on the hard ten percent of requests, route just those to the expensive option and leave everything else where it is.