Who Tests an AI Model Before It Is Released?
Four groups test a frontier model before it ships, and none of them checks whether it is any good at your work. Knowing what was covered tells you what is left for you.
Who tests an ai model before it is released? Four groups, with four different jobs, and none of them is testing the thing you care about. The vendor runs internal capability and safety evaluations, a published safety framework sets thresholds that gate deployment, external red teamers attack the model under contract, and in the EU a legal regime now requires evaluation and adversarial testing for the largest models. What none of them do is check whether the model is any good at your work, on your data, for your users.
That gap is the practical reason to read this. Knowing what was tested tells you precisely which risks remain yours.
Layer one: the vendor's own safety framework
The frontier labs publish policies that commit them to evaluate before deploying, and those policies are public documents rather than marketing. Anthropic's Responsible Scaling Policy is the clearest worked example: a tiered framework, loosely modelled on biosafety levels, where each AI Safety Level carries stricter safety, security and operational requirements, and where models are placed by their potential for catastrophic harm.
The gating commitment is the part that matters. At the higher level the framework commits not to deploy a model that shows meaningful catastrophic misuse risk under adversarial testing, and specifies testing by world-class red teamers rather than internal review alone. The framework also implies a pause: if capability outruns the safety procedures, training of more powerful models waits.
Two honest caveats. The announcement describes its own commitments as an early iteration expected to be revised, so the current published version is what binds today rather than the original text. And the scope is narrow by design: the policy addresses catastrophic risk, which it defines as an AI model directly causing large-scale harm. It is not a quality assurance programme.
Layer two: what internal evaluation covers
Internal pre-release testing is broad but categorical. It asks whether the model will help someone build a weapon, whether it can act autonomously in dangerous ways, whether it refuses what it should refuse and complies with what it should allow, and how it performs on recognised capability benchmarks.
Those are population-level questions. They are answered against curated evaluation sets, and a model that passes has been shown not to be dangerous in the ways the evaluators thought to test. That is worth a great deal and it is not the same as being correct about your domain.
Layer three: external red teams and third parties
External testers are brought in precisely because internal teams develop blind spots about their own system. Their output is usually summarised rather than published in full, which means you can see that adversarial testing happened without seeing what it found.
The published artefact is the system card, and it is the most useful document in this whole chain for a builder. It is where evaluation results, known weaknesses and deployment conditions are written down. If you only read one thing before adopting a model, reading the model's system card is a better use of twenty minutes than any launch post, and what a model card contains covers the shorter sibling document.
Layer four: the legal floor in the EU
Pre-release testing is no longer purely voluntary for the biggest models. Article 55 of the EU AI Act sets obligations on providers of general-purpose AI models with systemic risk, and they read like a testing specification.
Providers must "perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art", and must carry out "adversarial testing of the model with a view to identifying and mitigating systemic risks", keeping records of that testing. They must also "keep track of, document, and report, without undue delay, to the AI Office" relevant information about serious incidents and corrective measures, and "ensure an adequate level of cybersecurity protection for the general-purpose AI model", covering its physical infrastructure.
Note the threshold. These duties attach to models with systemic risk, not to every model and not to your application built on one. Compliance can be demonstrated through codes of practice until a harmonised standard exists.
What each layer does not cover
Layer | Answers | Does not answer |
|---|---|---|
Safety framework | Is this model catastrophically dangerous? | Is it accurate on your domain? |
Internal evaluation | How does it score on known benchmarks? | How does it score on your tasks? |
External red teaming | Can an attacker make it misbehave? | Will your own users trip it accidentally? |
EU Article 55 | Did the provider evaluate and document? | Anything about your deployment |
System card | What are the known weaknesses? | Which matter for your use case |
Read the right-hand column as your job description. Everything in it is a risk that pre-release testing deliberately does not address, because nobody outside your company knows your workload.
The part that is yours
Build a small evaluation set from your own real inputs, including the awkward ones, and score the model on it before switching.
Test the failure path, not just the happy path. What your product does with a wrong answer is a product decision, not a model property.
Check the deployment conditions in the system card against how you actually plan to use the model.
Re-run your evaluation after any model change, including an alias quietly resolving to something newer.
None of this is exotic, and it is the difference between adopting a model and hoping. It is also how you tell a genuinely significant release from a well-marketed one, which is the subject of how to tell if a model release is a big deal and of reading a release note without the hype. The wider map of what can go wrong, and who owns each part, is in AI risks.
FAQ
Does any government test models before release?
Not as a general precondition of launch. The EU regime requires the provider to evaluate, document and report rather than requiring a state body to test first. Several countries have safety institutes that evaluate models, often with voluntary cooperation, which is different from a licensing gate.
Can I see the red team results?
Usually only in summary, inside the system card. Full adversarial findings are rarely published because they double as an attack manual. Treat the summary as evidence the work happened, not as a list of everything found.
If a model passed safety testing, is it safe for my product?
No, and the frameworks do not claim it. Passing means it did not cross a catastrophic-risk threshold. Accuracy in your domain, behaviour with your users, and what your interface does with a confident wrong answer are all untested by anyone but you.
Does any of this apply to open-weight models I run myself?
The published framework commitments belong to the provider that trained the model, and once weights are in your hands the deployment choices, including guardrails, are yours. Check whether an evaluation report was released alongside the weights, because many are, and read it as a starting point rather than a clearance certificate.
How did this land?
About the author
Senior Editor, AI & Product
Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.
