What Is Interpretability in AI? A Plain-English Guide
Interpretability is the attempt to look inside an AI model and see what it is computing. Here is what researchers have actually found, and the limits.
Interpretability is the work of looking inside an AI model to see what it is computing, rather than judging it only by what it outputs. A model is a huge pile of numbers. Interpretability tries to turn that pile into something a person can read: which concepts the model tracks, how they connect, and why a given answer came out.
If you build with AI, this matters for a practical reason. When a model gives a wrong answer, you normally have two tools: change the prompt, or change the model. Interpretability is the research field trying to give you a third, which is to find out what actually went wrong inside.
The three things people mean by "explaining" a model
The word gets used loosely, so it helps to split it into three levels.
Explaining outputs. Asking the model why it said something. Cheap and easy, and the least trustworthy, because the model is generating a plausible account rather than reporting its internals.
Finding features. Identifying patterns of internal activity that correspond to a concept, such as a city, a coding bug, or a flattering tone.
Tracing circuits. Following how features feed into each other, step by step, to produce one specific answer.
Most of what gets called interpretability in news coverage is level one. Most of what researchers mean by it is levels two and three.
Why you cannot just read the weights
A neural network stores what it learned as billions of numbers. A transformer model uses those numbers to turn text into more numbers, layer after layer. No single number means anything on its own. A concept like "the Golden Gate Bridge" is not stored in one neuron. It is spread across many, and each neuron also contributes to many unrelated concepts.
That is the core difficulty. Looking at neurons one at a time is like trying to read a book by examining individual ink molecules.
What Anthropic found when it looked
The first study is more than two years old, but it is still the clearest public example, so it is worth reading closely. In Mapping the Mind of a Large Language Model, published on 21 May 2024, Anthropic used a technique it calls dictionary learning on Claude 3.0 Sonnet. The idea is to find recurring patterns of neuron activity and treat each pattern as one "feature". The team reported extracting millions of features from the model's middle layer.
Two details from that study are useful to remember.
Features are concepts, across languages and media. A feature tied to the Golden Gate Bridge activated on English and non-English mentions of the bridge and on an image of it.
Features can be turned up. When Anthropic amplified that feature, Claude began describing itself as the bridge. When asked about its physical form, it answered that it was the iconic bridge itself.
That second result is the important one. It shows a feature is not just a label a researcher stuck on afterwards. Changing it changes the model's behaviour in the direction the label predicts.
Tracing a whole answer
Features alone do not explain an answer. In Tracing the Thoughts of a Large Language Model, published on 27 March 2025, Anthropic followed chains of features through Claude 3.5 Haiku on simple tasks. Three findings stand out:
Behaviour | What the tracing showed |
|---|---|
Writing a rhyming line | The model picked candidate rhyme words before writing the line, so it planned ahead rather than improvising word by word |
Working in several languages | Shared features existed across English, French and Chinese, and the shared portion grew with model scale |
Mental arithmetic | It ran parallel paths, one estimating the rough answer and one fixing the last digit, yet when asked it described the schoolbook carry-the-one method |
The arithmetic row is the one to dwell on. The explanation the model gave for its own work did not match what the tracing found. That is direct evidence for why asking a model "how did you get that?" is not interpretability.
What interpretability cannot do yet
Anthropic is candid about the limits, and they are the part most summaries skip.
The features found are, in its words, a small subset of everything the model learned. Finding a complete set with current techniques would cost more compute than training the model did.
Knowing what a model represents does not show how it uses those representations. The authors say researchers still need to find the circuits involved.
Even on short prompts, the circuit-tracing method captures only a fraction of the computation. It can also produce artifacts that do not reflect the underlying model.
Understanding a single circuit currently takes a few hours of human effort, even on prompts only tens of words long.
So the honest summary is that interpretability has produced real, checkable results on narrow questions, and nothing close to a full readout of a model.
What this means if you build with AI
You will not run these tools yourself. The practical consequences are about how much to trust what you are told.
Do not treat a model's self-explanation as evidence. If it justifies an answer, that justification is more output to verify. The arithmetic example shows the account can differ from the process.
Test behaviour, not stories. An AI eval on your own cases tells you more than a persuasive explanation.
Read safety claims for what they measured. A system card that cites internal-feature findings is making a narrower claim than "we understand the model".
Link it to alignment. Interpretability is one of the ways researchers hope to check that a model's goals match what was intended, which is the subject of model alignment.
For the wider picture of how these systems are built, the pillar guide on how AI models work covers training, inference and where interpretability fits.
Frequently asked questions
What is the difference between interpretability and explainability?
The terms overlap and are used inconsistently. Explainability usually means producing a human-readable reason for one output. Interpretability usually means understanding the model's internal mechanism. The first can be done from outside the model, the second needs access to its internals.
What is mechanistic interpretability?
It is the branch that tries to reverse-engineer a model into parts and connections, much as an engineer might reverse-engineer an unknown circuit. Finding features and tracing circuits, as in the two Anthropic studies above, are examples.
Can you interpret a closed model like ChatGPT?
Not from the outside. Feature and circuit work needs access to the model's internal activations, which only the developer or a partner with internal access has. Outsiders can test behaviour, not read internals.
Does interpretability make AI safe?
No, not by itself. It is a way of checking and understanding a model, which could help spot problems earlier. The studies cited here describe understanding, not guarantees.
How did this land?
About the author
Senior Editor, AI & Product
Cecilia leads the Swarmz editorial desk. She has spent a decade turning complex AI and product topics into writing people actually finish, and she owns the blog's quality bar.
