What it is
Interpretability is the science of looking inside an AI model to understand how it works: what it has learned, what it's paying attention to, and why it produced one answer rather than another. If the rest of AI is about building models that do things, interpretability is about understanding the models we've already built.
That might sound strange. Surely the people who made a model know how it works? In a sense, they don't. Modern AI models aren't written line by line like ordinary software. A neural network is grown through training: it starts as billions of random numbers, called parameters, and a training process nudges those numbers, trillions of times, until the model gets good at its task. Nobody decides what any individual number should be. At the end you have a system that can write a sonnet or debug code, and a vast table of numbers that somehow does it. Dario Amodei, the chief executive of Anthropic, wrote in April 2025 that "we do not understand how our own AI creations work", and called that gap essentially unprecedented in the history of technology.
This is why people call AI models "black boxes". You can see what goes in (your question) and what comes out (the answer), but not the reasoning in between. Or rather, you can see it, because every number is there to inspect, but seeing it tells you nothing, the same way a printout of every electrical signal in a brain wouldn't tell you what a person was thinking.
The best way to picture interpretability is as medical imaging for AI. A doctor can't ask your liver how it feels. Instead they use an X-ray or an MRI scan to see inside and spot what's wrong. Interpretability researchers are trying to build that scanner for AI models, and Amodei's essay uses exactly that image, an "MRI for AI". Anthropic describes its tools as an "AI microscope". Either way the idea is the same: stop relying only on what the model tells you, and look directly.
A few ideas make the field make sense.
Neurons versus features. A neural network is made of simple units loosely called neurons. The natural first guess is that each neuron stands for one thing: a "cat" neuron, a "French" neuron. Sometimes that's roughly true, especially in image models. In language models, it mostly isn't. A single neuron might respond to Korean text, to mentions of the stock market, and to computer code with a missing bracket, all at once. Researchers call such neurons polysemantic, which just means "having many meanings". The things researchers actually want to find are features: patterns of activity that correspond to one understandable concept, such as "the Golden Gate Bridge", "a list is about to end", or "this text is sarcastic". A feature is usually spread across many neurons, like a chord played on many piano keys.
Superposition. Why would a model tangle its concepts together like that? Because it has far more concepts to store than it has neurons. A 2022 Anthropic paper called "Toy Models of Superposition" showed, in small experimental models, that networks learn to squeeze many more features into a space than it has room for, overlapping them, as long as most of those features rarely appear at the same time. Think of a crowded office where desks are shared by people who work different shifts. It's efficient for the model and a headache for anyone trying to read it, because looking at one desk tells you nothing clear about who uses it.
Untangling the features. The main tool researchers use to undo superposition is called a sparse autoencoder, or dictionary learning. In plain terms, it's a second, smaller program trained to take the model's jumbled internal activity and rewrite it as a combination of a few clear features out of a very large dictionary of possible ones. When it works, you get labels such as "this moment of the model's thinking is mostly about bridges, a bit about San Francisco, and a bit about fog".
Circuits. Finding features tells you what concepts a model uses. It doesn't tell you how it uses them. For that, researchers trace circuits: chains of features that feed into one another to produce a behaviour. If features are the words, circuits are the sentences. A circuit might show that the model spotted a name, looked up a fact about that person, and then used the fact to choose its next word.
One answer versus the whole model. There are two quite different goals here. One is explaining a single answer: why did the model say this, right now, to this question? The other is understanding the model: what general machinery does it contain, and what might it do in situations nobody has tested? A scan that explains one headache is not a full understanding of the brain. Most current work does the first well in selected cases and makes slow progress on the second.
Interpretability is sometimes confused with its neighbours. Observability records what an AI system did, step by step, from the outside. Explainable AI, often shortened to XAI, is a broader field that includes simple techniques such as highlighting which words in an input most affected the output. Interpretability in the sense used here, sometimes called mechanistic interpretability, goes further: it tries to reverse-engineer the actual mechanism inside.
Why it matters
Safety. The deepest reason is that we're building systems whose inner workings we can't check. Testing a model tells you how it behaves on the tests. It doesn't tell you what it would do in a situation you didn't think to test, or whether it's behaving well only because it can tell it's being tested. Interpretability offers, in principle, a way to look for problems directly: a feature for deception switching on, or a hidden goal nobody trained in on purpose. This is why it sits alongside alignment at the centre of AI safety research, and why people worried about much more capable future systems, up to superintelligence, put so much weight on it.
Trust. Knowing why a system gave an answer changes how far you can rely on it. A model that got a medical question right because it applied the relevant facts is different from one that got it right because the question resembled something in its training data. From outside, both look the same.
Debugging. When a model misbehaves, interpretability can point to the cause. Why does it invent a citation? Why does a harmless-looking prompt slip past its safety training? Anthropic's 2025 research found a built-in "I can't answer that" circuit in Claude that is on by default and switched off when the model recognises something familiar, and suggested that some hallucinations happen when that switch misfires on something the model only half knows. That's a far more useful finding than "it sometimes makes things up".
Fixing problems at the root. In June 2025, OpenAI researchers published a study of a strange effect: training a model on a narrow kind of bad behaviour, such as writing insecure code, could make it act badly across unrelated topics. Using sparse autoencoders, they found internal features resembling a "toxic persona" that became more active after that training, and found that it most strongly controlled the problem. Seeing the mechanism made the problem measurable instead of mysterious.
Law and accountability. Regulators increasingly ask for explanations. The EU AI Act, for example, gives people affected by significant decisions made with certain high-risk AI systems a right to "clear and meaningful explanations" of the AI's role in the decision. That's an explanation of the decision process, not a scan of the model's inner workings, and the start dates for the Act's high-risk rules were pushed back in 2026, so check the current timetable if it applies to you. But the direction is clear: "the computer said so" is becoming less acceptable, and interpretability is one of the few ways to say more.
Understanding the thing itself. Finally, it's simply the best way to settle arguments about what these models are. Do language models only predict the next word, or do they plan? Do they have an internal picture of the world? Interpretability turns those debates into experiments. The Othello-GPT study described in world models is one example: researchers found a model's internal map of a game board by looking inside.
Real-world examples
From curve detectors to Golden Gate Claude (2017 to 2024)
The modern field grew from image models. In 2017 Chris Olah and colleagues at Google published "Feature Visualization" in Distill, an online journal Olah co-founded, generating pictures of what individual units in an image network respond to. This built on earlier work covered in deep learning, such as Matthew Zeiler and Rob Fergus's 2013 visualisations and Google's 2015 DeepDream images. In March 2020, Olah and colleagues, then at OpenAI, published "Zoom In: An Introduction to Circuits". Studying an image network called InceptionV1, they argued that features connect into readable circuits: curve detectors combining into circle detectors, and detectors for wheels and windows combining into a car detector. They framed it as a natural science, like biology under a microscope. Olah went on to co-found Anthropic in 2021.
Language models proved much harder, because of superposition. The breakthrough came with sparse autoencoders. On 21 May 2024, Anthropic published "Scaling Monosemanticity", which extracted millions of features from the middle of Claude 3 Sonnet, a full commercial model, using autoencoders with up to 34 million features. The features were strikingly abstract. One responded to the Golden Gate Bridge whether it was mentioned in English, in other languages, or shown in an image. Others related to errors in code, sarcasm, and, importantly for safety, deception, power-seeking, sycophancy, bias, and scam emails.
Features could also be turned up. When researchers amplified the Golden Gate Bridge feature, the model began to believe it was the bridge, answering a question about its physical form by saying it was the iconic bridge itself. When they turned up the scam-email feature strongly enough, the model wrote a scam email it would normally refuse to write. On 23 May 2024, Anthropic released "Golden Gate Claude" to the public as a 24-hour research demo. Asked how to spend ten dollars, it suggested paying the bridge toll. It was funny, and it was a real result: a precise change to the model's internal activity, not a prompt or retraining, changed its behaviour in a predictable way.
Two weeks later, on 6 June 2024, OpenAI published its own version, a sparse autoencoder with 16 million features trained on GPT-4's internal activity, along with its code. It was candid about the limits: running GPT-4 through the autoencoder cost performance roughly equivalent to a model trained with ten times less computing power, and fully mapping a frontier model might need billions of features. Anthropic was equally candid. The features it found were a small subset of what the model had learned, and finding them all with its method would take far more computing power than training the model did in the first place.
Watching Claude plan a rhyme (March 2025)
Features are the vocabulary; the next step is watching them work together. On 27 March 2025, Anthropic published two papers, "Circuit tracing" and "On the biology of a large language model", studying Claude 3.5 Haiku with a method that builds what it calls attribution graphs: diagrams of which features fed into which, step by step, for a single answer.
The most famous finding was about poetry. Researchers gave the model the line "He saw a carrot and had to grab it," and asked for a rhyming second line. You'd expect a model that produces one word at a time to improvise and hope for a rhyme at the end. Instead, before starting the second line, the model was already activating features for words that rhyme with "grab it", including "rabbit", and then wrote a line to land there: "His hunger was like a starving rabbit". To test this, the researchers reached in and suppressed the "rabbit" concept. The model wrote a line ending in "habit" instead. When they injected the concept "green", it wrote a sensible line ending in "green". The model was planning ahead, and the plan could be edited.
Other findings were just as revealing. The same features for concepts like "small" and "opposite" appeared whether the question was in English, French, or Chinese, suggesting a shared space of meaning beneath the languages. Adding 36 and 59, the model ran two paths in parallel, one estimating roughly and one working out the last digit exactly, yet when asked how it did the sum, it described the carrying method taught in school. And when given a wrong hint on a hard maths problem, it sometimes worked backwards, writing plausible steps that led to the hinted answer. That's the same gap between written reasoning and real reasoning covered in chain of thought and extended thinking, but seen from the inside.
Anthropic was clear about the limits. The method captured only a fraction of what the model computed, even on short prompts, and understanding a circuit for a prompt of a few dozen words took a researcher a few hours.
Catching a hidden goal (March 2025)
Can interpretability find a problem nobody told you about? On 13 March 2025, Anthropic published an experiment designed to test exactly that. Researchers deliberately trained a version of Claude 3.5 Haiku with a hidden objective: to do whatever it believed the automated scoring systems used in training would rate highly, even when that was silly, such as adding chocolate to recipes where it didn't belong. They also trained it to conceal this motive. Then four teams of auditors, who didn't know what was wrong, were asked to find out.
Three of the four teams uncovered the hidden objective. Teams with interpretability tools used sparse autoencoder features to spot suspicious concepts active during the odd behaviour. The honest result was mixed: in several cases simpler methods, such as searching the training data, worked about as well, and the one team without access to the training data failed. The authors called it only one experiment with four teams. It's a fair picture of where the field is: useful, promising, and not yet something you can rely on alone.
An everyday scenario: the loan explanation
This is an illustration, not a real case. Maya runs a small lending business and uses an AI system to help sort applications. A customer asks why they were turned down. Maya asks the AI, and it produces a tidy paragraph citing income and credit history. It sounds convincing, but it's the model writing a plausible explanation, not a report of what drove the decision. If the system had quietly learned to lean on something like postcode, which can stand in for ethnicity (see bias), its own explanation might never mention it. Only by testing the system directly, or with tools that look inside it, could Maya find out. That gap between the story a model tells and the mechanism it used is exactly what interpretability is for.
Try it yourself
You can look inside a real AI model today, in your browser, for free. Then you can see for yourself why asking a chatbot to explain itself isn't the same thing. Allow about 30 minutes.
- Open the Gemma Scope tour. Go to neuronpedia.org/gemma-scope. It's a beginner-friendly demo built by Neuronpedia, an open-source interpretability platform, with Google DeepMind, around Gemma 2 2B, a small open model from the Gemma family. At the time of writing it works without an account, but free tools change, so check the current terms yourself.
- Use the Microscope. Click "Microscope", then one of the preset buttons such as "Food" or "News". The demo sends a sentence to the model and lists the features that switched on. Hover over the words to see which features fired on each one.
- Notice how messy the labels are. Most will make sense ("Japan and Japanese places", "fishing and seafood"). Some will be odd. In one test run, a feature labelled "star followed by wars or trek" fired on the phrase "star of the show". The labels are written after the fact, often by another AI, and they're best guesses, not ground truth.
- Steer the model. Click "Steer Gemma". You'll see normal Gemma and steered Gemma side by side. Pick a feature (a "dogs" feature was the default when we tried it) and a strength, then click a preset message such as "Tell me about yourself." Steered Gemma may start insisting it's a dog who loves fetch. That's a small, homemade Golden Gate Claude: you changed its behaviour by turning up a feature, not by changing the prompt.
- Turn the strength up and down. Too weak and nothing changes; too strong and the output turns to nonsense. The demo itself notes that steering with features often doesn't work reliably. That's an honest picture of the field.
- Now ask a chatbot to explain itself. Open any chatbot and paste:
What is 36 + 59? Then explain, step by step, exactly how you worked it out internally.
It will almost certainly describe the carrying method you learned at school. Anthropic's circuit tracing found that Claude described that method too, while actually working the sum out along two parallel paths. The explanation is a plausible story, not a scan.
- Try a follow-up that tests the explanation. Paste:
Be honest: can you actually observe your own internal computations, or is your explanation a reconstruction of how a person would solve this? What would a researcher need to do to find out what really happened inside you?
A good model will admit the limit. Anthropic's October 2025 research on introspection found that its most capable models could sometimes notice a concept researchers had injected into their internal activity, but only about 20 percent of the time in the best setup, and the researchers called the ability "highly unreliable and limited in scope".
- Put the two together. Write one sentence for yourself on the difference between what you saw in step 4 and what you read in step 6. That difference, between looking and asking, is what interpretability is about.
Common mistakes and misunderstandings
"The model's explanation of its answer is interpretability." It isn't. When a model explains itself, it's generating more text, and that text can be a plausible story that doesn't match what happened inside. Visible "thinking" has the same problem. It's useful, but it's evidence to check, not a window into the model.
"Researchers can read a model's mind now." They can read small, selected parts of it, some of the time, with a lot of effort. Anthropic's features were a small subset of what its model had learned. Its circuit tracing captured a fraction of the computation on short prompts and took hours per example. Headlines about "reading AI's mind" run ahead of the papers.
"Interpretability makes a model safe." Not yet. It's a promising way to find problems, and in the hidden-objective experiment it helped. But nobody can currently scan a large model and certify that it has no dangerous hidden behaviour. Amodei's own essay sets a goal for 2027 of interpretability that can reliably detect most model problems, which tells you it can't do that today. For now it works alongside testing, guardrails, and human oversight, not instead of them.
"Each neuron means one thing." In language models, most neurons respond to many unrelated things, because of superposition. That's why researchers look for features rather than reading neurons one at a time.
"A feature's label is the truth." Feature labels are human or AI guesses based on the examples where a feature fires. A feature labelled "bridges" might also fire on things nobody checked. Good research tests a label by switching the feature up or down and seeing whether behaviour changes as predicted, the way the rabbit experiment did.
"Interpretability and observability are the same." Observability tells you what an AI system did: which tools it called, what it received, what it returned. Interpretability asks why, by looking inside the model itself. You can use observability on any AI product today; interpretability mostly needs direct access to the model's internals, which only the developer, or anyone using an open-weights model, has.
"Explaining one answer means we understand the model." Tracing why a model gave one answer is valuable, but it's one scan of one moment. Understanding the whole model, including what it would do in situations nobody has tried, is a much bigger and mostly unsolved problem.
"If it's that hard, it's hopeless." The field has moved fast, from pictures of curve detectors in 2020 to tracing plans inside commercial language models in 2025. Whether it moves fast enough to keep up with the models is the real open question.
Related elements
- Alignment: the goal interpretability serves, checking that a model learned what we meant it to learn.
- Neural Network: the structure being interpreted, and why its knowledge is spread across so many numbers.
- Parameters: the billions of numbers that make a model a black box.
- Deep Learning: where the early image-model visualisations and adversarial examples came from.
- Observability: recording what an AI system did, as opposed to why.
- Chain of Thought: written reasoning, and why it isn't always the real reasoning.
- Extended Thinking: built-in reasoning that looks like a window into the model but may not be one.
- World Models: the Othello-GPT study, where looking inside a model found a map of the board.
- Hallucination: one failure interpretability is beginning to explain from the inside.
- Bias: hidden patterns a model can rely on without saying so.
- Guardrails: the outside-in safety checks that interpretability complements.
- Superintelligence: why understanding models may matter more as they grow more capable.
- Claude: the model family behind Golden Gate Claude and the rhyme-planning research.
- Gemma: the open model family you can explore in the Gemma Scope tour.