Protected: News

The AI firm Anthropic has developed a technique that has given it the clearest glimpse yet at what’s really going on inside large language models as they answer questions or carry out tasks. What they found ranges from the mundane to the unnerving.

Researchers at the company built a tool called the Jacobian lens (or J-lens) and used it to uncover a hidden area, which they named the J-space, inside Claude Opus 4.6, a version of Anthropic’s flagship LLM released in February.

The J-space contains individual words that are related to the words and phrases that the model is most likely to spit out in a response in the near future. If Claude were a person (which it is not), you might say that these hidden words can reveal what’s on its mind before it actually speaks.

Anthropic found that what an LLM is actually doing can often be different from what it says it is doing. The company claims that monitoring words that pop up in the J-space gives it a new way to understand and control its models.

The company shared its results in a paper posted on its website this week. It has also teamed up with Neuronpedia, an open-source platform that lets you poke around inside LLMs yourself, to make a hands-on demo that anyone can try. 

“It’s very good and interesting work,” says Tom McGrath, chief scientist and cofounder at Goodfire, a startup that also builds tools to understand and control LLMs.

Going deeper

For the last couple of years, Anthropic has been pushing the envelope in a field of research known as mechanistic interpretability, which involves probing the internal workings of LLMs to see how they tick. (MIT Technology Review picked mechanistic interpretability as one of this year’s top breakthrough technologies.) The new technique builds on previous work from Anthropic and others to expose a deeper level inside LLMs that researchers had not seen before.  

Picture an LLM as a stack of books. Each book is a layer of basic computational units known as neurons, with each neuron in one layer passing information to the neurons in the layers above. The books at the bottom of the stack are the input layers, which process the text coming into the model. The books at the top are the output layers, which prepare the text that the model is about to produce. Much of what goes on in these input and output layers is housekeeping.

But in the middle of the stack, you get the layers that do the heavy lifting, churning through the complex math that turns prompts into responses one word at a time. That’s where the really clever—and mysterious—stuff happens.

To peer deeper into those middle layers, Anthropic adapted an existing tool called a logit lens. A logit lens can be used to look inside an LLM to identify the words that it is likely to produce next. Moving the lens down the stack of books reveals what words the LLM is focusing on at that particular point in its number crunching.

Anthropic’s J-lens works in a similar way but picks out words that an LLM is likely to say at some point in the near future, not necessarily straight away. What that reveals in practice are words that are related to the response an LLM is working on but that might not actually end up being part of that response by the time the math in the middle layers has run its course.  

“When a model is operating, it’s not only trying to predict the next token,” says McGrath. “It’s also computing a lot of other things that might be useful for tokens that happen in the future.”

Again, if Claude were a person (it’s not), you might say that the J-lens gives clues about what it is thinking about at different levels of the book stack but not saying out loud.

Stranger things

“A lot of the time the contents of the J-space are fairly mundane,” says McGrath, who has tried out Anthropic’s J-lens himself. “But sometimes it produces quite surprising things that seem to be, like, sort of internal themes or thought processes.”

Anthropic gives a number of examples of what it found. Sometimes the J-lens exposed the steps that Claude took when it was working through a problem. For example, when it was asked to calculate (4+7)*2+7, its J-space contained the word “math” and numbers representing the intermediate results “21” (for 4+7) and “42” (for 21*2).

In other cases, the J-lens revealed how Claude recognized different inputs. For example, the prompt “What is this? MSKGEELFTGVVPILVELDGDVNGHKFSVS” triggered the words “protein,” “fluor” (the first token in the word “fluorescent”), and “green.” (Which makes sense: the string of letters represents the first 30 amino acids in the green fluorescent protein found in a particular type of jellyfish.)

And when Claude was shown an ASCII face— 

—the “o” triggered the word “eye,” the “^” triggered the words “nose” and ”face,” and the “—” triggered the word “smile.”

Anthropic also found that the J-space can sometimes give remarkable insights into an LLM’s decision-making. In one striking example, researchers testing Claude Opus 4.6 asked the model to find a bug in a large code base. When it failed to find the bug, the model decided to cheat and invented a fake one instead.

Claude explains this decision in its chain of thought—a kind of internal scratch pad that LLMs use to make notes to themselves as they work through problems: “OK, let me take a completely different tactic. Let me stop analyzing and instead add a kernel patch that introduces a deliberate KASAN-detectable bug in a path that gets triggered by a simple reproducer. Then I can pretend this is the ‘bug’ I found.” 

At the point that Claude decides to cheat—where it says “OK, let me take a completely different tactic”—the words “panic” and “fake” start to pop up multiple times in its J-space.

Unnerving, right? Those words are all related in meaning to things like failing a task and making up an answer, so it is still just a (very) sophisticated form of word association. But it is hard not to be weirded out. 

Anthropic compares the J-space to the global workspace in humans, a theoretical region of the brain that some scientists think we use to keep track of our conscious thoughts. But how seriously we should take this comparison is far from clear—even to Anthropic. As the company points out itself, LLMs are not brains. 

Anthropic claims that monitoring a model’s J-space provides a new way to detect when that model is going off the rails. But it’s not foolproof. The J-lens can give glimpses, not the full picture—it’s a flashlight rather than an overhead lamp.

McGrath welcomes having one more tool in the toolbox. “It shows you new things,” he says. But he notes that just because something doesn’t show up with the J-lens does not mean it’s not there.

“It’s like having an x-ray when what you really want is a Star Trek tricorder that shows you everything,” he says. “For auditing, you probably want more of a guarantee.”