A research paper details how decomposing groups of neurons in a neural network into interpretable “features” may improve safety by enabling monitoring of LLMs

Neural networks are trained on data, not programmed to follow rules. With each step of training …

Anthropic 2023-10-08

Context & Ripple Effects

This work extends a longer effort to map neural-network representations, including earlier activation-atlas research on how networks organize visual concepts. Its distinctive move is to treat interpretable internal features as a possible safety-monitoring surface for language models.

It also complements OpenAI's earlier effort to use a model to interpret individual neurons and attention heads, while shifting attention from isolated units to groups of neurons and their features.

First-order effects

Anthropic gains a research framework for identifying internal model features that may be relevant to safety monitoring, rather than relying only on a model's outward responses.
LLM safety researchers get a concrete interpretability target—feature decomposition—to evaluate whether internal representations can reveal problematic behavior.

Second-order effects

Competing labs and interpretability-tool builders face pressure to show whether their monitoring methods can inspect meaningful internal features at useful scale, not merely probe model outputs.
If feature-level monitoring proves informative, it could become a complementary layer in evaluation workflows where behavioral safety tests alone leave blind spots.

Third-order effects

If the approach generalizes, AI assurance could shift toward combining output testing with evidence about a model's internal representations; its practical value will depend on whether those representations remain legible as models grow.
The work points toward safety claims being supported by more auditable technical measurements, rather than solely by training procedures or observed behavior.

The trend: LLM safety research is moving from black-box behavioral evaluation toward operational interpretability tools that could monitor models' internal mechanisms.

Discussion

@deliprao @deliprao on x
@natfriedman This one is very hebbian. Interestingly this is moving closer to probing (like RepE) than the OG mechanistic interpretability works.
@tunguz Bojan Tunguz on x
Or maybe you should build your ML algorithms from the ground up with, oh I don't know, decision trees, and then interpretability comes for free. 🤷‍♂️
@alexjlevenston Alex Levenston on x
seriously is this an anthropic psyop?
@mattredlon Matt Redlon on x
This is extremely important / super exciting work. As important, @AnthropicAI and @ch402 shared it. Once an idea is in the world, thousands of brilliant minds will run with it to the benefit of all of humanity.
@chinmaykak Chinmay Kak on x
The black box is not black anymore.
@native_mode @native_mode on x
The import of this thread, if proven valid, is immense. This is (potentially) a prodigious breakthrough.
@husainhz7 Husain Zaidi on x
Lobotomy is gonna come back in fashion (for digital neural networks only hopefully) Curious to know if there is some way to do it without impairing the smart capabilities of the network.
@_vltn Michelle Valentine on x
We'll look back and say this was up there with the Scaling Laws paper in terms of impact to what happens next in AI development
@pashakamyshev Pasha Kamyshev on x
This is pretty interesting research. A lot of specific neurons break down nicely and are understandable. This is not a full solution to alignment, which requires actually having a good philosophical concept of what you want, but it will prevent the most egregious “AI plans a...
@we4v3r Joshua Weaver on x
For people who worry about the “black box” nature of AI, this is huge. 👏
@advancedcskills @advancedcskills on x
Reminds me of going to dinner with a GSM packet engineer. (early 90s) What do you do for a job.? O.. I open packets and try to squash as much in there as possible and send them faster.. 🏃 In the following case.. Huge ramifications going forward.. https://twitter.com/...
@jeremydanielfox @jeremydanielfox on x
Really incredible work by Anthropic and Chris Olah. Congrats all. This is amazing and I really thank you for what you've done here.
@leeknowlton Lee Knowlton on x
Turning a research problem into an engineering problem is 🔥 and a huge win for interpretability. Also fascinating that features split but remain clustered as they scale in number. [image]
@nathanmcahill Nathan Cahill on x
This has to be one of the coolest papers I've seen this year.
@sherjilozair Sherjil Ozair on x
Linear probes are a classic way of grounding distributed representations! They were famously used as an evaluation protocol for unsupervised representation learning methods like CPC, SimCLR, etc. @gyomalin_ML wrote about it in 2016: https://arxiv.org/...
@yonashav Yo Shavit on x
it's beginning to feel a lot like wagmi
@ultrarareaf Zoe on x
fed the thread and visual components to gpt-4V and, [image]
@ch402 Chris Olah on x
It increasingly seems to me that the next big barrier in mechanistic interpretability will be an engineering one. If you are an engineer who wants to help us scale up this work, please consider applying! Your support could really accelerate interpretability right now.
@natfriedman Nat Friedman on x
RIP the phrase “giant inscrutable matrices of floating point numbers”
@anthropicai @anthropicai on x
The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
@nickcammarata Nick on x
in my opinion it's likely the most exciting time in mech interpretability is about to start to build great interfaces for studying models you want the preliminaries of clean neurons (this work) and short labels (feature vis for image, auto-interpretability labeling for language)
@brickroad7 Renji on x
This is earth-shattering news. The “hard problem” of mechanistic interpretability has been solved. The formal/cautious/technical language of most ppl commenting on this obscures the gravity of it. What this means -> not just AGI, but *safe* *superintelligence* is 100% coming🧵
@ch402 Chris Olah on x
If you'd asked me a year ago, superposition would have been by far the reason I was most worried that mechanistic interpretability would hit a dead end. I'm now very optimistic. I'd go as far as saying it's now primarily an engineering problem — hard, but less fundamental risk.
@neelnanda5 Neel Nanda on x
I'm excited to see this come out! All the recent excitement about SAEs seems great, and makes me optimistic that superposition is actually solvable, which seems like a really big deal for ambitious mech interp! I particularly enjoyed the deep dives into eg the Arabic feature [ima…
@jason_lopatecki Jason Lopatecki on x
I have a hot anti take: on Anthropic and Open AI's neuron level interpretability. I don't think mapping features to Neurons is the future of understanding models. Its insightful, but will fall way short. Neurons work together to map out surfaces in latent space.
@trentonbricken Trenton Bricken on x
Our paper is out! It feels like we've built a really powerful new microscope to see all sorts of incredible features and mechanisms in transformers for the first time e.g. finite state automata. I'm optimistic this work is scalable to real models and we're hiring so come help!
@richinseattle Richard Johnson on x
Fascinating progress on interpretability. It's possible to artificially stimulate a small number of precisely chosen weights to affect the semantic category of tokens chosen during inference.
@bentossell Ben Tossell on x
Anthropic's new research extracts a large number of interpretable features from a one-layer transformer. What does that mean? The neural networks in large language models show superposition. That means each neuron in the network represents more than one unique feature....
@sussillodavid David Sussillo on x
Using a sparse autoencoder to make sense of mixed representations. This is clever, and I wonder if it could be used for real neurons (if it hasn't been already in neuroscience, and putting aside that we have far less labels to regress to sparse encodings).
@tobyordoxford Toby Ord on x
One of the key sources of danger from modern AI systems is that the trained systems are giant inscrutable networks. This new paper by researchers at Anthropic makes significant progress in building automated tools to help us understand what AI systems are actually doing.
@brayden_mclean Brayden McLean on x
Even though it's hard to understand what neurons are doing in LLMs, we've learned that it's possible decompose LLMs into understandable “features” and published our results: https://www.anthropic.com/...
@sashadem Sasha de Marigny on x
A step forward on the path to breaking down complex neural networks into parts we can understand—and making AI systems more interpretable and safe. https://www.anthropic.com/...
@thebasepoint Joshua Batson on x
In writing this paper, there were countless features we thought might be bugs. After careful inspection, ~all of them revealed surprising and subtle model properties. To me this capacity for surprise is the true test of a new technique. This thread is about my favorite finding.
@andrew_sears Andrew Sears on x
Being able to inspect the compressed, encoded neurons of a LLM is really interesting! Finding different ways to deconstruct, compose and combine them will be the next wave of precision AI. I think there needs to be a biological or natural component to power it. At same level.
@anthropicai @anthropicai on x
We hope this will eventually enable us to diagnose failure modes, design fixes, and certify that models are safe for adoption by enterprises and society. It's much easier to tell if something is safe if you can understand how it works!
@hrishioa @hrishioa on x
Holy hell this is amazing - and this is the tweet that matters Not only can we understand neuronal structures and connect them to things, manipulating them produces the expected result. We are very early
@shahabbakht Shahab Bakhtiari on x
I love it every time @AnthropicAI mentions the link between their exciting work and prior work in neuroscience. It's not something you see frequently in AI these days
@xaviermalina Xavier Malina on x
I'm getting more and more excited about explainable AI, especially for LLM-powered systems. This new research from @AnthropicAI is one of the most exciting developments in a while. If anyone knows of similar papers, let me know!
@anthropicai @anthropicai on x
There's a lot more in the paper if you're interested, including universality, “feature splitting”, more evidence for the superposition hypothesis, and tips for training a sparse autoencoder to better understand your own network! 📄https://transformer- circuits.pub/...
@nottombrown Tom Brown on x
Progress on making the “inscrutable matrices” inside of Transformers more understandable! It seems like this technique is now “shovel ready” for engineers who want to work on scaling it up on frontier LLMs.
r/singularity r on reddit
AnthropicAI “demonstrates a method for decomposing groups of neurons into interpretable features” in Mechanistic Interpretability Breakthrough

Chronicles