Zac Hatfield-Dodds

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Text of post based on our blog post as a linkpost for the full paper which is considerably longer and more detailed. Neural networks are trained on data, not programmed to follow rules. We understand the math of the trained network exactly – each neuron in a neural network performs simple arithmetic – but we don't understand why those mathematical operations result in the behaviors we see. This makes it hard to diagnose failure modes, hard to know how to fix them, and hard to certify that a model is truly safe. Luckily for those of us trying to understand artificial neural networks, we can simultaneously record the activation of every neuron in the network, intervene by silencing or stimulating them, and test the network's response to any possible input. Unfortunately, it turns out that the individual neurons do not have consistent relationships to network behavior. For example, a single neuron in a small language model is active in many unrelated contexts, including: academic citations, English dialogue, HTTP requests, and Korean text. In a classic vision model, a single neuron responds to faces of cats and fronts of cars. The activation of one neuron can mean different things in different contexts. In our latest paper, Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, we outline evidence that there are better units of analysis than individual neurons, and we have built machinery that lets us find these units in small transformer models. These units, called features, correspond to patterns (linear combinations) of neuron activations. This provides a path to breaking down complex neural networks into parts we can understand, and builds on previous efforts to interpret high-dimensional systems in neuroscience, machine learning, and statistics. In a transformer language model, we decompose a layer with 512 neurons into more than 4000 features which separately represent things like DNA sequences, legal language, HTTP requests, Hebrew

289Oct 5, 2023

Zac Hatfield-Dodds

Message

Technical staff at Anthropic (views my own), previously #3ainstitute; interdisciplinary, interested in everything, ongoing PhD in CS, bets tax bullshit, open sourcerer, more at zhd.dev

3648

397

432

Anthropic's updated Responsible Scaling Policy

Today we are publishing a significant update to our Responsible Scaling Policy (RSP), the risk governance framework we use to mitigate potential catastrophic risks from frontier AI systems. This update introduces a more flexible and nuanced approach to assessing and managing AI risks while maintaining our commitment not to train...

Oct 15, 202438

Simple probes can catch sleeper agents

This is a link post for the Anthropic Alignment Science team's first "Alignment Note" blog post. We expect to use this format to showcase early-stage research and work-in-progress updates more in the future. Twitter thread here. Top-level summary: > In this post we present "defection probes": linear classifiers that use...

Apr 23, 2024133

Third-party testing as a key ingredient of AI policy

(nb: this post is written for anyone interested, not specifically aimed at this forum) We believe that the AI sector needs effective third-party testing for frontier AI systems. Developing a testing regime and associated policy interventions based on the insights of industry, government, and academia is the best way to...

Mar 25, 202411

Dario Amodei’s prepared remarks from the UK AI Safety Summit, on Anthropic’s Responsible Scaling Policy

I hope Dario's remarks to the Summit can shed some light on how we think about RSPs in general and Anthropic's RSP in particular, both of which have been discussed extensively since I shared our RSP announcement. The full text of Dario's remarks follows: Before I get into Anthropic’s Responsible...

Nov 1, 202385

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Oct 5, 2023289

Anthropic's Responsible Scaling Policy & Long-Term Benefit Trust

I'm delighted that Anthropic has formally committed to our responsible scaling policy. We're also sharing more detail about the Long-Term Benefit Trust, which is our attempt to fine-tune our corporate governance to address the unique challenges and long-term opportunities of transformative AI. Today, we’re publishing our Responsible Scaling Policy (RSP)...

Sep 19, 202385

Anthropic's Core Views on AI Safety

> We founded Anthropic because we believe the impact of AI might be comparable to that of the industrial and scientific revolutions, but we aren’t confident it will go well. And we also believe this level of impact could start to arrive soon – perhaps in the coming decade. >...

Mar 9, 2023173

Load More (7/9)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Zac Hatfield-Dodds

Zac Hatfield-Dodds

Zac Hatfield-Dodds

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Anthropic's Core Views on AI Safety

Simple probes can catch sleeper agents

Concrete Reasons for Hope about AI

Zac Hatfield-Dodds

Anthropic's updated Responsible Scaling Policy

Simple probes can catch sleeper agents

Third-party testing as a key ingredient of AI policy

Dario Amodei’s prepared remarks from the UK AI Safety Summit, on Anthropic’s Responsible Scaling Policy

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Anthropic's Responsible Scaling Policy & Long-Term Benefit Trust

Anthropic's Core Views on AI Safety

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Anthropic's Core Views on AI Safety

Simple probes can catch sleeper agents

Concrete Reasons for Hope about AI

Anthropic's updated Responsible Scaling Policy

Simple probes can catch sleeper agents

Third-party testing as a key ingredient of AI policy

Dario Amodei’s prepared remarks from the UK AI Safety Summit, on Anthropic’s Responsible Scaling Policy

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Anthropic's Responsible Scaling Policy & Long-Term Benefit Trust

Anthropic's Core Views on AI Safety