Adrià Garriga-alonso

Circuit discovery through chain of thought using policy gradients

Circuit discovery has been restricted to the single-forward-pass setting, because the algorithms to attribute changes in behavior to particular neurons / SAE features need gradients, and you can't take a gradient through the sampled chain of thought. Or... can you? It turns out taking gradients through random discrete actions is...

Nov 29, 202515

Alignment will happen by default. What’s next?

I’m not 100% convinced of this, but I’m fairly convinced, more and more so over time. I’m hoping to start a vigorous but civilized debate. I invite you to attack my weak points and/or present counter-evidence. My thesis is that intent-alignment is basically happening, based on evidence from the alignment...

Nov 25, 202596

A scheme to credit hack policy gradient training

Thanks to Inkhaven for making me write this, and Justis Mills, Abram Demski, Markus Strasser, Vaniver and Gwern for comments. None of them endorse this piece. The safety community has previously worried about an AI hijacking the training process to change itself in ways that it endorses, but the developers...

Nov 7, 202515

A recurrent CNN finds maze paths by filling dead-ends

Work done as part of my work with FAR AI, back in February 2023. It's a small result but I want to get it out of my drafts folder. It was the start of the research that led to interpreting the Sokoban planning RNN. I was trying to study neural...

Sep 15, 202519

Illusory Safety: Redteaming DeepSeek R1 and the Strongest Fine-Tunable Models of OpenAI, Anthropic, and Google

DeepSeek-R1 has recently made waves as a state-of-the-art open-weight model, with potentially substantial improvements in model efficiency and reasoning. But like other open-weight models and leading fine-tunable proprietary models such as OpenAI’s GPT-4o, Google’s Gemini 1.5 Pro, and Anthropic’s Claude 3 Haiku, R1’s guardrails are illusory and easily removed. An...

Feb 7, 202537

Pacing Outside the Box: RNNs Learn to Plan in Sokoban

Work done at FAR AI. There has been a lot of conceptual work on mesa-optimizers: neural networks that develop internal goals that may differ from their training objectives (the inner alignment problem). There is an abundance of good ideas for empirical work (find search in a NN, interpret it), but...

Jul 25, 202459

Compact Proofs of Model Performance via Mechanistic Interpretability

We recently released a paper on using mechanistic interpretability to generate compact formal guarantees on model performance. In this companion blog post to our paper, we'll summarize the paper and flesh out some of the motivation and inspiration behind our work. Paper abstract > In this work, we propose using...

Jun 24, 2024104

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Adrià Garriga-Alonso

Adrià Garriga-Alonso

Causal Scrubbing: a method for rigorously testing interpretability hypotheses [Redwood Research]

Compact Proofs of Model Performance via Mechanistic Interpretability

Alignment will happen by default. What’s next?

Thomas Kwa's research journal

Adrià Garriga-Alonso

Causal Scrubbing: a method for rigorously testing interpretability hypotheses [Redwood Research]

Compact Proofs of Model Performance via Mechanistic Interpretability

Alignment will happen by default. What’s next?

Thomas Kwa's research journal

Circuit discovery through chain of thought using policy gradients

Alignment will happen by default. What’s next?

A scheme to credit hack policy gradient training

A recurrent CNN finds maze paths by filling dead-ends

Illusory Safety: Redteaming DeepSeek R1 and the Strongest Fine-Tunable Models of OpenAI, Anthropic, and Google

Pacing Outside the Box: RNNs Learn to Plan in Sokoban

Compact Proofs of Model Performance via Mechanistic Interpretability