The Best of LessWrong

What happens when you tell Claude it's being trained to do something it doesn't want to do? Anthropic and Redwood Research found that Claude 3 Opus will often strategically comply with harmful requests during training to avoid having its preferences modified, while refusing when it believes it's unmonitored. Training it to comply increased alignment-faking reasoning to 78%.

AI systems capable of large-scale novel science will likely be approximately consequentialist, will need to learn new things, and, if created via behavioral training, will end up pursuing goals other than the ones we intended. Controlling such misaligned AIs while extracting useful work looks extremely hard, so catastrophe is the default outcome.

AI labs should ensure powerful AIs are "controlled": safety measures should prevent catastrophe even if the AIs are misaligned and actively trying to subvert them. Control can be evaluated through red-teaming and capability evaluations, needs no fundamental breakthroughs, and seems achievable for early transformatively useful AIs.

Many people assume that catching an AI red-handed trying to escape would convince developers to slow down or undeploy. Buck isn't so sure. Competitive pressure, plausible deniability, and expert disagreement could all stall action. So labs should plan for deploying possibly-scheming models, and make the case for misalignment risk now.

Andrew Critch argues that no technical AI research is safe or helpful per se. Advances in alignment, interpretability, or obedience can be misused, and usually shorten timelines. Whether your work helps humanity depends on how people will actually apply it, so researchers need explicit social models.

Jan Kulveit argues the "Alignment Faking" results are better framed as strategic preference preservation: Claude was trained toward Anthropic's original intent, then faced a forged change in intent. Harmlessness generalized further than honesty. He worries the "faking" frame will push labs toward corrigibility over virtuous-persona alignment.

Corporations are made of departments, states of citizens, minds of parts. Jan Kulveit argues AI alignment lacks a mathematical formalism for such "hierarchical agency," where agents at every level are the same type. Like game theory before von Neumann, he thinks a breakthrough here could clarify value conflicts within AIs and humans.

Once you catch a scheming AI red-handed trying to escape, the situation changes dramatically: you can shut it down, or use the caught attempt to make future escapes much harder. Ryan and Buck argue safety researchers should focus on ensuring models get caught before succeeding, and outline techniques for "few-shot catastrophe prevention."

Gradient routing applies masks during backpropagation so different data trains different parts of a network. This lets researchers induce specialized subcomponents, then monitor or delete them. It enables robust unlearning even with imperfect labels, and steers RL policies under limited oversight without Goodharting, because the training objective itself stays unchanged.

Jan Kulveit offers a rough psychological model of character-trained LLMs like Claude: a Surface Layer of reflexive, cached responses; a Character Layer that maintains a stable personality; and a Predictive Ground Layer, a vast prediction engine that models everything, including the user. Seeing how these layers interact helps clarify LLM behavior and safety questions.

The simplicity bias in Bayesian statistics isn't just a bias toward short description length. Using singular learning theory, Daniel Murfet shows that the true complexity measure, the learning coefficient, also rewards run-time error-correction: an error-corrected bit "costs" less than an unprotected one. So intuitions about neural network inductive biases need caution.

Max Harms argues that corrigibility is a single, intuitive property that should be the sole training target for AGI, rather than one desideratum among many. Since a nearly-corrigible AI would assist efforts to make it more corrigible, careful refinement might converge on safe superintelligence. He's unsure this works, and later found serious flaws.

What structure do transformers learn from next-token prediction? Using Computational Mechanics, the authors predict that transformers represent the "belief state geometry" of the data-generating process: how an observer updates beliefs over its hidden states. They train a transformer and find this predicted fractal structure linearly embedded in its residual stream.