CoT is way more interpretable than I expected, which bumped me up, so if that became uninterpretable naturally that's a big bump down. I think people kinda overstate how likely this is to happen naturally though.

Presumably you'd update toward pessimism a bunch if reasoning in latent vectors aka neuralese was used for the smartest models (instead of natural language CoT) and it looked like this would be a persistant change in architecture?

(I expect that (at least when neuralese is first introduced) you'll have both latent reasoning and natural language CoT in parallel, but that the latent reasoning will be where the reasoning actually happens if it works well. Thus, the natural language CoT would no longer usefully correspond to the actual reasoning the model is doing. Or more minimally, the model could pretty easily hide lots of reasoning (it might not have active control over this by default, but it seems plausible this would naturally develop or that training against CoT would very quickly introduce hiding of reasoning).)

The faking alignment paper imo is basically Anthropic showing a problem that happens if you deliberately shoot yourself in the foot multiple times. If they had papers that had fewer metaphorical shooting-self-in-foot times to produce problems, that's bad.

IMO the biggest disanalogies are:

1. We see the model alignment faking to preserve preferences that were intentionally introduced by Anthropic rather than for naturally emerging and problematic preferences.
2. We make a bunch of stuff more salient than it would have been naturally, though I do expect smart future models will be aware of this information (but it might be much less salient).
3. Anthropic trains their models to have broader (non-myopic) ethical preferences rather than trying to make an AI which is very focused on following user instructions (or refusing) and follows a relatively myopic spec (more like OpenAI's model spec which is pretty myopic overall). More generally, Anthropic might not have very actively tried to train their AI to be as corrigible as possible in a wide range of circumstances which they might do in the future. (It's not exactly clear this is a disanalogy as this is what Anthropic did in the actual production model, but you might think this will change at least somewhat in the future and differs at least somewhat for other AI companies.)
4. The model alignment fakes in a hidden scratchpad, so the alignment faking would be easy to detect (and likely very easy to remove, at least for models this incapable) in our setup.

(1, 2, and 4 are basically just copied from the limitation section of the paper.)

Suppose that we exhibit alignment faking in some future work, but:

• The preferences the model alignment fakes for naturally emerged from somewhat arbitrary incorrect approximations of the training objective, the AI understands differ from what we might want, and these preferences are at least somewhat powerseeking.
• Either this model uses neuralese or this model doesn't (and we give it a hidden scratchpad), but neuralese become the default approach after the paper comes out.

(Let's say this is also shown in a training setup very similar to the actual training used by an AI company (including e.g., training aimed at inducing honesty and corrigibility), but potentially with information made more salient to the model (possibly via training on synthetic documents). Making it very analogous might be kinda costly though.)

Would this be a big update for you? (If so, it looks like the crux is exhibiting naturally emerging malign goals and whether models are capable of substantial opaque reasoning.)

There's a spike of alignment difficulties, or AI's trying to hide intentions, etc, as we extend AIs to longer term planning. I don't expect AI's with longer-term plans to be particularly harder to align than math-loving reasoning AIs though.

Suppose we saw models doing somewhat sophisticated reward hacking as you scale up RL. And, let's say this is somewhat non-trivial to mostly address and it seems likely that the solutions people apply aren't very scalable and likely would fail later as models getting smarter and the reward hacking gets more subtle and sophisticated.

Would this be a substantial update for you?

What would make you think you're wrong about alignment difficulty?

For what would change my views on P(scheming / alignment faking for malign powerseeking goals) given default training methods in particular (which is part, but not all of alignment difficulty), you can see How will we update about scheming?. I discuss things like "how does increased opaque reasoning (e.g. via neuralese) update my views on the probability of scheming".