james.lucassen

This post has an accompanying SPAR project! Apply here if you’re interested in working on this with me. Huge thanks to Mikita Balesni for helping me implement the MVP. Regular-sized thanks to Aryan Bhatt, Rudolph Laine, Clem von Stengel, Aaron Scher, Jeremy Gillen, Peter Barnett, Stephen Casper, and David Manheim for helpful comments. 0. Key Claims 1. Most alignment work today doesn’t aim for alignment that is stable under value-reflection[1]. 2. I think this is probably the most sensible approach to alignment. 3. But there is a threat model which could potentially be a serious obstacle to this entire class of alignment approaches, which is not currently being tracked or evaluated. It goes like this: 1. Long-horizon capabilities require a particular capability I call “stepping back” (described in more detail in the body of the post) 2. Stepping back seems likely to be learned in a very generalizable way 3. If stepping back generalizes in what seems like the simplest / most natural way, it will create a tendency for AIs to do value-reflection 4. If we’re unable to prevent this value-reflection, it will cause any AI whose alignment is not stable under value-reflection to become misaligned 4. However, I’m quite uncertain about whether or not this threat model will go through, for a few reasons: 1. The chain of argument is highly conjunctive and therefore fragile. 2. The track record of anthropomorphic reasoning about which AI capabilities will imply which other AI behaviors seems poor. 3. There is an intuitive counterexample provided by humans, at least at human-level intelligence. 5. Even given this uncertainty, work on this threat model seems very neglected given the wide swath of highly-invested-in alignment approaches that it could affect. 6. I don’t think further abstract argument about the likelihood of this threat model will reduce my uncertainty much. Instead, I propose some evals to keep track of each component of this thre