Summary and Table of Contents

The goal of this post is to discuss the so-called “sharp left turn”, the lessons that we learn from analogizing evolution to AGI development, and the claim that “capabilities generalize farther than alignment” … and the competing claims that all three of those things are complete baloney. In particular,

Section 1 talks about “autonomous learning”, and the related human ability to discern whether ideas hang together and make sense, and how and if that applies to current and future AIs.
Section 2 presents the case that “capabilities generalize farther than alignment”, by analogy with the evolution of humans.
Section 3 argues that the analogy between AGI and the evolution of humans is not a great analogy. Instead, I offer a new and (I claim) better analogy between AGI training and, umm, a weird fictional story that has a lot to do with the evolution of humans, but it’s definitely not evolution, I swear. I draw some lessons from this improved analogy, including reasons for both reassurance and concern about an AGI “sharp left turn”.
Section 4 presents the opposite case that “alignment generalizes farther than capabilities”.
Section 5 tries to reconcile the two competing claims, partly in terms of different assumptions regarding the level of AI autonomy versus supervision. I wind up mostly pessimistic, but suggest some approaches that might work.

By the way, I categorized this post as part of the 2023 lesswrong annual review, as it’s kinda a response to, and riff on, the following:

All these posts are well worth reading, but hopefully this post will be self-contained. There are othe...

2023 Longform Reviews