Quick book review of "If Anyone Builds It, Everyone Dies" (cross-post from X/twitter & bluesky):
Just read the new book If Anyone Builds It, Everyone Dies. Upshot: Recommended! I ~90% agree with it.
The authors argue that people are trying to build ASI (superintelligent AI), and we should expect them to succeed sooner or later, even if they obviously haven’t succeeded YET. I agree. (I lean “later” more than the authors, but that’s a minor disagreement.)
Ultra-fast minds that can do superhuman-quality thinking at 10,000 times the speed, that do not age and die, that make copies of their most successful representatives, that have been refined by billions of trials into unhuman kinds of thinking that work tirelessly and generalize more accurately from less data, and that can turn all that intelligence to analyzing and understanding and ultimately improving themselves—these minds would exceed ours.
The possibility of a machine intellect that manages to exceed human performance in all pragmatically important domains in which we operate has been called many things. We will describe it using the term “superintelligence,” meaning a mind much more capable than any human at almost every sort of steering and prediction problem—at least, those problems where there is room to substantially improve over human performance.[ii] …
(It sounds like sci-fi, but remember that every technology is sci-fi until it’s invented!)
They further argue that we should expect people to accidentally make misaligned ASI, utterly indifferent to whether humans live or die, even its own creators. They have a 3-part disjunctive argument:
Anyway, I agree with the conclusion of (A) but disagree with much of the book’s argument for it, as I have discussed many times (e.g. §3 of my “Sharp Left Turn” post). I think their arguments for (B) & (C) are solid. …And sufficient by themselves! It seems overdetermined!
The authors propose to get an international treaty to pause progress towards superintelligence, including both scaling & R&D. I’m for it, although I don’t hold out much hope for such efforts to have more than marginal impact. I expect that AI capabilities would rebrand as AI safety, and plow ahead:
The problem is: public advocacy is way too centered on LLMs, from my perspective. Thus, those researchers I mentioned, who are messing around with new paradigms on arXiv, are in a great position to twist “Pause AI” type public advocacy into support for what they’re doing!
“You don’t like LLMs?”, the non-LLM AGI capabilities researchers say to the Pause AI people, “Well how about that! I don’t like LLMs either! Clearly we are on the same team!”
This is not idle speculation—almost everyone that I can think of who is doing the most dangerous kind of AI capabilities research, the kind aiming to develop a new more-powerful-than-LLM AI paradigm, is already branding their work in a way that vibes with safety. For example, see here where I push back on someone using the word “controllability” to talk about his work advancing AI capabilities beyond the limits of LLMs. Ditto for “robustness” (example), “adaptability” (e.g. in the paper I was criticizing here), and even “interpretability” (details).
I think these people are generally sincere but mistaken, and I expect that, just as they have fooled themselves, they will also successfully fool their friends, their colleagues, and government regulators…
(source). For my part, I’m gonna keep working directly on (A). I think the world will be diving into the whirling knives of (A–C), sooner or later, and we’d better prepare as best we can.
The target audience of the book is not AI alignment experts like me, but rather novices. I obviously can’t speak from personal experience as to whether it’s a good read for those people, but anecdotally lots of people seem to think it is. So, I recommend the book to anyone.