Is the widespread false belief that LLMs are AGI-complete going to kill us all?

(Prompted by this post by @Steven Byrnes, with which I basically agree.)

Suppose we live in a world in which LLMs don't scale to AGI.^[1] If so, I think it's fairly plausible that LLM-alignment-optimists are correct (on the narrow question of LLM alignment). That is: all the technical arguments and empirical evidence in favor of LLMs being relatively easy to align/control, even in the limit of LLM capabilities, are valid, and the doomsaying about LLMs is wrong.

After all, most of the strongest arguments for AGI doom route through arguments about AGIs/generally intelligent agents. If LLMs are not those, keeping them under control needn't be hard! The orthodox AI doomsaying and the LLM optimism can both be true.

But if we do live in such a world, we have a very annoying comms problem. Consider:

• The reasoning of capabilities people usually goes, "LLMs are AGI-complete" (invalid) and "LLMs are pretty easy to align" (valid), which leads them to conclude "AGI is pretty easy to align" (invalid).
• Meanwhile, AI Safety advocates also start from "LLMs are AGI-complete" (invalid) but "AGI is not easy to align" (valid), and so end up at "LLMs are not easy to align" (invalid).

Which then leads to AI Safety advocates trying to argue AI Risk to capabilities people by trying to find arguments and demos in favor of "LLMs are not easy to align".

Which... is maybe possibly completely incorrect? Meaning they're deploying false arguments that ought to be, and will continually be, mercilessly destroyed by the empirical truth? Oops.

This makes the "LLMs are AGI-complete" meme a double-edged sword. On one hand, it leads to most of the AI industry getting distracted by LLMs, which means they're spending tons of effort not actually doing anything dangerous. Awesome (or is it?). On the other hand, this sure poisons/confuses the communications, and makes the AI industry falsely convinced of "AGI is easy to align". Which might lead to...

I think one fairly plausible scenario of how we all die goes as follows:

1. LLMs are not AGI-complete.
2. LLMs are easy to align.
3. An LLM lab, staffed by people who have validly convinced themselves that LLMs are easy to align and invalidly convinced themselves that LLMs are AGI-complete, tries various architectural and training-loop tweaks to increase their LLMs' capabilities.
4. One of these tweaks happens to be the secret sauce that does result in an AGI-complete architecture/training loop.
5. The LLM lab is still operating on the assumption that what they're working on has the same properties that made LLMs easy to align.
6. This isn't an indefensible assumption for them to make: at first glance, the tweak may legitimately look like just a quantitative algorithmic improvement or something. (Especially if they're operating on the model of the world where LLMs are AGI-complete, which would mean that no further qualitative improvements/phase changes should be expected.)
7. But this is not in fact the case. The tweak significantly changes the true nature, the type signature, of the AI system under development. The LLM intuitions no longer apply to it... and yet they get smuggled over.
8. They train the thing that is no longer really an LLM, it goes FOOM, kills everyone, blah blah blah.

It's possible that the LLM paradigm is so far from being AGI-complete that this can't happen, in which case LLM researchers really are mostly harmless and the real threat is all these guys working on fundamentally different approaches. But I'm not very strongly convinced of that.

What should we do about any of this? I don't really know, there's a lot of uncertainty and any action that's helpful under one model is actively harmful under the other. But, well, here's another problem we may be having on our hands, I guess.

(Addendum: @Vladimir_Nesov outlines a model here involving a slow distributed/industry-wide process of LLM RSI, which ends up AGI-complete when taken together, but without any of its individual components being the sort of scary fast AGI that can cause doom. I have plenty of probability mass on that as well, I think something like this can be cobbled together. And if so, I think the LLMs-are-easy-to-align properties may carry over to that process... However, it would be really very slow. So slow that AGI-by-any-other-route would likely kill us well before we can limp our way to the Singularity on the straining backs of LLMs.

Worse: on that route, the tragicomedic scenario above may well play out within this process. Just substitute "an LLM lab" above for "a swarm of LLM agents".

(Which, please note, would be subtly but meaningfully distinct from the distributed-LLM-RSI-process "directly" going out of control and bringing the doom about. Like, this would be an unexpected unintended disaster within that process as well, for the LLM agents, and obviously the technical details are very different.))

1. ^{^}

   I am not convinced we do, but this is still my mainline guess. I am unmoved by Mythos/Fable and the unit distance conjecture: LLMs scaling to superhuman math-proving and cyberwarfare was always possible on my model. (Though I did also separately hope that LLMs would top out even below that level.)