I was prompted to think about why standard human checks-and-balances against violent takeover would fail for AIs given that they work for humans. Here's some notes on that.
The analogy to human society isn’t that reassuring. Wars, coups, and other violations of property rights are common in history. I think recent history features more than one civil war and coup per country per century. These are driven by technological changes, economic changes, political negotiations, that are ultimately strongly related to cognitive work. During takeoff, I expect more than a centuries’ worth of economic and technological changes. So if the normal historical trends speed up proportionately, then it would be unsurprising to have a civil war. In a war like that, humans might a group that:
If AIs are broadly like humans, then it would be unsurprising for humans to all be on the losing side of this conflict.
This doesn’t require all the AIs to coordinate with each other. See e.g. here: Cortés fought other Spaniards during his conquest of the Aztec empire. Pizarro fought a civil war against other Spaniards in the middle of his conquest of Peru. They still succeeded in taking over.
That said, two specific reasons checks and balances could work even worse for AIs than humans:
On the first one: AIs are way more correlated than humans.
Ok, that was the first disanalogy with humans (that AIs are more correlated). The second answer is that normal, human checks and balances will fail when humans don’t understand what’s going on well enough to have any idea about whether there’s a conspiracy or not, whether anyone’s trying to take over or not, etc.
Some even scrappier notes on some further, only somewhat related questions here. (Notes on a random story of schemer takeover, and likelihood of dying from AI takeover in some situations.