Paul Christiano presented some low-key AI catastrophe scenarios; in response, Robin Hanson argued that Paul's scenarios were not consistent with the "large (mostly economic) literature on agency failures".
He concluded with:
For concreteness, imagine a twelve year old rich kid, perhaps a king or queen, seeking agents to help manage their wealth or kingdom. It is far from obvious that this child is on average worse off when they choose a smarter more capable agent, or when the overall pool of agents from which they can choose becomes smarter and more capable. And its even less obvious that the kid becomes maximally worse off as their agents get maximally smart and capable. In fact, I suspect the opposite.
Thinking on that example, my mind went to Edward the Vth of England (one of the "Princes in the Tower"), deposed then likely killed by his "protector" Richard III. Or of the Guangxu Emperor of China, put under house arrest by the Regent Empress Dowager Cixi. Or maybe the ten year-old Athitayawong, king of Ayutthaya, deposed by his main administrator after only 36 days of reign. More examples can be dug out from some of Wikipedia's list of rulers deposed as children.
We have no reason to restrict to child-monarchs - so many Emperors, Kings, and Tsars have been deposed by their advisers or "agents". So yes, there are many cases where agency fails catastrophically for the principal and where having a smarter or more rational agent was a disastrous move.
By restricting attention to agency problems in economics, rather than in politics, Robin restricts attention to situations where institutions are strong and behaviour is punished if it gets too egregious. Though even today, there is plenty of betrayal by "agents" in politics, even if the results are less lethal than in times gone by. In economics, too, we have fraudulent investors, some of which escape punishment. Agents betray their principals to the utmost - when they can get away with it.
So Robin's argument is entirely dependent on the assumption that institutions or rivals will prevent AIs from being able to abuse their agency power. Absent that assumption, most of the "large (mostly economic) literature on agency failures" becomes irrelevant.
So, would institutions be able to detect and punish abuses by future powerful AI agents? I'd argue we can't count on it, but it's a question that needs its own exploration, and is very different from what Robin's economic point seemed to be.
This suffers from an obvious huge selection bias, right? Also:
I guess this is a general argument that we shouldn't delegate things to others. But the interesting argument to worry about AI is that delegating to smarter artificial agents is much worse than the usual delegation scenario that we're used to, because of how much smarter they are than us. So if child-monarchs are deposed by advisers at the same rate as emperors, that's some evidence that the gap between the wisdom of the principal and the agent doesn't make things worse (although it could be that the absolute level of the wisdom of the agent modulates the effect of the gap, since this is held approximately constant in the sample).
I mainly mentioned child-rulers because Robin was using that example; and I used "getting deposed" as an example of agency problems that weren't often (ever?) listed in the economics literature.
TBC I agree that child-rulers are a relevant case, I just think that the frequency with which they are deposed relative to adult-rulers is what matters evidentially. I think I agree that getting deposed doesn't fit well into the agency literature though.