Intended for a broad audience.^[1]

My belief is that keeping AI under human control would be an unprecedented global challenge, if it’s even possible at all.

Many other people’s believe that AIs will naturally remain as tools under human control. Either they see this as inevitable—“how could it be otherwise?”—or they see it as not quite inevitable, but still an outcome that can easily happen via ordinary government regulations, civil society, and so on.

Who’s right? What’s the right way to think about this?

This is an issue where people really don’t see eye-to-eye, and tend to get stuck endlessly talking past each other. To pin down the disagreement, I think that the following is a good conversation-starter:

Conversation-starting question: Do you expect almost all companies to eventually be founded and run by AIs rather than humans?

I find that people give a number of different answers:

Possible Answer 1: “No, because the best humans will always be better than large language models (LLMs) at founding and running companies.”

Huh? Whoever said anything about LLMs? I wrote “AIs”!

As it happens, I do think that humans will always be better than LLMs at founding and running companies! But if I’m right about that, then, well, so much the worse for LLMs! More powerful AI than that is possible—humans are an existence proof.

More details at: What do I mean by “Artificial General Intelligence”?

Possible Answer 2: “No, because the best humans will always be better than AIs at founding and running companies.”

I think that most people with this view are generally rejecting the idea that actually powerful AI (as in the right side of the table above) is possible at all, or are failing to think about its implications. Again, humans are an existence proof for what is physically possible for AI. Thus, for example:

Humans can acquire real-world first-person experience? Well, an AI could acquire a thousand lifetimes of real-world first-person experience.^[2]

Humans can form relationships with other humans? So can AIs—look at the bitter outcry when OpenAI cut access to GPT-4o, still festering as I write this five months later. And that was despite GPT-4o being remarkably stupid and forgetful in many ways, and lacking any expressive face or body. I expect future technology to go far beyond that example.

Humans can collaborate with each other and learn from culture? So can AIs.

Humans can walk around? Well, robotics is bottlenecked on algorithms; after AI gets better, robots will rapidly become cheap and abundant.^[3] And even if we somehow prevented anyone on Earth from making robot bodies (and destroyed the existing ones), heck, an AI could hire a human to walk around carrying a camera and microphone, while an AI whispers in their ear what to say and do—an ersatz robot body with a winning smile and handshake, for the mere cost of a low-skilled human salary.

Possible Answer 3: “No, because humans will always be equally good as AIs at founding and running companies.”

I think that most people with this view don’t really believe, in their guts, that skill and competence are a thing. Or at least, they don’t believe that skill and competence are relevant to founding and running successful companies.

When these people imagine what Jeff Bezos was doing day-to-day to build Amazon from scratch, I’m not quite sure what’s in their head. Befriending people? Scaring people? Puppies can also befriend and scare people, but no puppy has ever become a hundred-billionaire by founding and running a successful company.

So what was Jeff Bezos doing day-to-day? He was deciding who to hire; who to fire; what to delegate and to whom; what to approve; what to say to investors, underlings, and politicians; and on and on. And he was doing all those things with much, much more skill and competence^[4] than I would be able to, if I were trying to get rich building a mega-corporation. …Which I’m not trying to do. But if I were trying, then I would sure be way better at those things than a 2026-era LLM. And the LLM would in turn be way better at those things than a monkey at a typewriter.

Anyway, if we acknowledge that founding and running companies involves applying skill and competence to make decisions, and that Jeff Bezos has more skill and competence at building companies than I do (who in turn has more of those things than a 2026-era LLM, which has more than a monkey), then I claim it becomes pretty clear that future AIs will in turn be far better than Jeff Bezos at founding and running companies—not just equally good.

For one thing, Jeff Bezos would be a far more competent CEO if he could spend 240 hours a day, rather than 24, understanding his business, market conditions, customers, etc. He can’t, but AIs can run at superhuman speeds. And speed is the least of it! Humans have only so much working memory, only so much life experience, only so much ability to have creative insights and notice connections and opportunities, and so on. Thus there’s no reason to expect that humans are anywhere near the theoretical pinnacle of business-running.

Last but not least, even if it were true that AIs can only tie humans at CEOing, rather than crushing them (as I expect), the conclusion still wouldn’t follow, because of considerations of cost and scale. There are only so many competent humans, but if we have software that can found and run a company as skillfully as Jeff Bezos or Warren Buffett, it would be insanely profitable to run as many copies of that software as there are chips in the world—and then manufacture even more chips to run even more copies, until there are millions, then billions, viciously competing in every imaginable business niche, including business niches that no one has even dreamt of. So even if AIs were only as competent as humans at founding and running businesses, it would still be the case that we should expect almost all future companies to be founded and run by autonomous AIs.

Possible Answer 4: “No, because we will pass laws preventing AIs from founding and running companies.”

Even if such laws existed in every country on Earth, and the letter of such laws was enforceable, the spirit wouldn’t be. Rather, the laws would be trivial to work around. For example, you could wind up with companies where AIs are making all the decisions, but there’s a human frontman signing the paperwork.

Or you could wind up with things that are effectively AI-controlled companies, but which lack legal incorporation, etc.

Possible Answer 5: “No, because if someone wants to start a business, they would prefer to remain in charge themselves, and ask an AI for advice when needed, rather than ‘pressing go’ on an autonomous entrepreneurial AI.”

This is a really nice vision, and I wish I could believe it. But even if lots of people do in fact take this approach, and they create lots of great businesses, it just takes one person to say “Hmm, why should I create one great business, when I can instead create 100,000 great businesses simultaneously?”

…And then let’s imagine that this one person starts “Everything, Inc.”, a conglomerate company running millions of AIs that in turn are autonomously scouting out new business opportunities and then founding, running, and staffing tens of thousands of independent business ventures.

Under the giant legal umbrella of “Everything, Inc.”, perhaps one AI has started a business venture involving robots building solar cells in the desert; another AI is using robots to run wet-lab biology experiments and patenting new ideas; another AI is buying land and getting permits to eventually build a new gas station in Hoboken; various AIs are designing ever-better next-generation AIs and robots; and on and on.

In short order, “Everything, Inc.” would be earning wildly-unprecedented, eye-watering amounts of money, and reinvesting that money to buy or build chips for even more AIs that can found and grow even more companies in turn, and so on forever, as this person becomes a quadrillionaire.

That’s a caricatured example—the story could of course be far more gradual and distributed than one guy starting “Everything, Inc.”—but the point remains: there will be an extraordinarily strong economic incentive to use AIs in increasingly autonomous ways, rather than as assistants to human decision-makers. And in general, when things are both technologically possible and supported by extraordinarily strong economic incentives, those things are definitely going to happen sooner or later, in the absence of countervailing forces.

So what might stop that? Here are some possible counterarguments:

No, there won’t even be one person anywhere in the world who would want to start a company like “Everything, Inc.” Oh c’mon—people don’t tend to leave obvious trillion-dollar bills lying on the ground.
No, we will design AI algorithms in such a way that they can only be used as assistants, not as autonomous agents. Who exactly is “we”? In other words, there’s a really thorny coordination and enforcement problem to make that happen. Even if most people would prefer for autonomy-compatible AI algorithms to not exist at all, those algorithms are just waiting to be discovered, and the combination of scientific interest and economic incentives makes it extremely likely for them to be invented and widely shared sooner or later, in the absence of unprecedented permanent global clampdowns on AI research.^[5]
No, we’re going to outlaw companies like “Everything, Inc.” —But then we get into the various enforcement challenges discussed in Possible Answer 4. In particular, note that we can wind up at the same destination via a gradual and distributed global race-to-the-bottom on human oversight, as opposed to one mustache-twirling CEO creating a very obvious “Everything, Inc.” out of nowhere. The umpteen-trillion-dollar bill is just sitting there on the ground, and maybe police can ensure that no one person will openly grab the whole thing, but it’s much harder to stop everyone on Earth from taking bites out of it until it’s gone. The international coordination aspect here is also particularly difficult: The more that a country’s government turns a blind eye to this kind of activity, or secretly uses autonomous AI themselves, the more that that country would unlock staggering, unprecedented amounts of wealth and economic progress within its borders. See “Four ways learning Econ makes people dumber re: future AI”.
Well, sure, but all those millions of AIs would still be “tools” of the humans at “Everything, Inc.” corporate headquarters. I think this is stretching the definition of “tool” way past its breaking point. Imagine I’m a human employee of “Everything, Inc.” If we solve the alignment problem and everything goes perfectly, then ideally my company will be making money hand over fist, and our millions of AIs will be doing things that I would approve of, if I were to hypothetically take infinite time to investigate. But I’m not in the loop. Indeed, if I tried to understand what even one of these millions of AIs was doing and why, it would be a massive research project, because each AI has idiosyncratic experience and domain expertise that I lack, and this expertise informs how that AI is making decisions and executing its own workflow. Like any incompetent human micromanager, if I start scrutinizing the AIs’ actions, it would only slow things down and make the AIs’ decisions worse, and my firm would be immediately outcompeted by the next firm down the block that applied less human oversight. So really, those millions of AIs would be autonomously exercising their judgment to do whatever they think is best as they rapidly transform the world, while I’m sitting in my chair almost completely clueless about what’s going on. I really don’t think this picture is what people have in mind when they talk about AIs as tools under human control.

Possible Answers 6 & 7: “No, because we will eternally globally ban further AI research and progress before it reaches a point where AI could be a competent CEO” -OR- “No, because after the AIs exterminate (or permanently disempower) humanity, they will not operate via ‘companies’ as we think of them today.”

…Surprise twist! Did you think my answer to the question at the top was “yes”? Tricked you! I do actually think that it’s unlikely that almost all future companies will be founded and run by autonomous AIs, but for very different reasons than any of the ones above! Instead, I think one of these two “possible answers” will come true, alas, and much more likely the latter. But I won’t defend that here.

Afterword

To be clear, my real point in writing this post is not to argue that our mental models of the future should incorporate AI founder-CEOs (which I don’t even expect anyway!), but rather that the sci-fi trope^[6] in which humans are automatically the protagonists of the future, is a vision that doesn’t hang together as a plausible forecast. So I’m trying to poke holes in this vision, and my hope is that you will then reexamine the vision more broadly, until you hopefully wind up somewhere much more weird and radical.

By “weird and radical”, I mean the attitude: “asking about the effect of machine superintelligence on the conventional human labor market is like asking how US-Chinese trade patterns would be affected by the Moon crashing into the Earth. There would indeed be effects, but you'd be missing the point.” (Yudkowsky 2013) See also: my post “Four ways learning Econ makes people dumber re: future AI”.

Thanks Justis Mills, philh, and Seth Herd for critical comments on earlier drafts.

^{^}
This post is a revised and expanded excerpt from a longer post I wrote 2 years ago.
^{^}
AIs can simply live 1000 lifetimes, but they could also get a thousand lifetimes of real-world company-running experience much sooner than that, by having thousands of instances in parallel that are trying to found and run companies, and frequently merging the updates of the different instances. There would be some efficiency hit because some things build upon other things in a way that requires learning them serially. But even so, I would expect useful real-world experience to build up at vastly superhuman speed.
^{^}
Robots are so heavily bottlenecked on today’s mediocre AI software that many don’t even realize how easy the robot hardware problem would be if we had real human-level AI. If you don’t believe me, just note that future AI will be at least as good at piloting a robot as a human teleoperator. Thus, we can start by examining the state of teleoperated robotics today. For example:
Here’s a video of a teleoperated robot vacuuming, making coffee, cleaning a table, emptying the dishwasher, washing & drying & folding & hanging laundry, making a bed, etc. They also have a video where it cooks a meal. Their website lists a bill of materials for the teleoperated robot of ≈$30K, or ≈$20K if hypothetically there were an AGI teleoperating it. And this is a one-off, made by students, which doesn’t even have fingers!
But if you also want dexterous fingers, no problem! Anyone can build themselves a 17-degree-of-freedom robotic hand for a mere $1700 in a few hours.
Thus, teleoperated robots are remarkably cheap and competent even today, despite tiny volumes. Needless to say, human-teleoperator-level AI would send the volume skyrocketing, thus vastly lowering the prices and improving the quality and variety.
^{^}
One thing I am NOT saying here is: “Yay Jeff Bezos, what a great guy!”. No opinion on that! Lots of people have skill and competence, and are applying them towards making the world a worse place. That’s not a contradiction. Quite the contrary: some kind of skill and competence is required to make the world a much worse place! For example, if you refuse to agree with the statement “Adolf Hitler was a political genius”, then I claim you are thinking in vibes rather than rigorously trying to understand the world.
^{^}
As it happens, I’m skeptical that such clampdowns are feasible, regardless of whether or not they’re a good idea, mostly because I expect that future AIs will require much, much less compute than the LLMs of today. But whatever, that’s a different topic—see “Possible Answer 6” below.
^{^}
I.e., in the majority of popular sci-fi books and movies in which powerful AI exists, those AIs are really not doing very much, and instead it’s humans who remain the protagonists of the story. E.g. Star Trek, Star Wars, Foundation, Neuromancer, Blade Runner, and on and on.