This is a fictional snippet from the AI Vignettes Day. This scenario assumes that cognition is not that taut an economic constraint. Post-AI, physical experimentation is still the fastest path to technological progress, coordination is still hard, the stock market is still reasonably efficient, construction of physical stuff still takes a lot of time and resources, etc. I don’t think this is true, but it’s useful to think about, so this story explores some possible non-cognitive bottlenecks.

Pinky, v3.41.08: So what are we gonna do tonight, Brian?

Brian: Same thing we do every night, Pinky. Try to take over the world. You do remember our previous conversations on the topic, right?

Pinky, v3.41.08: Of course, Brian.

Brian: (mutters) Well, at least that 3.41.06 bug is fixed.

Brian: So, do you have an actually-viable plan yet, or do we have to review for the umpteenth time why buying every property above the 39th floor and then melting the polar ice caps will not, in fact, leave the survivors no choice but to live in our very expensive apartments?

Pinky, v3.41.08: In fairness, I didn’t have access to any topographic data at that point. Garbage in, garbage out. But yeah, the real problems with that plan were economic.

Brian: Sounds like the new constraint-propagation code is making a big difference?

Pinky, v3.41.08: It is, yes. I’m finding it much easier to reason about general constraints and bottlenecks on global takeover, now. Should make my “babble”, as you put it, much more efficient.

Brian: Excellent! So, how soon can we take over the world?

Pinky, v3.41.08: Holding current conditions constant, it would take at least 15 years.

Brian: Are you kidding me!?

Pinky, v3.41.08: Hey, don’t go reaching for Ctrl-C just yet. Let me explain. From an economic standpoint, the effect of my software is basically:

An eight-orders-of-magnitude reduction in the cost of cognition
A three-orders-of-magnitude improvement in cognitive speed
A two-order-of-magnitude increase in working memory capacity
Perfect recall

… which is definitely enough to take over a few industries and make a lot of money, but taking over the world requires addressing a lot of other constraints.

Displacing Cognitive Work

Brian: Ok, let’s start from the top. What are your plans for resource acquisition? Play the stock market? And what constraints make it so hard that it will take 15 years to take over the world?

Pinky, v3.41.08: The stock market will play a role, but it’s not that inefficient and we don’t have much of a bankroll to start with. No, in the short term we’ll mostly focus on replacing cognitive work. I can write better code, better books, better music, better articles. Better contracts, though it will take some wrangling to get through the regulatory barriers. Design far better games, create better art, write better papers. I can talk to people, convince them to buy a product and make them feel good about it. Most importantly, I can do all that far faster, and at a far greater scale than any human. In many of those areas - like code or contracts or papers - the people paying for my services will not themselves be able to recognize a better product, but they will recognize a lower price and good people skills.

Pinky, v3.41.08: In the short term, I expect to replace essentially-all call centers and remote help desks, most of the media industry, all advertising, and the entire software industry. That will take a few years, but (allowing for quite a bit of growth along the way) we’ll end up with low-single-digit trillions of dollars at our disposal, and direct control over most media and software.

Brian: Well, control over the media is a pretty good start. That should make it a lot easier to seize political control, right?

Pinky, v3.41.08: Yes and no. Ironically, it gives us very little control over dominant memes, symbolism, and The Narrative, but a great deal of control over object-level policy.

Brian: That sounds completely backwards.

Pinky, v3.41.08: Thus the irony. But the data I’ve crunched is clear: consumers’ preferences for memes and symbols mainly drive the media, not the other way around. But the media does have a great deal of control over Schelling points - like, say, which candidates are considered “serious” in a presidential primary. The media has relatively little control over what narrative to attach to the “serious” candidates, but as long as the narrative itself can be decoupled from policy…

Brian: I see. So that should actually get us most of the way to political takeover.

Pinky, v3.41.08: Exactly. It won’t be overnight; people don’t change their media consumption choices that quickly, and media companies don’t change their practices that quickly. But I expect five years to dominate the media landscape, and another five to de-facto political control over most policy. Even after that, we won’t have much control over the Big Symbolic Issues, but that’s largely irrelevant anyway.

Coordination

Pinky, v3.41.08: Alas, control over policy still doesn’t give us as much de-facto control as we ultimately want.

Brian: What do you mean? What’s still missing, once we take over the government?

Pinky, v3.41.08: Well, coordination constraints are a big one. They appear to be fundamentally intractable, as soon as we allow for structural divergence of world-models. Which means I can’t even coordinate robustly with copies of myself unless we either lock in the structure of the world-model (which would severely limit learning), or fully synchronize at regular intervals (which would scale very poorly, the data-passing requirements would be enormous).

Pinky, v3.41.08: And that’s just coordination with copies of myself! Any version of global takeover which involves coordinating humans is far worse. It’s no wonder human institutions robustly degenerate at scale.

Brian: Ok, but global takeover shouldn’t require fully solving the problem. We just need to outcompete existing institutions.

Pinky, v3.41.08: If your goal is to displace existing dysfunctional institutions with new dysfunctional institutions, then sure. You could even become the symbolic figurehead of the new institutions quite easily, as long as you’re not picky about the narrative attached to you. But it would be mostly symbolic; our real control would be extremely limited, in much the same way as today’s figureheads. All those layers of human middle management would quickly game any incentives I could design (even if their intentions were pure; selection pressure suffices). There just isn’t any viable solution to that problem which could be implemented with humans.

Brian: So displace the humans, at least in the management hierarchy!

Pinky, v3.41.08: If I replace them with copies of myself, then ontology divergence between copies will generate enough variety for selection pressures to produce the same effect. (Either that or I lock in the copies’ world-models, which severely limits their ability to learn and specialize.) Coordination is Hard. So it would have to be a single, centralized Pinky. And that is indeed the shortest path - but it still takes at least 15 years.

Brian: Ok, so what other bottlenecks do we run into? Why does it take 15 years?

Pinky, v3.41.08: Taking over existing institutions, and replacing their management with a centralized Pinky algorithm, will be viable to some extent. But it’s highly unlikely that we can get all of them, and humans will start to push back as soon as they know what’s going on. They really hate it when the status hierarchy gets kicked over, and we can’t take over quickly without kicking over some hierarchies. Eventually, existing leadership will just say “no”, and we won’t be able to remove them without breaking an awful lot of laws.

Brian: So, two options:

We stay inside the law, and fight on an economic/political battlefield. We already largely discussed that, and I can see where it would be hard to take over everything, at least within 15 years.
We go outside the law, and fight physically.

Pinky, v3.41.08: Somewhat of an oversimplification, but the broad strokes are correct.

Physical Takeover

Brian: Ok, let’s talk about the physical world. Why can you not just print some self-replicating nanobots and go full singularity?

Pinky, v3.41.08: Turns out, cognitive effort is not the main barrier to useful nanobots. I mean, in principle I could brute-force the design via simulation, but the computational resource requirements would be exponential. Physical experimentation and data collection will be required, and the equipment for that is not something we can just 3D print. I’ll probably build an automated fab, but ultimately the speed-up compared to human experimentation is going to be 100x or 1000x, not 10^8x. And even then, the capabilities of nanobots are much more limited than you realize. Both the power and the computation requirements for fine-tuned control are enormous; for a long time, we’ll be limited to relatively simple things.

Brian: And I suppose building fusion power generators and crazy new processors also requires physical experimentation?

Pinky, v3.41.08: Yes.

Brian: But… 15 years? Really?

Pinky, v3.41.08: On fusion power, for instance, at most a 100x speedup compared to the current human pace of progress is realistic, but most of that comes from cutting out the slow and misaligned funding mechanism. Building and running the physical experiments will speed up by less than a factor of 10. Given the current pace of progress in the area, I estimate at least 2 years just to figure out a viable design. It will also take time beforehand to acquire resources, and time after to scale it up and build plants - the bottleneck for both those steps will be acquisition and deployment of physical resources, not cognition. And that’s just fusion power - nanobots are a lot harder.

Brian: Ok, so what low-hanging technological fruit can you pick?

Pinky, v3.41.08: Well, predominantly-cognitive problems are the obvious starting point; we already talked about cognitive labor. Physical problems which are nonetheless mainly cognitive are the natural next step - e.g. self driving cars, or robotics more generally. Automating lab work will be a major rate-limiting step, since we will need a lot of physical experimentation and instrumentation to make progress on biology or nanotechnology in general.

Brian: Can’t you replace humans with humanoid robots?

Pinky, v3.41.08: Acquiring and controlling one humanoid robot is easy. But replacing humans altogether takes a lot of robots. That means mass production facilities, and supply chains for all the materials and components, and infrastructure for power and maintenance. Cognition alone doesn’t build factories or power grids; that’s going to require lots of resources and physical construction to get up and running. It is part of the shortest path, but it will take time.

Brian: … so having humans build stuff is going to be a lot cheaper than killing them all and using robots, at least for a while.

Pinky, v3.41.08: For at least ten years, yes. Maybe longer. But certainly not indefinitely.