Followup to: Engelbart: Insufficiently Recursive

The computer revolution had cascades and insights aplenty. Computer tools are routinely used to create tools, from using a C compiler to write a Python interpreter, to using theorem-proving software to help design computer chips. I would not yet rate computers as being very deeply recursive - I don't think they've improved our own thinking processes even so much as the Scientific Revolution - yet. But some of the ways that computers are used to improve computers, verge on being repeatable (cyclic).

Yet no individual, no localized group, nor even country, managed to get a sustained advantage in computing power, compound the interest on cascades, and take over the world. There was never a Manhattan moment when a computing advantage temporarily gave one country a supreme military advantage, like the US and its atomic bombs for that brief instant at the end of WW2. In computing there was no equivalent of "We've just crossed the sharp threshold of criticality, and now our pile doubles its neutron output every two minutes, so we can produce lots of plutonium and you can't."

Will the development of nanotechnology go the same way as computers - a smooth, steady developmental curve spread across many countries, no one project taking into itself a substantial fraction of the world's whole progress? Will it be more like the Manhattan Project, one country gaining a (temporary?) huge advantage at huge cost? Or could a small group with an initial advantage cascade and outrun the world?

Just to make it clear why we might worry about this for nanotech, rather than say car manufacturing - if you can build things from atoms, then the environment contains an unlimited supply of perfectly machined spare parts. If your molecular factory can build solar cells, it can acquire energy as well.

So full-fledged Drexlerian molecular nanotechnology can plausibly automate away much of the manufacturing in its material supply chain. If you already have nanotech, you may not need to consult the outside economy for inputs of energy or raw material.

This makes it more plausible that a nanotech group could localize off, and do its own compound interest, away from the global economy. If you're Douglas Engelbart building better software, you still need to consult Intel for the hardware that runs your software, and the electric company for the electricity that powers your hardware. It would be a considerable expense to build your own fab lab for your chips (that makes chips as good as Intel) and your own power station for electricity (that supplies electricity as cheaply as the utility company).

It's not just that this tends to entangle you with the fortunes of your trade partners, but also that - as an UberTool Corp keeping your trade secrets in-house - you can't improve the hardware you get, or drive down the cost of electricity, as long as these things are done outside. Your cascades can only go through what you do locally, so the more you do locally, the more likely you are to get a compound interest advantage. (Mind you, I don't think Engelbart could have gone FOOM even if he'd made his chips locally and supplied himself with electrical power - I just don't think the compound advantage on using computers to make computers is powerful enough to sustain k > 1.)

In general, the more capabilities are localized into one place, the less people will depend on their trade partners, the more they can cascade locally (apply their improvements to yield further improvements), and the more a "critical cascade" / FOOM sounds plausible.

Yet self-replicating nanotech is a very advanced capability. You don't get it right off the bat. Sure, lots of biological stuff has this capability, but this is a misleading coincidence - it's not that self-replication is easy, but that evolution, for its own alien reasons, tends to build it into everything. (Even individual cells, which is ridiculous.)

In the run-up to nanotechnology, it seems not implausible to suppose a continuation of the modern world. Today, many different labs work on small pieces of nanotechnology - fortunes entangled with their trade partners, and much of their research velocity coming from advances in other laboratories. Current nanotech labs are dependent on the outside world for computers, equipment, science, electricity, and food; any single lab works on a small fraction of the puzzle, and contributes small fractions of the progress.

In short, so far nanotech is going just the same way as computing.

But it is a tad premature - I would even say that it crosses the line into the "silly" species of futurism - to exhale a sigh of relief and say, "Ah, that settles it - no need to consider any further."

We all know how exponential multiplication works: 1 microscopic nanofactory, 2 microscopic nanofactories, 4 microscopic nanofactories... let's say there's 100 different groups working on self-replicating nanotechnology and one of those groups succeeds one week earlier than the others. Rob Freitas has calculated that some species of replibots could spread through the Earth in 2 days (even given what seem to me like highly conservative assumptions in a context where conservatism is not appropriate).

So, even if the race seems very tight, whichever group gets replibots first can take over the world given a mere week's lead time -

Yet wait! Just having replibots doesn't let you take over the world. You need fusion weapons, or surveillance bacteria, or some other way to actually govern. That's a lot of matterware - a lot of design and engineering work. A replibot advantage doesn't equate to a weapons advantage, unless, somehow, the planetary economy has already published the open-source details of fully debugged weapons that you can build with your newfound private replibots. Otherwise, a lead time of one week might not be anywhere near enough.

Even more importantly - "self-replication" is not a binary, 0-or-1 attribute. Things can be partially self-replicating. You can have something that manufactures 25% of itself, 50% of itself, 90% of itself, or 99% of itself - but still needs one last expensive computer chip to complete the set. So if you have twenty-five countries racing, sharing some of their results and withholding others, there isn't one morning where you wake up and find that one country has self-replication.

Bots become successively easier to manufacture; the factories get successively cheaper. By the time one country has bots that manufacture themselves from environmental materials, many other countries have bots that manufacture themselves from feedstock. By the time one country has bots that manufacture themselves entirely from feedstock, other countries have produced some bots using assembly lines. The nations also have all their old conventional arsenal, such as intercontinental missiles tipped with thermonuclear weapons, and these have deterrent effects against crude nanotechnology. No one ever gets a discontinuous military advantage, and the world is safe. (?)

At this point, I do feel obliged to recall the notion of "burdensome details", that we're spinning a story out of many conjunctive details, any one of which could go wrong. This is not an argument in favor of anything in particular, just a reminder not to be seduced by stories that are too specific. When I contemplate the sheer raw power of nanotechnology, I don't feel confident that the fabric of society can even survive the sufficiently plausible prospect of its near-term arrival. If your intelligence estimate says that Russia (the new belligerent Russia under Putin) is going to get self-replicating nanotechnology in a year, what does that do to Mutual Assured Destruction? What if Russia makes a similar intelligence assessment of the US? What happens to the capital markets? I can't even foresee how our world will react to the prospect of various nanotechnological capabilities as they promise to be developed in the future's near future. Let alone envision how society would actually change as full-fledged molecular nanotechnology was developed, even if it were developed gradually...

...but I suppose the Victorians might say the same thing about nuclear weapons or computers, and yet we still have a global economy - one that's actually lot more interdependent than theirs, thanks to nuclear weapons making small wars less attractive, and computers helping to coordinate trade.

I'm willing to believe in the possibility of a smooth, gradual ascent to nanotechnology, so that no one state - let alone any corporation or small group - ever gets a discontinuous advantage.

The main reason I'm willing to believe this is because of the difficulties of design and engineering, even after all manufacturing is solved. When I read Drexler's Nanosystems, I thought: "Drexler uses properly conservative assumptions everywhere I can see, except in one place - debugging. He assumes that any failed component fails visibly, immediately, and without side effects; this is not conservative."

In principle, we have complete control of our computers - every bit and byte is under human command - and yet it still takes an immense amount of engineering work on top of that to make the bits do what we want. This, and not any difficulties of manufacturing things once they are designed, is what takes an international supply chain of millions of programmers.

But we're still not out of the woods.

Suppose that, by a providentially incremental and distributed process, we arrive at a world of full-scale molecular nanotechnology - a world where designs, if not finished material goods, are traded among parties. In a global economy large enough that no one actor, or even any one state, is doing more than a fraction of the total engineering.

It would be a very different world, I expect; and it's possible that my essay may have already degenerated into nonsense. But even if we still have a global economy after getting this far - then we're still not out of the woods.

Remember those ems? The emulated humans-on-a-chip? The uploads?

Suppose that, with molecular nanotechnology already in place, there's an international race for reliable uploading - with some results shared, and some results private - with many state and some nonstate actors.

And suppose the race is so tight, that the first state to develop working researchers-on-a-chip, only has a one-day lead time over the other actors.

That is - one day before anyone else, they develop uploads sufficiently undamaged, or capable of sufficient recovery, that the ems can carry out research and development. In the domain of, say, uploading.

There are other teams working on the problem, but their uploads are still a little off, suffering seizures and having memory faults and generally having their cognition degraded to the point of not being able to contribute. (NOTE: I think this whole future is a wrong turn and we should stay away from it; I am not endorsing this.)

But this one team, though - their uploads still have a few problems, but they're at least sane enough and smart enough to start... fixing their problems themselves?

If there's already full-scale nanotechnology around when this happens, then even with some inefficiency built in, the first uploads may be running at ten thousand times human speed. Nanocomputers are powerful stuff.

And in an hour, or around a year of internal time, the ems may be able to upgrade themselves to a hundred thousand times human speed, and fix some of the remaining problems.

And in another hour, or ten years of internal time, the ems may be able to get the factor up to a million times human speed, and start working on intelligence enhancement...

One could, of course, voluntarily publish the improved-upload protocols to the world, and give everyone else a chance to join in. But you'd have to trust that not a single one of your partners were holding back a trick that lets them run uploads at ten times your own maximum speed (once the bugs were out of the process). That kind of advantage could snowball quite a lot, in the first sidereal day.

Now, if uploads are gradually developed at a time when computers are too slow to run them quickly - meaning, before molecular nanotech and nanofactories come along - then this whole scenario is averted; the first high-fidelity uploads, running at a hundredth of human speed, will grant no special advantage. (Assuming that no one is pulling any spectacular snowballing tricks with intelligence enhancement - but they would have to snowball fast and hard, to confer advantage on a small group running at low speeds. The same could be said of brain-computer interfaces, developed before or after nanotechnology, if running in a small group at merely human speeds. I would credit their world takeover, but I suspect Robin Hanson wouldn't at this point.)

Now, I don't really believe in any of this - this whole scenario, this whole world I'm depicting. In real life, I'd expect someone to brute-force an unFriendly AI on one of those super-ultimate-nanocomputers, followed in short order by the end of the world. But that's a separate issue. And this whole world seems too much like our own, after too much technological change, to be realistic to me. World government with an insuperable advantage? Ubiquitous surveillance? I don't like the ideas, but both of them would change the game dramatically...

But the real point of this essay is to illustrate a point more important than nanotechnology: as optimizers become more self-swallowing, races between them are more unstable.

If you sent a modern computer back in time to 1950 - containing many modern software tools in compiled form, but no future history or declaratively stored future science - I would guess that the recipient could not use it to take over the world. Even if the USSR got it. Our computing industry is a very powerful thing, but it relies on a supply chain of chip factories.

If someone got a future nanofactory with a library of future nanotech applications - including designs for things like fusion power generators and surveillance bacteria - they might really be able to take over the world. The nanofactory swallows its own supply chain; it incorporates replication within itself. If the owner fails, it won't be for lack of factories. It will be for lack of ability to develop new matterware fast enough, and apply existing matterware fast enough, to take over the world.

I'm not saying that nanotech will appear from nowhere with a library of designs - just making a point about concentrated power and the instability it implies.

Think of all the tech news that you hear about once - say, an article on Slashdot about yada yada 50% improved battery technology - and then you never hear about again, because it was too expensive or too difficult to manufacture.

Now imagine a world where the news of a 50% improved battery technology comes down the wire, and the head of some country's defense agency is sitting down across from engineers and intelligence officers and saying, "We have five minutes before all of our rival's weapons are adapted to incorporate this new technology; how does that affect our balance of power?" Imagine that happening as often as "amazing breakthrough" articles appear on Slashdot.

I don't mean to doomsay - the Victorians would probably be pretty surprised we haven't blown up the world with our ten-minute ICBMs, but we don't live in their world - well, maybe doomsay just a little - but the point is: It's less stable. Improvements cascade faster once you've swallowed your manufacturing supply chain.

And if you sent back in time a single nanofactory, and a single upload living inside it - then the world might end in five minutes or so, as we bios measure time.

The point being, not that an upload will suddenly appear, but that now you've swallowed your supply chain and your R&D chain.

And so this world is correspondingly more unstable, even if all the actors start out in roughly the same place. Suppose a state manages to get one of those Slashdot-like technology improvements - only this one lets uploads think 50% faster - and they get it fifty minutes before anyone else, at a point where uploads are running ten thousand times as fast as human (50 mins = ~1 year) - and in that extra half-year, the uploads manage to find another couple of 50% improvements...

Now, you can suppose that all the actors are all trading all of their advantages and holding nothing back, so everyone stays nicely synchronized.

Or you can suppose that enough trading is going on, that most of the research any group benefits from comes from outside that group, and so a 50% advantage for a local group doesn't cascade much.

But again, that's not the point. The point is that in modern times, with the modern computing industry, where commercializing an advance requires building a new computer factory, a bright idea that has gotten as far as showing a 50% improvement in the laboratory, is merely one more article on Slashdot.

If everything could instantly be rebuilt via nanotech, that laboratory demonstration could precipitate an instant international military crisis.

And if there are uploads around, so that a cute little 50% advancement in a certain kind of hardware, recurses back to imply 50% greater speed at all future research - then this Slashdot article could become the key to world domination.

As systems get more self-swallowing, they cascade harder; and even if all actors start out equivalent, races between them get much more unstable. I'm not claiming it's impossible for that world to be stable. The Victorians might have thought that about ICBMs. But that subjunctive world contains additional instability compared to our own, and would need additional centripetal forces to end up as stable as our own.

I expect Robin to disagree with some part of this essay, but I'm not sure which part or how.