Followup to: Truly Part of You

"Myself, and Morisato-san... we want to live together by our own strength."

Jared Diamond once called agriculture "the worst mistake in the history of the human race". Farmers could grow more wheat than hunter-gatherers could collect nuts, but the evidence seems pretty conclusive that agriculture traded quality of life for quantity of life. One study showed that the farmers in an area were six inches shorter and seven years shorter-lived than their hunter-gatherer predecessors—even though the farmers were more numerous.

I don't know if I'd call agriculture a mistake. But one should at least be aware of the downsides. Policy debates should not appear one-sided.

In the same spirit—

Once upon a time, our hunter-gatherer ancestors strung their own bows, wove their own baskets, whittled their own flutes.

And part of our alienation from that environment of evolutionary adaptedness, is the number of tools we use that we don't understand and couldn't make for ourselves.

You can look back on Overcoming Bias, and see that I've always been suspicious of borrowed strength. (Even before I understood the source of Robin's and my disagreement about the Singularity, that is.) In Guessing the Teacher's Password I talked about the (well-known) problem in which schools end up teaching verbal behavior rather than real knowledge. In Truly Part of You I suggested one test for false knowledge: Imagine deleting a fact from your mind, and ask if it would grow back.

I know many ways to prove the Pythagorean Theorem, including at least one proof that is purely visual and can be seen at a glance. But if you deleted the Pythagorean Theorem from my mind entirely, would I have enough math skills left to grow it back the next time I needed it? I hope so—certainly I've solved math problems that seem tougher than that, what with benefit of hindsight and all. But, as I'm not an AI, I can't actually switch off the memories and associations, and test myself in that way.

Wielding someone else's strength to do things beyond your own understanding—that really is as dangerous as the Deeply Wise phrasing makes it sound.

I observed in Failing to Learn from History (musing on my childhood foolishness in offering a mysterious answer to a mysterious question): "If only I had personally postulated astrological mysteries and then discovered Newtonian mechanics, postulated alchemical mysteries and then discovered chemistry, postulated vitalistic mysteries and then discovered biology. I would have thought of my Mysterious Answer and said to myself: No way am I falling for that again."

At that point in my childhood, I'd been handed some techniques of rationality but I didn't exactly own them. Borrowing someone else's knowledge really doesn't give you anything remotely like the same power level required to discover that knowledge for yourself.

Would Isaac Newton have remained a mystic, even in that earlier era, if he'd lived the lives of Galileo and Archimedes instead of just reading about them? If he'd personally seen the planets reduced from gods to spheres in a telescope? If he'd personally fought that whole war against ignorance and mystery that had to be fought, before Isaac Newton could be handed math and science as a start to his further work?

We stand on the shoulders of giants, and in doing so, the power that we wield is far out of proportion to the power that we could generate for ourselves. This is true even of our revolutionaries. And yes, we couldn't begin to support this world if people could only use their own strength. Even so, we are losing something.

That thought occurred to me, reading about the Manhattan Project, and the petition that the physicists signed to avoid dropping the atomic bomb on Japan. It was too late, of course; they'd already built the bombs and handed them to the military, and they couldn't take back that gift. And so nuclear weapons passed into the hands of politicians, who could never have created such a thing through their own strength...

Not that I'm saying the world would necessarily have been a better place, if physicists had possessed sole custody of ICBMs. What does a physicist know about international diplomacy, or war? And it's not as if Leo Szilard—who first thought of the fission chain reaction—had personally invented science; he too was using powers beyond his own strength. And it's not as if the physicists on the Manhattan Project raised the money to pay for their salaries and their materials; they were borrowing the strength of the politicians...

But if no one had been able to use nuclear weapons without, say, possessing the discipline of a scientist and the discipline of a politician—without personally knowing enough to construct an atomic bomb and make friends—the world might have been a slightly safer place.

And if nobody had been able to construct an atomic bomb without first discovering for themselves the nature and existence of physics, then we would have been much safer from atomic bombs, because no one would have been able to build them until they were two hundred years old.

With humans leaving the game after just seventy years, we couldn't support this world using only our own strengths. But we have traded quality of insight for quantity of insight.

It does sometimes seem to me that many of this world's problems, stem from our using powers that aren't appropriate to seventy-year-olds.

And there is a higher level of strength-ownership, which no human being has yet achieved. Even when we run, we're just using the muscles that evolution built for us. Even when we think, we're just using the brains that evolution built for us.

I'm not suggesting that people should create themselves from scratch without a starting point. Just pointing out that it would be a different world if we understood our own brains and could redesign our own legs. As yet there's no human "rationalist" or "scientist", whatever they know about "how to think", who could actually build a rational AI—which shows you the limits of our self-understanding.

This is not the sort of thing that I'd suggest as an immediate alteration. I'm not suggesting that people should instantly on a silver platter be given full knowledge of how their own brains work and the ability to redesign their own legs. Because maybe people will be better off if they aren't given that kind of power, but rather have to work out the answer for themselves.

Just in terms of anomie versus fun, there's a big difference between being able to do things for yourself, and having to rely on other people to do them for you. (Even if you're doing them with a brain you never designed yourself.)

I don't know if it's a principle that would stay until the end of time, to the children's children. Maybe better-designed minds could handle opaque tools without the anomie.

But it is part of the commonly retold prophecy of Artificial Intelligence and the Age of the Machine, that this era must be accompanied by greater reliance on things outside yourself, more incomprehensible tools into which you have less insight and less part in their creation.

Such a prophecy is not surprising. That is the way the trend has gone so far, in our culture that is too busy staying alive to optimize for fun. From the fire-starting tools that you built yourself, to the village candleseller, and then from the candleseller to the electric light that runs on strange mathematical principles and is powered by a distant generator... we are surrounded by things outside ourselves and strengths outside our understanding; we need them to stay alive, or we buy them because it's easier that way.

But with a sufficient surplus of power, you could start doing things the eudaimonic way. Start rethinking the life experience as a road to internalizing new strengths, instead of just trying to keep people alive efficiently.

A Friendly AI doesn't have to be a continuation of existing trends. It's not the Machine. It's not the alien force of technology. It's not mechanizing a factory. It's not a new gadget for sale. That's not where the shape comes from. What it is—is not easy to explain; but I'm reminded of doc Smith's description of the Lens as "the physical manifestation of a purely philosophical concept". That philosophical concept doesn't have to manifest as new buttons to press—if, on reflection, that's not what we would want.