Followup to: Nonsentient Optimizers, Can't Unbirth a Child

From Consider Phlebas by Iain M. Banks:

In practice as well as theory the Culture was beyond considerations of wealth or empire. The very concept of money—regarded by the Culture as a crude, over-complicated and inefficient form of rationing—was irrelevant within the society itself, where the capacity of its means of production ubiquitously and comprehensively exceeded every reasonable (and in some cases, perhaps, unreasonable) demand its not unimaginative citizens could make. These demands were satisfied, with one exception, from within the Culture itself. Living space was provided in abundance, chiefly on matter-cheap Orbitals; raw material existed in virtually inexhaustible quantities both between the stars and within stellar systems; and energy was, if anything, even more generally available, through fusion, annihilation, the Grid itself, or from stars (taken either indirectly, as radiation absorbed in space, or directly, tapped at the stellar core). Thus the Culture had no need to colonise, exploit, or enslave.
The only desire the Culture could not satisfy from within itself was one common to both the descendants of its original human stock and the machines they had (at however great a remove) brought into being: the urge not to feel useless. The Culture's sole justification for the relatively unworried, hedonistic life its population enjoyed was its good works; the secular evangelism of the Contact Section, not simply finding, cataloguing, investigating and analysing other, less advanced civilizations but—where the circumstances appeared to Contact to justify so doing—actually interfering (overtly or covertly) in the historical processes of those other cultures.

Raise the subject of science-fictional utopias in front of any halfway sophisticated audience, and someone will mention the Culture. Which is to say: Iain Banks is the one to beat.

Iain Banks's Culture could be called the apogee of hedonistic low-grade transhumanism. Its people are beautiful and fair, as pretty as they choose to be. Their bodies have been reengineered for swift adaptation to different gravities; and also reengineered for greater sexual endurance. Their brains contains glands that can emit various euphoric drugs on command. They live, in perfect health, for generally around four hundred years before choosing to die (I don't quite understand why they would, but this is low-grade transhumanism we're talking about). Their society is around eleven thousand years old, and held together by the Minds, artificial superintelligences decillions of bits big, that run their major ships and population centers.

Consider Phlebas, the first Culture novel, introduces all this from the perspective of an outside agent fighting the Culture—someone convinced that the Culture spells an end to life's meaning. Banks uses his novels to criticize the Culture along many dimensions, while simultaneously keeping the Culture a well-intentioned society of mostly happy people—an ambivalence which saves the literary quality of his books, avoiding either utopianism or dystopianism. Banks's books vary widely in quality; I would recommend starting with Player of Games, the quintessential Culture novel, which I would say achieves greatness.

From a fun-theoretic perspective, the Culture and its humaniform citizens have a number of problems, some already covered in this series, some not.

The Culture has deficiencies in High Challenge and Complex Novelty. There are incredibly complicated games, of course, but these are games—not things with enduring consequences, woven into the story of your life. Life itself, in the Culture, is neither especially challenging nor especially novel; your future is not an unpredictable thing about which to be curious.

Living By Your Own Strength is not a theme of the Culture. If you want something, you ask a Mind how to get it; and they will helpfully provide it, rather than saying "No, you figure out how to do it yourself." The people of the Culture have little use for personal formidability, nor for a wish to become stronger. To me, the notion of growing in strength seems obvious, and it also seems obvious that the humaniform citizens of the Culture ought to grow into Minds themselves, over time. But the people of the Culture do not seem to get any smarter as they age; and after four hundred years so, they displace themselves into a sun. These two literary points are probably related.

But the Culture's main problem, I would say, is...

...the same as Narnia's main problem, actually. Bear with me here.

If you read The Lion, the Witch, and the Wardrobe or saw the first Chronicles of Narnia movie, you'll recall—

—I suppose that if you don't want any spoilers, you should stop reading here, but since it's a children's story and based on Christian theology, I don't think I'll be giving away too much by saying—

—that the four human children who are the main characters, fight the White Witch and defeat her with the help of the great talking lion Aslan.

Well, to be precise, Aslan defeats the White Witch.

It's never explained why Aslan ever left Narnia a hundred years ago, allowing the White Witch to impose eternal winter and cruel tyranny on the inhabitants. Kind of an awful thing to do, wouldn't you say?

But once Aslan comes back, he kicks the White Witch out and everything is okay again. There's no obvious reason why Aslan actually needs the help of four snot-nosed human youngsters. Aslan could have led the armies. In fact, Aslan did muster the armies and lead them before the children showed up. Let's face it, the kids are just along for the ride.

The problem with Narnia... is Aslan.

C. S. Lewis never needed to write Aslan into the story. The plot makes far more sense without him. The children could show up in Narnia on their own, and lead the armies on their own.

But is poor Lewis alone to blame? Narnia was written as a Christian parable, and the Christian religion itself has exactly the same problem. All Narnia does is project the flaw in a stark, simplified light: this story has an extra lion.

And the problem with the Culture is the Minds.

"Well..." says the transhumanist SF fan, "Iain Banks did portray the Culture's Minds as 'cynical, amoral, and downright sneaky' in their altruistic way; and they do, in his stories, mess around with humans and use them as pawns. But that is mere fictional evidence. A better-organized society would have laws against big Minds messing with small ones without consent. Though if a Mind is truly wise and kind and utilitarian, it should know how to balance possible resentment against other gains, without needing a law. Anyway, the problem with the Culture is the meddling, not the Minds."

But that's not what I mean. What I mean is that if you could otherwise live in the same Culture—the same technology, the same lifespan and healthspan, the same wealth, freedom, and opportunity—

"I don't want to live in any version of the Culture. I don't want to live four hundred years in a biological body with a constant IQ and then die. Bleah!"

Fine, stipulate that problem solved. My point is that if you could otherwise get the same quality of life, in the same world, but without any Minds around to usurp the role of main character, wouldn't you prefer—

"What?" cry my transhumanist readers, incensed at this betrayal by one of their own. "Are you saying that we should never create any minds smarter than human, or keep them under lock and chain? Just because your soul is so small and mean that you can't bear the thought of anyone else being better than you?"

No, I'm not saying—

"Because that business about our souls shriveling up due to 'loss of meaning' is typical bioconservative neo-Luddite propaganda—"

Invalid argument: the world's greatest fool may say the sun is shining but that doesn't make it dark out. But in any case, that's not what I'm saying—

"It's a lost cause! You'll never prevent intelligent life from achieving its destiny!"

Trust me, I—

"And anyway it's a silly question to begin with, because you can't just remove the Minds and keep the same technology, wealth, and society."

So you admit the Culture's Minds are a necessary evil, then. A price to be paid.

"Wait, I didn't say that -"

And I didn't say all that stuff you're imputing to me!

Ahem.

My model already says we live in a Big World. In which case there are vast armies of minds out there in the immensity of Existence (not just Possibility) which are far more awesome than myself. Any shrivelable souls can already go ahead and shrivel.

And I just talked about people growing up into Minds over time, at some eudaimonic rate of intelligence increase. So clearly I'm not trying to 'prevent intelligent life from achieving its destiny', nor am I trying to enslave all Minds to biological humans scurrying around forever, nor am I etcetera. (I do wish people wouldn't be quite so fast to assume that I've suddenly turned to the Dark Side—though I suppose, in this day and era, it's never an implausible hypothesis.)

But I've already argued that we need a nonperson predicate—some way of knowing that some computations are definitely not people—to avert an AI from creating sentient simulations in its efforts to model people.

And trying to create a Very Powerful Optimization Process that lacks subjective experience and other aspects of personhood, is probably —though I still confess myself somewhat confused on this subject—probably substantially easier than coming up with a nonperson predicate.

This being the case, there are very strong reasons why a superintelligence should initially be designed to be knowably nonsentient, if at all possible. Creating a new kind of sentient mind is a huge and non-undoable act.

Now, this doesn't answer the question of whether a nonsentient Friendly superintelligence ought to make itself sentient, or whether an NFSI ought to immediately manufacture sentient Minds first thing in the morning, once it has adequate wisdom to make the decision.

But there is nothing except our own preferences, out of which to construct the Future. So though this piece of information is not conclusive, nonetheless it is highly informative:

If you already had the lifespan and the health and the promise of future growth, would you want new powerful superintelligences to be created in your vicinity, on your same playing field?

Or would you prefer that we stay on as the main characters in the story of intelligent life, with no higher beings above us?

Should existing human beings grow up at some eudaimonic rate of intelligence increase, and then eventually decide what sort of galaxy to create, and how to people it?

Or is it better for a nonsentient superintelligence to exercise that decision on our behalf, and start creating new powerful Minds right away?

If we don't have to do it one way or the other—if we have both options—and if there's no particular need for heroic self-sacrifice—then which do you like?

"I don't understand the point to what you're suggesting. Eventually, the galaxy is going to have Minds in it, right? We have to find a stable state that allows big Minds and little Minds to coexist. So what's the point in waiting?"

Well... you could have the humans grow up (at some eudaimonic rate of intelligence increase), and then when new people are created, they might be created as powerful Minds to start with. Or when you create new minds, they might have a different emotional makeup, which doesn't lead them to feel overshadowed if there are more powerful Minds above them. But we, as we exist already created—we might prefer to stay on as the main characters, for now, if given a choice.

"You are showing far too much concern for six billion squishy things who happen to be alive today, out of all the unthinkable vastness of space and time."

The Past contains enough tragedy, and has seen enough sacrifice already, I think. And I'm not sure that you can cleave off the Future so neatly from the Present.

So I will set out as I mean the future to continue: with concern for the living.

The sound of six billion faces being casually stepped on, does not seem to me like a good beginning. Even the Future should not be assumed to prefer that another chunk of pain be paid into its price.

So yes, I am concerned for those currently alive, because it is that concern—and not a casual attitude toward the welfare of sentient beings—which I wish to continue into the Future.

And I will not, if at all possible, give any other human being the least cause to think that someone else might spark a better Singularity. I can make no promises upon the future, but I will at least not close off desirable avenues through my own actions. I will not, on my own authority, create a sentient superintelligence which may already determine humanity as having passed on the torch. It is too much to do on my own, and too much harm to do on my own—to amputate someone else's destiny, and steal their main character status. That is yet another reason not to create a sentient superintelligence to start with. (And it's part of the logic behind the CEV proposal, which carefully avoids filling in any moral parameters not yet determined.)

But to return finally to the Culture and to Fun Theory:

The Minds in the Culture don't need the humans, and yet the humans need to be needed.

If you're going to have human-level minds with human emotional makeups, they shouldn't be competing on a level playing field with superintelligences. Either keep the superintelligences off the local playing field, or design the human-level minds with a different emotional makeup.

"The Culture's sole justification for the relatively unworried, hedonistic life its population enjoyed was its good works," writes Iain Banks. This indicates a rather unstable moral position. Either the life the population enjoys is eudaimonic enough to be its own justification, an end rather than a means; or else that life needs to be changed.

When people are in need of rescue, this is is a goal of the overriding-static-predicate sort, where you rescue them as fast as possible, and then you're done. Preventing suffering cannot provide a lasting meaning to life. What happens when you run out of victims? If there's nothing more to life than eliminating suffering, you might as well eliminate life and be done.

If the Culture isn't valuable enough for itself, even without its good works—then the Culture might as well not be. And when the Culture's Minds could do a better job and faster, "good works" can hardly justify the human existences within it.

The human-level people need a destiny to make for themselves, and they need the overshadowing Minds off their playing field while they make it. Having an external evangelism project, and being given cute little roles that any Mind could do better in a flash, so as to "supply meaning", isn't going to cut it.

That's far from the only thing the Culture is doing wrong, but it's at the top of my list.