Epistemic status: pretty confident. Probability this does well on LessWrong: pretty low

This is part 2 of my value learning sequence in which I talk about what humans actually value and how to get AGI to do it.

In this post, I wish to push back against certain visions of the future which seem both dystopian and unachievable. My personal good visions have a simple notion: humanity must seek to correct its problems, but in doing so not get overzealous in its hatred of the human form. Retro sci-fi had somewhat of the right idea (my writing for emphasis).

Many other visions expressed online from both sides of the AI safety debate seem to want to force the idea of “digital life,” “digital minds” or “uploads” onto people. This idea seems to occupy a substantial mind space of certain people. Examples –Hanson, Bostrom, Soares. I have been privy to declarations of “biology is certainly over” in the past. MIRI’s “strategy” seems to have been thought to “upload” people according to this thread. Moreover, because of this idea, actually promising AI alignment techniques get rejected, since they “can’t handle it.” In other words, visions of “upload” might be a core philosophical block that leads people to dismiss the by far most likely path to learning actual human values - just observing human behavior and inferring goals from it.

This is not how most people think. This is not what regular people want from the future, including many leaders of major nations.

It seems that the discussion of these ideas in certain circles has simply assumed that “uploads” are both possible and desirable – neither of which seems to be the case to me.

Let’s take a deep breath.

“Uploads” are likely impossible. Nobody has thought them through enough to even being to address the multiple likely impossibility results that make the process of a “living upload” either a pipe dream or the worst thing to happen to you. There is an implicit attempt to connect physics to computer science using proper philosophy and then have a culture that accepts them. Each one of the disciplines has something to say about it.

1. Physics

Your first problem is that in order to have extremely accurate beliefs about something, you have to observe it and interact with it. If you want to be perfect you either have to do this at a temperature of absolute zero or the system ends up at absolute zero after you are done with it. If you are interested in the relationship between information and temperature, see these posts: Engines of Cognition and Entropy and Temperature

This, by the way, ignores any quantum effects that might come up. You can also screw the molecule level accuracy and hope that a sufficiently large scanner gives you enough information and pray that the brain doesn’t actually use things like temperature for cognition (it probably does).

I suspect many people’s self-conception of this relies on an assumption that the ontology of Being is a solved problem (it’s not) AND that “what we ARE” are easily detectable “electrical signals in the brain,” and everything else in the body literally carries no relevant information. Parts of this are easily falsifiable through the fact that organ transplant recipients sometimes get donor’s memories and preferences (another paper here and another article here)

The issue is that either you either have to assume only a small subset of molecular information is relevant (likely a false assumption) OR you have to identify the exact large subset (more on this later) OR you run into thermodynamic information issues where you can’t actually scan a physical object to desired "each molecular location and velocity" accuracy without destroying it. This also ignores any quantum issues that could make everything even more complicated.

Now, the rest of the essay does not rely on this result and is going to assume you managed to miraculously bend some thermodynamic laws and get enough information out of the body without hurting it, you are faced with the real problem:

2. Computer Science

The problem here is – how do you verify an upload completed successfully and without errors? If I give two complex programs and ask you to verify that they have identical outputs (including “not halting” outputs) for all possible inputs is equivalent to the halting problem. Verifying that the input and output and the “subjective experience” is harder than the halting problem.

You might come up with a cached thought like “compare outputs” or something. Again, this doesn’t even work in theory. Even if you somehow claim that the human mind is not Turing complete (it probably is) and therefore the halting problem doesn’t apply, a number of issues crop up.

The space of algorithms that can pretend to be you without being you is pretty high. This problem is even worse if certain parts of you, such as your memories are easier to get than the rest of “you.” An LLM with access to your memories can sound a lot like you for the purposes of verification by another person if the only channel of communication is text. Most people are not sufficiently “unique” to have an easily discernible “output.” Besides you can’t run highly arbitrary tests on the biological human to determine “identical-ness.” This is made harder if the human was dead and frozen when you tried to do this.

On the intersection of computer science and philosophy, you also have an ontological issue of “what” to upload. A lot of people who do cryonics seem to think that “the head” is a sufficient carrier of personality. To me, it’s obvious large parts of the personality are stored in the body. Now, a lot of things stored in the body are “trauma”, however for many people “trauma” is load-bearing.

Uploading the whole body also gets into these issues of which specific cells are good and bad to upload. Fat cells? Parasites? Viruses? Which bacteria are good or bad? You might think that science has a consistent notion of “harm/benefit” that could objectively identify a cell, but it’s not really the case. Many things in the body impact cognition.

Here you have another scale of undesirable and impossible tradeoffs. If you wish to upload because you want to escape your “weak body” you may want to create certain changes to it. However, the more changes you add, the more the resulting person would differ from you and make it hard to verify it is you. The more you aim to “escape” biology, the more the verification problem gets harder, which is an issue given that it starts out being harder than the halting problem.

This doesn’t even get into the issue of many people’s cognition being more or less dependent on the specific people around them and the implied reaction those people have to a physical presence. being spending time in a luxurious resort where your only contact with people is occasional Zoom calls. Your personality will change and not in a positive direction.

The ontological issues of which cells form a true and valuable part of a human are rooted in deep disagreements about the nature of the “self.” These are not ideas from hard science that have impossibility results, so they falsely seem “more possible.” However, these disagreements have a hard time being resolved, since it is very hard to draw on the underlying principle “more meta” than this, and such universal principles may simply not exist.

Software is not the kind of thing that “just works” without verification, especially with such difficult processes, which brings me to

3. Philosophy of Self

Let’s somehow consider you have both bent the laws of thermodynamics and solved the halting problem.

If you have “succeeded,” the verification problem gets harder. How does the outside “know” that a particular upload is proceeding correctly or it has been “corrupted” through an implementation bug somewhere? Given that people are not “static” objects and the static “verifiable subcomponent of self” may not actually exist in principle, there is no grounding from which one can say that an “upload” is “developing” correctly vs. it is encountering errors.

Given that people are not static objects and create new experiences, and get new beliefs and personalities, the exact data that compromises a “digital mind” will begin to alter. How do you know these alterations are because of “reasonable development” or a software bug? This is different from the halting problem because at least in the halting problem you have two programs, and you need to know if they are identical (output and halting-wise) on the same inputs. Here you have only one program and you need to know if its essence or “soul” is identical to a properly developed “soul.” of a counter-factual program that doesn’t exist.

In essence, this is the question of mathematically identifying what the “soul” is for the purpose of verifying that the soul in the upload is still “the same” as the one uploaded. Buddha basically declared there isn’t such a thing and I am going to take his pronouncement as sufficient evidence that you just can’t do this.

In the absence of this mathematically and phenomenologically grounded idea of a soul, the way the verification problem will get solved in reality, if human beings tried to implement it, is a purely political notion of verification. If the upload agrees with all politics of the server operators, it is “working”. If it keeps agreeing with the ever-changing politics of operators, then it is “developing properly” and if not, it will be “tweaked” by any means possible to make it work.

Of course, you can get shitty statistical data on the average person. You can just give up on the “exact soul” idea and just get training data on how people developed in the past and compare this to the development of uploads in general. Two problems – this doesn’t work once the age of uploads exceeds the max human age AND you can never alter your culture from the cultures of the past, which brings me to:

4. Impossibility of a desirable upload culture

Let’s once again consider you have bent the laws of thermodynamics, solved the halting problem, and proven the Buddha wrong. You still have to solve the problem of your upload not being hurt by the people running the servers. This is less of a problem of the impossibility of creating uploads and more of a problem of desirability or impossibility of avoiding results with highly negative utility.

In some of today’s subcultures, many of the old heroes are judged harshly because they fail to comply with a particular modern standard. The statues of Thomas Jefferson and George Washington are torn down by today’s youth. J. K. Rowling is under attack. Imagine for a second that the people tearing down the statues or canceling Rowling had access to the upload of said person or they could put political pressure on the people who have access to it. What would they do it? I suspect it is nothing pretty.

If your knee-jerk reaction is to wave your hands and say we will have a culture that doesn’t hurt people, I have some bad news for you.

The problem is culture itself. Many cultures and sub-cultures crave hurting the Outgroup. Girardian mimetic competition dynamics are only going to be MORE amplified with the introduction of more similar people and minds. Therefore, the follow-up scapegoating and sacrificial dynamics are only going to become stronger.

I realize that I am drawing on Girard here, which most people are not familiar with. However, consider the general possibility that “culture-induced suffering through zero-sum high/low status designations” behaves in predictable ways with predictable invariants that could in the future be studied the same way we study conservation of energy or Gini coefficients. And effectively you can’t have high/low status designations without some people attempting to climb to “high” by hurting “low”.

To put the problem in “culture war” terminology, if you believe in “moral progress”, how do you prevent the “moral progressives” of the future from torturing your upload for not being “morally progressive” enough by their standards?

If your answer is: “This is not what moral progress means,” consider the notion that this is what “moral progress” means for many people. Consider the possibility that you are fundamentally confused about the concept of moral progress. See the value learning parts before and after for my attempt to resolve the confusion.

If your answer is “I am morally progressive compared to my peers,” I have more bad news for you. So were George Washington and J.K. Rowling. In fact, moral signaling theory requires hurting people in the near group rather than the “far group.”

Let’s all take another deep breath.

I suspect that I am going to get a lot of objections of the type: “This is solved by safe AGI.” The first problem is that some people, including some AI safety researchers, think “uploads” are some trivial technical thing easier than a basic safe AGI, as mentioned above. A “not-killing-everyone AGI” is a difficult engineering challenge, however, it is not starting down multiple impossibility results from multiple fields.

You should not expect AGI to break the laws of physics or computer science. Solving certain aspects of the ontology of Being may actually be a harder task for an AGI than building a Dyson sphere. Expecting AGI to completely alter “culture” in a particular direction too much does not grant future people the capacity to control their destiny.

If you still believe in this, consider the following scenario: you turn on your first safe AGI and humanity is still alive in a year’s time. You ask it to “solve uploads” and it tells you it’s impossible. Do you turn it off and keep building new ones until they lie to you? Or do you give up on unrealistic visions?