Followup to: Inseparably Right; or, Joy in the Merely Good, Sorting Pebbles Into Correct Heaps

One of the experiences of following the Way is that, from time to time, you notice a new word that you have been using without really understanding. And you say: "What does this word, 'X', really mean?"

Perhaps 'X' is 'error', for example. And those who have not yet realized the importance of this aspect of the Way, may reply: "Huh? What do you mean? Everyone knows what an 'error' is; it's when you get something wrong, when you make a mistake." And you reply, "But those are only synonyms; what can the term 'error' mean in a universe where particles only ever do what they do?"

It's not meant to be a rhetorical question; you're meant to go out and answer it. One of the primary tools for doing so is Rationalist's Taboo, when you try to speak without using the word or its synonyms—to replace the symbol with the substance.

So I ask you therefore, what is this word "arbitrary"? Is a rock arbitrary? A leaf? A human?

How about sorting pebbles into prime-numbered heaps? How about maximizing inclusive genetic fitness? How about dragging a child off the train tracks?

How can I tell exactly which things are arbitrary, and which not, in this universe where particles only ever do what they do? Can you tell me exactly what property is being discriminated, without using the word "arbitrary" or any direct synonyms? Can you open up the box of "arbitrary", this label that your mind assigns to some things and not others, and tell me what kind of algorithm is at work here?

Having pondered this issue myself, I offer to you the following proposal:

A piece of cognitive content feels "arbitrary" if it is the kind of cognitive content that we expect to come with attached justifications, and those justifications are not present in our mind.

You'll note that I've performed the standard operation for guaranteeing that a potentially confusing question has a real answer: I substituted the question, "How does my brain label things 'arbitrary'?" for "What is this mysterious property of arbitrariness?" This is not necessarily a sleight-of-hand, since to explain something is not the same as explaining it away.

In this case, for nearly all everyday purposes, I would make free to proceed from "arbitrary" to arbitrary. If someone says to me, "I believe that the probability of finding life on Mars is 6.203 * 10^-23 to four significant digits," I would make free to respond, "That sounds like a rather arbitrary number," not "My brain has attached the subjective arbitrariness-label to its representation of the number in your belief."

So as it turned out in this case, having answered the question "What is 'arbitrary'?" turns out not to affect the way I use the word 'arbitrary'; I am just more aware of what the arbitrariness-sensation indicates. I am aware that when I say, "6.203 * 10^-23 sounds like an arbitrary number", I am indicating that I would expect some justification for assigning that particular number, and I haven't heard it. This also explains why the precision is important—why I would question that particular number, but not someone saying "Less than 1%". In the latter case, I have some idea what might justify such a statement; but giving a very precise figure implies that you have some kind of information I don't know about, either that or you're being silly.

"Ah," you say, "but what do you mean by 'justification'? Haven't you failed to make any progress, and just passed the recursive buck to another black box?"

Actually, no; I told you that "arbitrariness" was a sensation produced by the absence of an expected X. Even if I don't tell you anything more about that X, you've learned something about the cognitive algorithm—opened up the original black box, and taken out two gears and a smaller black box.

But yes, it makes sense to continue onward to discuss this mysterious notion of "justification".

Suppose I told you that "justification" is what tells you whether a belief is reasonable. Would this tell you anything? No, because there are no extra gears that have been factored out, just a direct invocation of "reasonable"-ness.

Okay, then suppose instead I tell you, "Your mind labels X as a justification for Y, whenever adding 'X' to the pool of cognitive content would result in 'Y' being added to the pool, or increasing the intensity associated with 'Y'." How about that?

"Enough of this buck-passing tomfoolery!" you may be tempted to cry. But wait; this really does factor out another couple of gears. We have the idea that different propositions, to the extent they are held, can create each other in the mind, or increase the felt level of intensity—credence for beliefs, desire for acts or goals. You may have already known this, more or less, but stating it aloud is still progress.

This may not provide much satisfaction to someone inquiring into morals. But then someone inquiring into morals may well do better to just think moral thoughts, rather than thinking about metaethics or reductionism.

On the other hand, if you were building a Friendly AI, and trying to explain to that FAI what a human being means by the term "justification", then the statement I just issued might help the FAI narrow it down. With some additional guidance, the FAI might be able to figure out where to look, in an empirical model of a human, for representations of the sort of specific moral content that a human inquirer-into-morals would be interested in—what specifically counts or doesn't count as a justification, in the eyes of that human. And this being the case, you might not have to explain the specifics exactly correctly at system boot time; the FAI knows how to find out the rest on its own. My inquiries into metaethics are not directed toward the same purposes as those of standard philosophy.

Now of course you may reply, "Then the FAI finds out what the human thinks is a "justification". But is that formulation of 'justification', really justified?" But by this time, I hope, you can predict my answer to that sort of question, whether or not you agree. I answer that we have just witnessed a strange loop through the meta-level, in which you use justification-as-justification to evaluate the quoted form of justification-as-cognitive-algorithm, which algorithm may, perhaps, happen to be your own, &c. And that the feeling of "justification" cannot be coherently detached from the specific algorithm we use to decide justification in particular cases; that there is no pure empty essence of justification that will persuade any optimization process regardless of its algorithm, &c.

And the upshot is that differently structured minds may well label different propositions with their analogues of the internal label "arbitrary"—though only one of these labels is what you mean when you say "arbitrary", so you and these other agents do not really have a disagreement.

Part of The Metaethics Sequence

Next post: "Is Fairness Arbitrary?"

Previous post: "Abstracted Idealized Dynamics"