Followup to: Math is Subjunctively Objective, The Moral Void, Is Morality Given?

Thus I recall the study, though I cannot recall the citation:

Children, at some relatively young age, were found to distinguish between:

The teacher, by saying that we're allowed to stand on our desks, can make it right to do so.

The teacher, by saying that I'm allowed to take something from another child's backpack, cannot make it right to do so.

Obert: "Well, I don't know the citation, but it sounds like a fascinating study. So even children, then, realize that moral facts are givens, beyond the ability of teachers or parents to alter."

Subhan: "You say that like it's a good thing. Children may also think that people in Australia have to wear heavy boots from falling off the other side of the Earth."

Obert: "Call me Peter Pan, then, because I never grew up on this one. Of course it doesn't matter what the teacher says. It doesn't matter what I say. It doesn't even matter what I think. Stealing is wrong. Do you disagree?"

Subhan: "You don't see me picking your pockets, do you? Isn't it enough that I choose not to steal from you—do I have to pretend it's the law of the universe?"

Obert: "Yes, or I can't trust your commitment."

Subhan: "A... revealing remark. But really, I don't think that this experimental result seems at all confusing, in light of the recent discussion of subjunctive objectivity—a discussion in which Eliezer strongly supported my position, by the way."

Obert: "Really? I thought Eliezer was finally coming out in favor of my position."

Subhan: "Huh? How do you get that?"

Obert: "The whole subtext of 'Math is Subjunctively Objective' is that morality is just like math! Sure, we compute morality inside our own brains—where else would we compute it? But just because we compute a quantity inside our own brains, doesn't mean that what is computed has a dependency on our own state of mind."

Subhan: "I think we must have been reading different Overcoming Bias posts! The whole subtext of 'Math is Subjunctively Objective' is to explain away why morality seems objective—to show that the feeling of a fixed given can arise without any external referent. When you imagine yourself thinking that killing is right, your brain-that-imagines hasn't yet been altered, so you carry out that moral imagination with your current brain, and conclude: 'Even if I thought killing were right, killing would still be wrong.' But this doesn't show that killing-is-wrong is a fixed fact from outside you."

Obert: "Like, say, 2 + 3 = 5 is a fixed fact. Eliezer wrote: 'If something appears to be the same regardless of what anyone thinks, then maybe that's because it actually is the same regardless of what anyone thinks.' I'd say that subtext is pretty clear!"

Subhan: "On the contrary. Naively, you might imagine your future self thinking differently of a thing, and visualize that the thing wouldn't thereby change, and conclude that the thing existed outside you. Eliezer shows how this is not necessarily the case. So you shouldn't trust your intuition that the thing is objective—it might be that the thing exists outside you, or it might not. It has to be argued separately from the feeling of subjunctive objectivity. In the case of 2 + 3 = 5, it's at least reasonable to wonder if math existed before humans. Physics itself seems to be made of math, and if we don't tell a story where physics was around before humans could observe it, it's hard to give a coherent account of how we got here. But there's not the slightest evidence that morality was at work in the universe before humans got here. We created it."

Obert: "I know some very wise children who would disagree with you."

Subhan: "Then they're wrong! If children learned in school that it was okay to steal, they would grow up believing it was okay to steal."

Obert: "Not if they saw that stealing hurt the other person, and felt empathy for their pain. Empathy is a human universal."

Subhan: "So we take a step back and say that evolution created the emotions that gave rise to morality, it doesn't put morality anywhere outside us. But what you say might not even be true—if theft weren't considered a crime, the other child might not feel so hurt by it. And regardless, it is rare to find any child capable of fully reconsidering the moral teachings of its society."

Obert: "I hear that, in a remarkable similarity to Eliezer, your parents were Orthodox Jewish and you broke with religion as a very young child."

Subhan: "I doubt that I was internally generating de novo moral philosophy. I was probably just wielding, against Judaism, the morality of the science fiction that actually socialized me."

Obert: "Perhaps you underestimate yourself. How much science fiction had you read at the age of five, when you realized it was dumb to recite Hebrew prayers you couldn't understand? Children may see errors that adults are too adept at fooling themselves to realize."

Subhan: "Hah! In all probability, if the teacher had in fact said that it was okay to take things from other children's backpacks, the children would in fact have thought it was right to steal."

Obert: "Even if true, that doesn't prove anything. It is quite coherent to simultaneously hold that:"

"Stealing is wrong."
"If a neutrino storm makes me believe 'stealing is right', then stealing is wrong."
"If a neutrino storm makes me believe 'stealing is right', then I will say, 'If a neutrino storm makes me believe ''stealing is wrong'', then stealing is right.'"

Subhan: "Fine, it's coherent, but that doesn't mean it's true. The morality that the child has in fact learned from the teacher—or their parents, or the other children, or the television, or their parents' science fiction collection—doesn't say, 'Don't steal because the teacher says so.' The learned morality just says, 'Don't steal.' The cognitive procedure by which the children were taught to judge, does not have an internal dependency on what the children believe the teacher believes. That's why, in their moral imagination, it feels objective. But where did they acquire that morality in the first place? From the teacher!"

Obert: "So? I don't understand—you're saying that because they learned about morality from the teacher, they should think that morality has to be about the teacher? That they should think the teacher has the power to make it right to steal? How does that follow? It is quite coherent to simultaneously hold that—"

Subhan: "I'm saying that they got the morality from the teacher! Not from some mysterious light in the sky!"

Obert: "Look, I too read science fiction and fantasy as a child, and I think I may have been to some degree socialized by it—"

Subhan: "What a remarkable coincidence."

Obert: "The stories taught me that it was right to care about people who were different from me—aliens with strange shapes, aliens made of something other than carbon atoms, AIs who had been created rather than evolved, even things that didn't think like a human. But none of the stories ever said, 'You should care about people of different shapes and substrates because science fiction told you to do it, and what science fiction says, goes.' I wouldn't have bought that."

Subhan: "Are you sure you wouldn't have? That's how religion works."

Obert: "Didn't work on you. Anyway, the novels said to care about the aliens because they had inner lives and joys—or because I wouldn't want aliens to mistreat humans—or because shape and substrate never had anything to do with what makes a person a person. And you know, that still seems to me like a good justification."

Subhan: "Of course; you were told it was a good justification—maybe not directly, but the author showed other characters responding to the argument."

Obert: "It's not like the science fiction writers were making up their morality from scratch. They were working at the end of a chain of moral arguments and debates that stretches back to the Greeks, probably to before writing, maybe to before the dawn of modern humanity. You can learn morality, not just get pressed into it like a Jello mold. If you learn 2 + 3 = 5 from a teacher, it doesn't mean the teacher has the power to add two sheep to three sheep and get six sheep. If you would have spouted back '2 + 3 = 6' if the teacher said so, that doesn't change the sheep, it just means that you don't really understand the subject. So too with morality."

Subhan: "Okay, let me try a different tack. You, I take it, agree with both of these statements:"

"If I preferred to kill people, it would not become right to kill people."
"If I preferred to eat anchovy pizzas, it would become right to eat anchovy pizzas."

Obert: "Well, there are various caveats I'd attach to both of those. Like, in any circumstance where I really did prefer to kill someone, there'd be a high probability he was about to shoot me, or something. And there's all kinds of ways that eating an anchovy pizza could be wrong, like if I was already overweight. And I don't claim to be certain of anything when it comes to morality. But on the whole, and omitting all objections and knock-on effects, I agree."

Subhan: "It's that second statement I'm really interested in. How does your wanting to eat an anchovy pizza make it right?"

Obert: "Because ceteris paribus, in the course of ordinary life as we know it, and barring unspecified side effects, it is good for sentient beings to get what they want."

Subhan: "And why doesn't that apply to the bit about killing, then?"

Obert: "Because the other person doesn't want to die. Look, the whole reason why it's right in the first place for me to eat pepperoni pizza—the original justification—is that I enjoy doing so. Eating pepperoni pizza makes me happy, which is ceteris paribus a good thing. And eating anchovy pizza—blegh! Ceteris paribus, it's not good for sentient beings to experience disgusting tastes. But if my taste in pizza changes, that changes the consequneces of eating, which changes the moral justification, and so the moral judgment changes as well. But the reasons for not killing are in terms of the other person having an inner life that gets snuffed out—a fact that doesn't change depending on my own state of mind."

Subhan: "Oh? I was guessing that the difference had something to do with the social disapproval that would be leveled at murder, but not at eating anchovy pizza."

Obert: "As usual, your awkward attempts at rationalism have put you out of touch with self-evident moral truths. That's just not how I, or other real people, actually think! If I want to bleep bleep bleep a consenting adult, it doesn't matter whether society approves. Society can go bleep bleep bleep bleep bleep -"

Subhan: "Or so science fiction taught you."

Obert: "Spider Robinson's science fiction, to be precise. 'Whatever turns you on' shall be the whole of the law. So long as the 'you' is plural."

Subhan: "So that's where you got that particular self-evident moral truth. Was it also Spider Robinson who told you that it was self-evident?"

Obert: "No, I thought about that for a while, and then decided myself."

Subhan: "You seem to be paying remarkably close attention to what people want. Yet you insist that what validates this attention, is some external standard that makes the satisfaction of desires, good. Can't you just admit that, by empathy and vicarious experience and evolved fellow-feeling, you want others to get what they want? When does this external standard ever say that it's good for something to happen that someone doesn't want?"

Obert: "Every time you've got to tell your child to lay off the ice cream, he'll grow more fat cells that will make it impossible for him to lose weight as an adult."

Subhan: "And could something good happen that no one wanted?"

Obert: "I rather expect so. I don't think we're all entirely past our childhoods. In some ways the human species itself strikes me as being a sort of toddler in the 'No!' stage."

Subhan: "Look, there's a perfectly normal and non-mysterious chain of causality that describes where morality comes from, and it's not from outside humans. If you'd been told that killing was right, or if you'd evolved to enjoy killing—much more than we already do, I mean—or if you really did have a mini-stroke that damaged your frontal lobe, then you'd be going around saying, 'Killing is right regardless of what anyone thinks of it'. No great light in the sky would correct you. There is nothing else to the story."

Obert: "Really, I think that in this whole debate between us, there is surprisingly litle information to be gained by such observations as 'You only say that because your brain makes you say it.' If a neutrino storm hit me, I might say '2 + 3 = 6', but that wouldn't change arithmetic. It would just make my brain compute something other than arithmetic. And these various misfortunes that you've described, wouldn't change the crime of murder. They would just make my brain compute something other than morality."

Part of The Metaethics Sequence

Next post: "Changing Your Metaethics"

Previous post: "Math is Subjunctively Objective"