Moral Uncertainty

Moral uncertainty (or normative uncertainty) is uncertainty about howwhat we ought, morally, to actdo given the diversity of moral doctrines. For example, suppose that we knew for certain that new technology would enable more humans to live on another planet with slightly less well-being than on Earth¹. An average utilitarian would consider these consequences bad, while a total utilitarian would endorse such technology. If we are uncertain about which of these two theories are right, what should we do?

Even with a high degree of moral uncertainty and a wide range of possible moral theories, there are still certain actions that seem highly valuable in any theory. Bostrom argues that Existential risk reduction is among them, showing that it is not only the most important task given most versions of consequentialism,consequentialism but highly recommended by many of the other widely acceptable moral theories³.

Even with a high degree of moral uncertainty and a wide range of possible moral theories, there are still certain actions that seem highly valuable in any theory. Bostrom argues that Existential risk reduction is among them, showing that it is not only it is the most important task given most versions of consequentialism, but highly recommended by many of the other widely acceptable moral theories³.

Moral uncertainty (or normative uncertainty) is uncertainty about how to act given the diversity of moral doctrines. For example, suppose that we knew for certain that a new technology would enable more humans to live on another planet with slightly less well-being than on Earth¹. An average utilitarian would consider these consequences bad, while a total utilitarian would endorse such technology. If we are uncertain about which of these two theories are right, what should we do?

Moral uncertainty includes a level of uncertainty above the more usual uncertainty of what to do given incomplete information, since it deals also with uncertainty about which moral theory is right. Even with complete information about the worldworld, this kind of uncertainty would still remain ¹. In one level of uncertainty, one can have doubts on how to act because all the relevant empirical information isn’t available, for exampleexample, choosing whether to implement or not a new technology (e.g.: AGI, Biological Cognitive Enhancement, Mind Uploading) not fully knowing about its consequences and nature. But even if we ideally get to know each and every consequencesconsequence of a new technology, we would still need to know which is the right ethical perspective for analyzing these consequences.

Even with a high degree of moral uncertainty and a wide range of possible moral theories, there are still certain actions which seemsthat seem highly valuable in any theory. Bostrom argues that Existential risk reduction is among then,them, showing that not only it is the most important task given most versions of consequentialism, but highly recommended by many of the other widely acceptable moral theories³.

Sequences

• Moral uncertainty

References

1. Crouch, William. (2010) “Moral Uncertainty and Intertheoretic Comparisons of Value” BPhil Thesis, 2010. p. 6. Available at: http://oxford.academia.edu/WilliamCrouch/Papers/873903/Moral_Uncertainty_and_Intertheoretic_Comparisons_of_Value↩
2. Sepielli, Andrew. (2008) “Moral Uncertainty and the Principle of Equity among Moral Theories". ISUS-X, Tenth Conference of the International Society for Utilitarian Studies, Kadish Center for Morality, Law and Public Affairs, UC Berkeley. Available at: http://escholarship.org/uc/item/7h5852rr.pdf↩
3. Bostrom, Nick. (2012) "Existential Risk Reduction as the Most Important Task for Humanity" Global Policy, forthcoming, 2012. p. 22. Available at: http://www.existential-risk.org/concept.pdf↩

Blog postsExternal links

References

Moral uncertainty (or normative uncertainty) is uncertainty about how to act given the diversity of moral doctrines. ItFor example, suppose that we knew for certain that a new technology would enable more humans to live on another planet with slightly less well-being than on Earth1. An average utilitarian would consider these consequences bad, while a total utilitarian would endorse such technology. If we are uncertain about which of these two theories are right, what should we do?

Moral uncertainty includes a level of uncertainty above the more usual uncertainty of what to do given incomplete information, since it deals also with uncertainty about which moral theory is right. Even with complete information about the world this kind of uncertainty would still remain ¹. In one level of uncertainty, one can have doubts on how to act because all the relevant empirical information isn’t available, for example choosing whether to implement or not a new technology (e.g.: AGI, Biological Cognitive Enhancement,Mind Uploading) not fully knowing about its consequences and nature. But even if we ideally get to know each and every consequences of a new technology, we would still need to know which is the right ethical perspective for analyzing these consequences. For example, if we known for certain that a new technology would enable more humans to live in another planet with slight less welfare than on Earth1. An average utilitarian would consider these consequences as bad, while a total utilitarian would endorse such technology. If we are uncertain about which of these two theories are right, what should we do?

Even with a high degree of moral uncertainty and a wide range of possible moral theories, there are still certain actions which seems highly valuable in any theory. Bostrom argues that Existential Riskrisk reduction is among then, showing that not only it is the most important task given most versions of consequentialismconsequentialism, but highly recommended by many of the other widely acceptable moral theories³.

One approach is to follow only the most probable theory. This has its own problems. For example, what if the most probable theory points only weakly in one way, and other theories point strongly the other way? A better approach is to “perform the action with the highest expected moral value. We get the expected moral value of an action by multiplying the subjective probability that some theory is true by the value of that action if it is true, doing the same for all of the other theories, and adding up the results.” ² However, we would still need a method of comparing value intertheories, an utilon in one theory may not be the same with an utilon in another theory. Outside consequentialism there, many ethical theories that doesn’don’t use utilions or even any quantifiable values. This is still an open problem.

Even with a high degree of moral uncertainty and a wide range of possible moral theories, there are still certain actions which seems highly valuable in any theory. Bostrom argues that Existential Risk reduction is among then, showing that not only it is the most important task given most versions of consequentialism but highly recommended by many of the other widely acceptable moral theories³.