Friendly Artificial Intelligence

Note for readers: the last substantial update of the content below dates back to 2014 and is severely outdated. The term of "friendly AI" is no longer used by current research, replaced by "AI alignment" from around 2015. This new term is also the subject of much debate.

Note for readers: the last substantial update of the content below dates back to 2014 and is severely outdated. The term of "friendly AI""friendly AI" is no longer used by current research, replaced by ""AI alignment"" from around 2015.

A Friendly Artificial Intelligence (Friendly AI, or FAI) is a superintelligence (i.e., a really powerful optimization process) that produces good, beneficial outcomes rather than harmful ones. The term was coined by Eliezer Yudkowsky, so it is frequently associated with Yudkowsky'Yudkowsky's proposals for how an artificial general intelligence (AGI) of this sort would behave.

""Friendly AI"AI" can also be used as a shorthand for Friendly AI theory, the field of knowledge concerned with building such an AI. Note that "Friendly""Friendly" (with a capital "F""F") is being used as a term of art, referring specifically to AIs that promote humane values. An FAI need not be "friendly""friendly" in the conventional sense of being personable, compassionate, or fun to hang out with. Indeed, an FAI need not even be sentient.

However, the fact that an AI has the ability to do something doesn'doesn't mean that it will make use of this ability. Yudkowsky'Yudkowsky's Five Theses suggest that a recursively self-improving AGI could quickly become a superintelligence, and that most such superintelligences will have convergent instrumental reasons to endanger humanity and its interests. So while building a Friendly superintelligence seems possible, building a superintelligence will generally result instead in an Unfriendly AI, a powerful optimization process that optimizes for extremely harmful outcomes. An Unfriendly AI could represent an existential risk even if it destroys humans, not out of hostility, but as a side effect of trying to do something entirely different.

  1. Some AGIs may be too weak to qualify as superintelligences. We could call these 'approximately'approximately human-level AIs'AIs'. Designing safety protocols for narrow AIs and arguably even for weak, non-self-modifying AGIs is primarily a machine ethics problem outside the purview of Friendly AI - although some have argued that even human-level AGIs may present serious safety risks.
  2. Some AGIs (e.g., hypothetical safe Oracle AIs) may not optimize strongly and consistently for harmful or beneficial outcomes, or may only do so contingent on how they'they're used by human operators.
  3. Some AGIs may be on a self-modification trajectory that will eventually make them Friendly, but are dangerous at present. Calling them 'Friendly''Friendly' or 'Unfriendly''Unfriendly' would neglect their temporal inconsistency, so ''Proto-Friendly AI'AI' is a better term here.

Requiring Friendliness makes the AGI problem significantly harder, because 'Friendly AI''Friendly AI' is a much narrower class than 'AI''AI'. Most approaches to AGI aren'aren't amenable to implementing precise goals, and so don'don't even constitute subprojects for FAI, leading to Unfriendly AI as the only possible 'successful''successful' outcome. Specifying Friendliness also presents unique technical challenges: humane values are...

Read More (49 more words)

References

Eliezer S. Yudkowsky (2008). "Artificial Intelligence as a Positive and Negative Factor in Global Risk". Global Catastrophic Risks. Oxford University Press. ([1])
Cindy Mason (2015). "Engineering Kindness: Building A Machine With Compassionate Intelligence". International Journal of Synthetic Emotion. ([2])

Note for readers: the last substantial update of the content below dates back to 2014 and is severely outdated. The term of "friendly AI" is no longer used by current research, replaced by "AI alignment" from around 2015.

A Friendly Artificial Intelligence (Friendly AI, or FAI) is a superintelligence (i.e., a really powerful optimization process) that produces good, beneficial outcomes rather than harmful ones. The term was coined by Eliezer Yudkowsky, so it is frequently associated with Yudkowsky'Yudkowsky's proposals for how an artificial general intelligence (AGI) of this sort would behave.

""Friendly AI"AI" can also be used as a shorthand for Friendly AI theory, the field of knowledge concerned with building such an AI. Note that "Friendly""Friendly" (with a capital "F""F") is being used as a term of art, referring specifically to AIs that promote humane values. An FAI need not be "friendly""friendly" in the conventional sense of being personable, compassionate, or fun to hang out with. Indeed, an FAI need not even be sentient.

However, the fact that an AI has the ability to do something doesn'doesn't mean that it will make use of this ability. Yudkowsky'Yudkowsky's Five Theses suggest that a recursively self-improving AGI could quickly become a superintelligence, and that most such superintelligences will have convergent instrumental reasons to endanger humanity and its interests. So while building a Friendly superintelligence seems possible, building a superintelligence will generally result instead in an Unfriendly AI, a powerful optimization process that optimizes for extremely harmful outcomes. An Unfriendly AI could represent an existential risk even if it destroys humans, not out of hostility, but as a side effect of trying to do something entirely different.

  1. Some AGIs may be too weak to qualify as superintelligences. We could call these 'approximately'approximately human-level AIs'AIs'. Designing safety protocols for narrow AIs and arguably even for weak, non-self-modifying AGIs is primarily a machine ethics problem outside the purview of Friendly AI - although some have argued that even human-level AGIs may present serious safety risks.
  2. Some AGIs (e.g., hypothetical safe Oracle AIs) may not optimize strongly and consistently for harmful or beneficial outcomes, or may only do so contingent on how they'they're used by human operators.
  3. Some AGIs may be on a self-modification trajectory that will eventually make them Friendly, but are dangerous at present. Calling them 'Friendly''Friendly' or 'Unfriendly''Unfriendly' would neglect their temporal inconsistency, so ''Proto-Friendly AI'AI' is a better term here.

Requiring Friendliness makes the AGI problem significantly harder, because 'Friendly AI''Friendly AI' is a much narrower class than 'AI''AI'. Most approaches to AGI aren'aren't amenable to implementing precise goals, and so don'don't even constitute subprojects for FAI, leading to Unfriendly AI as the only possible 'successful''successful' outcome. Specifying Friendliness also presents unique technical challenges: humane values are very complex; a lot of seemingly simple-sounding normative concepts conceal hidden complexity; and locating encodings of human values in the physical world seems impossible to do in any direct way. It will likely be technologically impossible to specify humane values by explicitly programming them in; if so, then FAI calls for a technique for generating such values automatically.

An open problem in Friendly AI (OPFAI) is a problem in mathematics, computer science, or philosophy of AI that needs to be solved in order to build a Friendly AI, and plausibly doesn'doesn't need to be solved in order to build a superintelligence with unspecified, 'random''random' values. Open problems include:

  1. Pascal'Pascal's mugging / Pascal'Pascal's muggle
  2. Self-modification and Löb'Löb's Theorem
  3. Naturalized induction

Links

Links

References

Created by Vladimir Nesov at 4y
  1. Some AGIs may be too weak to qualify as superintelligences. We could call these 'approximately human-level AIs'. Designing safety protocols for narrow AIs and arguably even for weak, non-self-modifying AGIs is primarily a machine ethics problem outside the purview of Friendly AI.AI - although some have argued that even human-level AGIs may present serious safety risks.
  2. Some AGIs (e.g., hypothetical safe Oracle AIs) may not optimize strongly and consistently for harmful or beneficial outcomes, or may only do so contingent on how they're used by human operators.
  3. Some AGIs may be on a self-modification trajectory that will eventually make them Friendly, but are dangerous at present. Calling them 'Friendly' or 'Unfriendly' would neglect their temporal inconsistency, so 'Proto-Friendly AI' is a better term here.

However, the fact that an AI has the ability to do something doesn't mean that it will make use of this ability. Yudkowsky's Five Theses suggest that a recursively self-improving AGI could quickly become a superintelligence, and that most such superintelligences wouldwill have convergent instrumental reasons to endanger humanity and its interests (assuming its values do not evolve into Frienly ones as part of its recursive self-improvement).interests. So while building a Friendly superintelligence seems possible, building a superintelligence will, under these assumptions,will generally result instead in an Unfriendly AI, a powerful optimization process that optimizes for extremely harmful outcomes. An Unfriendly AI could represent an existential risk even if it destroys humans, not out of hostility, but as a side effect of trying to do something entirely different.

However, the orthogonality and convergent instrumental goals theses give reason to think that the vast majority of theoretically possible superintelligences will be Unfriendly. On the other hand, the generalized anti-zombie principle gives good reason to distingusih conceptual possibility from real likelihood.

Requiring Friendliness makes the AGI problem significantly harder, because 'Friendly AI' is a much narrower class than 'AI'. Most approaches to AGI aren't amenable to implementing precise goals, and so don't even constitute subprojects for FAI, leading to Unfriendly AI (either actually dangersous, or no-fun-0but-no-threat) as the only possible 'successful' outcomes. Assuming thatoutcome. Specifying Friendliness requires understanding of all human values, pecifying it will also presentpresents unique technical challenges. Humanechallenges: humane values are very complex; a lot of seemingly simple-sounding normative concepts conceal hidden complexity; and locating encodings of human values in the physical world seems impossible to do in any direct way. It will likely be technologically impossible to specify humane values by explicitly programming them in; if so, then FAI calls for a technique for generating such values automatically.

However, the fact that an AI has the ability to do something doesn't mean that it will make use of this ability. Yudkowsky's Five Theses suggest that a recursively self-improving AGI could quickly become a superintelligence, and that most such superintelligences willwould have convergent instrumental reasons to endanger humanity and its interests.interests (assuming its values do not evolve into Frienly ones as part of its recursive self-improvement). So while building a Friendly superintelligence seems possible, building a superintelligence will generallywill, under these assumptions, result instead in an Unfriendly AI, a powerful optimization process that optimizes for extremely harmful outcomes. An Unfriendly AI could represent an existential risk even if it destroys humans, not out of hostility, but as a side effect of trying to do something entirely different.

However, the orthogonality and convergent instrumental goals theses give reason to think that the vast majority of theoretically possible superintelligences will be Unfriendly. On the other hand, the generalized anti-zombie principle gives good reason to distingusih conceptual possibility from real likelihood.

Requiring Friendliness makes the AGI problem significantly harder, because 'Friendly AI' is a much narrower class than 'AI'. Most approaches to AGI aren't amenable to implementing precise goals, and so don't even constitute subprojects for FAI, leading to Unfriendly AI (either actually dangersous, or no-fun-0but-no-threat) as the only possible 'successful' outcome. Specifyingoutcomes. Assuming that Friendliness requires understanding of all human values, pecifying it will also presentspresent unique technical challenges: humanechallenges. Humane values are very complex; a lot of seemingly simple-sounding normative concepts conceal hidden complexity; and locating encodings of human values in the physical world seems impossible to do in any direct way. It will likely be technologically impossible to specify humane values by explicitly programming them in; if so, then FAI calls for a technique for generating such values automatically.

"Friendly AI" can also be used as a shorthand for Friendly AI theory, the the field of knowledge required to buildconcerned with building such an AI. Note that "Friendly" (with a capital "F") is being used as a term of art, referring specifically to AIs that promote humane values. An FAI need not be "friendly" in the conventional sense of being personable, compassionate, or fun to hang out with. Indeed, an FAI need not even be sentient.

A Friendly Artificial Intelligence (Friendly AI, or FAI) is a superintelligence (i.e., a really powerful optimization process) that produces good, beneficial outcomes rather than harmful ones. The term was coined by Eliezer Yudkowsky, so it is frequently associated with Yudkowsky's proposals for how such an artificial general intelligence (AGI) of this sort would behave.

Any AGI that is not friendly is said to be Unfriendly.

AI risk

A Friendly Artificial Intelligence (Friendly AI, or FAI) has two meanings. Its general meaning refers to anyis a superintelligence (i.e., a really powerful optimization process) that produces good, beneficial outcomes rather than harmful ones. The term was coined by Eliezer Yudkowsky, so it is frequently associated with Yudkowsky's proposals for how such an artificial general intelligence that has a positive rather than negative effect on humanity. The more specific meaning refers to the kinds of AGI designs which Eliezer Yudkowsky argues to be the only ones that can be expected to have a positive effect. The rest(AGI) of this article uses the term in its more general sense.sort would behave.

In this context, "Friendly AI" can also be used as a shorthand for Friendly AI also refers to theory, the the field of knowledge required to build such an AI. Note that Friendly (capital-F"Friendly" (with a capital "F") is being used as a term of art, referring specifically to AIs that protect humans andpromote humane values; anvalues. An FAI need not be "friendly" in the conventional sense andof being personable, compassionate, or fun to hang out with. Indeed, an FAI need not even be sentient.

An AI that underwent an intelligence explosion could exert unprecedented optimization power over its future; therefore,future. Therefore a Friendly AI could very well create an unimaginably good futurefuture, of the sort described in fun theory. However, just becausethe fact that an AI has the meansability to do something, something doesn't mean that it willmake use of this ability. Yudkowsky's Five Theses suggest that a recursively self-improving AGI could quickly become a superintelligence, and that most such superintelligences will have convergent instrumental reasons to endanger humanity and its interests. So while building a Friendly superintelligence seems possible, building a superintelligence will generally result instead in an Unfriendly AI, a powerful optimization process that optimizes for extremely harmful outcomes. An Unfriendly AI could represent an existential risk: destroying all even if it destroys humans, not out of hostility, but as a side effect of trying to do something entirely different.

Not all AGIs are Friendly or Unfriendly:

  1. Some AGIs may be too weak to qualify as superintelligences. We could call these 'approximately human-level AIs'. Designing safety protocols for narrow AIs and weak, non-self-modifying AGIs is primarily a machine ethics problem outside the purview of Friendly AI.
  2. Some AGIs (e.g., safe Oracle AIs) may not optimize strongly and consistently for harmful or beneficial outcomes, or may only do so contingent on how they're used by human operators.
  3. Some AGIs may be on a self-modification trajectory that will eventually make them Friendly, but are dangerous at present. Calling them 'Friendly' or 'Unfriendly' would neglect their temporal inconsistency, so 'Proto-Friendly AI' is a better term here.

However,...

Read More (86 more words)

AI that underwent an intelligence explosion could exert unprecedented optimization power over its future; therefore, a Friendly AI could very well create an unimaginably good future. Conversely,However, just because an AI has the means to do something, doesn't mean it will. An Unfriendly AI could represent an existential risk: destroying all humans, not out of hostility, but as a side effect of trying to do something entirely different. Just because an AI has the means to do something, doesn't mean it will.

A Friendly Artificial Intelligence (FAI) is anhas two meanings. Its general meaning refers to any artificial general intelligence that has a positive rather than negative effect on humanity. The more specific meaning refers to the kinds of AGI designs which Eliezer Yudkowsky argues to be the only ones that can be expected to have a positive effect. The rest of this article uses the term in its more general sense.

In this context, Friendly AI also refers to the field of knowledge required to build such an AI. Note that Friendly (capital-F) is being used as a term of art, referring specifically to AIs that protect humans and humane values; an FAI need not be "friendly" in the conventional sense and need not even be sentient. Any AGI that is not friendly is said to be Unfriendly.