We analyze the usefulness of the framework of preference types [Berridge et al. 2009] to value learning by an artificial intelligence. In the context of AI the purpose of value learning is giving an AI goals aligned with humanity. We will lay the groundwork for establishing how human preferences of different types are (descriptively) or ought to be (normatively) aggregated.

This blogpost (1) describes the framework of preference types and how these can be inferred, (2) considers how an AI could aggregate our preferences, and (3) suggests how to choose an aggregation method. Lastly, we consider potential future directions that could be taken in this area.

Motivation

The reason that the concept of multiple preference types is useful for AI is that people often have internal conflicts. Examples of internal conflicts:

You are on a diet and prefer not to eat fattening food, but still really enjoy it.
Addicts that no longer enjoy their drug.
A friend who says he doesn’t want your help, but does.
In conversation with someone racist, you may have conflicting goals. On one hand, you want them to change their mind and not be racist. On the other hand, you want to make them feel bad about their racism. Unfortunately, if you act on the latter, you are unlikely to achieve the former.
Anything you would like to do, but have trouble staying engaged with; for example, you may enjoy exercise or learning a language, and have a goal of getting fit or being fluent, but not be motivated to go for a run or open a textbook.

We think that these internal conflicts can be understood as us having preferences of different types that compete with one another. If an AI ignores the fact that we can have competing preferences, then when it considers a state it will only infer our preference for that state based on one proxy, which will often leave the AI with an incomplete picture. Examples in which taking into account only one data source for preferences leads to complications:

In the case that the AI only takes into account what humans say they want, there is the problem of people lying, leaving out details (because they haven’t thought of them), or not knowing their preferences.
In the case of inverse reinforcement learning (IRL) the AI infers preferences based on behavior only. In the diet example, if people ‘have little willpower’ then an AI using IRL may incorrectly infer that the humans

...

Wanting vs Liking

Blog posts

See also