The Problem

This post is an exploration of a very simple worry about the concept of utility maximisers - that they seem capable of explaining any exhibited behaviour. It is one that has, in different ways, has been brought up many times before. Rohin Shah, for example, complained that the behaviour of everything from robots to rocks can be described by utility functions. The conclusion seems to be that being an expected utility maximiser tells us nothing at all about the way a decision maker acts in the world - the utility function does not constrain. This clashes with arguments that suggest, for example, that a future humanity or AI would wish to self-modify its preferences to be representable as a utility function.

According to Wikipedia’s definition, a decision maker can be modelled as maximising expected utility if its preferences over mutually exclusive outcomes satisfy completeness, transitivity, continuity and independence. This is neat. However, it still leaves us with free choice in ordering over our set of mutually exclusive outcomes. The more outcomes we have, the less constrained we are by utility maximisation and when we look closely there are often a LOT of mutually exclusive outcomes, if we give ourselves truly free range over the variables in question.

In the context of thought around AI safety, introductions to these axioms have not denied this but added a gloss of something like ‘if you have a consistent direction in which you are trying to steer the future, you must be an expected utility maximizer’, in this case from Benya's post in 2012. Here the work done by ‘consistent direction’ seems to imply that you have reduced the world to a sufficiently low dimensional space of outcomes. Though I agree with this sentiment, it seems fuzzy. I hope to add some clarity to this kind of qualification. To put it another way, I want to see what restrictions we need to add to regain something like the intuitive notion of the maximiser and see if it is still sufficiently general as to be worth applying to theoretical discussions of agents.

Pareto Frontier of Concepts

A quick aside: Why do we want to use the idea of a utility maximiser? The explanation that comes to mind is that it is felt to be a concept that lies on the Pareto frontier of a trade-off between the explanatory power of a concept and the lack of troubling assumptions. I doubt this framing is ne...