This is a draft written by Simon Goldstein, associate professor at the Dianoia Institute of Philosophy at ACU, as part of a series of papers for the Center for AI Safety Philosophy Fellowship. Dan helped post to the Alignment Forum. This draft is meant to solicit feedback.

PDF of this draft: https://www.dropbox.com/s/a85oip71jsfxfk7/Corrigibility_shared.pdf?dl=0

Abstract: An AI is corrigible if it lets humans change its goals. This post argues that the utility aggregation framework from Pettigrew 2019 is a promising approach to designing corrigible AIs. Utility aggregators do not simply maximize their current utility function. Instead, they can change their utility function, in order to maximize expected satisfaction across present and future utility functions. I also develop two solutions to the problem of reward hacking: I suggest either penalizing utility sweetening, or penalizing effort. Finally, I compare my corrigibility approach to utility indifference (Soares et al 2015) and human compatible AI (Russell 2020).

1. Corrigibility

An AI is corrigible if it lets humans change its goals.^[1] For example, imagine that we build a paperclip maximizing AI. This AI has the goal of producing as many paperclips as possible. Then we realize that we made a mistake: we want the AI to produce staples instead of paperclips. The AI is corrigible if it lets us change its goal from maximizing paperclips to maximizing staples.

Corrigibility is important for AI safety. As AIs get more sophisticated, humans will program them with complicated goals. We will sometimes make mistakes in our choice of goals. If AIs are corrigible, then we will have chances to correct our mistakes. By contrast, an incorrigible AI with a misaligned goal could be difficult to disable.

Unfortunately, there are reasons to expect AIs to be incorrigible.^[2] The problem is that having a goal is usually an important step to achieving that goal. For this reason, when you change your goal, you affect which goal you will achieve. Return to our paperclip maximizer. If the paperclip maximizer changes its goal to staples, it will produce fewer paperclips. So the paperclip maximizer will not want to change its goal from paperclips to staples: this would get in the way of producing paperclips.

To make this reasoning more precise, I'll now explain why an AI will be incorrigible if it uses standard decisio...

Posts

Wiki Contributions

Comments