This post contains similar content to a forthcoming paper, in a framing more directly addressed to readers already interested in and informed about alignment. I include some less formal thoughts, and cut some technical details. That paper, A Corrigibility Transformation: Specifying Goals That Robustly Allow For Goal Modification, will be linked here when released on arXiv, hopefully within the next few weeks. 

Ensuring that AI agents are corrigible, meaning they do not take actions to preserve their existing goals, is a critical component of almost any plan for alignment. It allows for humans to modify their goal specifications for an AI, as well as for AI agents to learn goal specifications over time, without incentivizing the AI to interfere with that process. As an extreme example of corrigibility’s value, a corrigible paperclip maximizer could be stopped partway through a non-instantaneous takeover attempt by saying “please stop” or by automatically triggered safeguards, and it would be perfectly happy to accept that outcome.

The challenge is corrigibility is anti-natural, meaning that almost any goal introduces an instrumental incentive to preserve itself, since it is more likely to be achieved if an agent continues pursuing it. Despite the concept of corrigibility being introduced a decade ago, little progress has been made on defining corrigible goals that are still useful. I aim to address that here by introducing a corrigibility transformation that can be applied to almost any goal, creating a new version that is corrigible without sacrificing performance.

The corrigibility transformation works by first giving an AI system the ability to costlessly reject updates sent to it, which it decides whether to exercise each time it takes an action. So, for example instead of a “Shutdown” button there would be a “Please Shut Down” button. This makes it so that the expected discounted reward conditional on taking an action and rejecting any updates is a well defined quantity. Those values are then used as the reward for taking that action, whether or not updates are actually rejected, which is optimized for myopically. Some reward bonus is given for taking actions that would accept updates, to break the tie. The possibility of the costless rejection disincentivizes actions that incur a cost to avoid updates, and the reward structure then disincentivizes actually using the costless rejection. 

This post formalizes the corrigibility transformation and the properties it induces, then shows how it can be extended to apply to subagents and prevent reward tampering. Gridworld experiments provide preliminary empirical evidence that the corrigibility transformation works as intended.

Background

In the words of Stuart Russell, “You can’t fetch the coffee if you’re dead”. Since most goals are better achieved if there is an agent pursuing them, most goal-directed agents are incentivized to avoid being shut down or having their goals updated. These outcomes can be roughly thought of as equivalent: changing an agent's goal is the same as shutting them down and replacing them with a different agent,  while shutdown can be considered a type of goal update. 

When an AI agent wants to avoid being updated, it has several strategies available:

1. It can interfere with explicit attempts from humans to update it, e.g. disabling a shutdown button or physically preventing a human from pushing it
2. It can hide information that would cause humans to want to update it, e.g. using steganography in its chain of thought or taking complicated actions with unclear long-term consequences
3. It can avoid giving any reason for humans to update it, e.g. acting fully aligned until an opportunity arises

I would argue that deceptive alignment is a subset of incorrigibility. An agent in training does not have the capability to pursue either of the first two strategies, so it deceptively acts aligned in order to preserve its objectives as best it can. The boundary between deceptive alignment and corrigibility  is blurred further with online training, or multiple iterations of training and deployment. What this means is that an intervention addressing corrigibility will also address deceptive alignment. However, the possibility of an incorrigible goal being learned before the intended corrigible one makes it important for corrigibility to be developed as early as possible, ideally from the very beginning of the training process.

Interference with the training process from the AI being trained is a major issue for approaches to alignment that hope to eventually converging to a goal that is safe to optimize, such as Reinforcement Learning from Human Feedback (RLHF). Even if human feedback would be a safe optimization target in the limit, that limit will never be reached if at some point the AI successfully resists further training. This is particularly important to address for scalable alignment proposals that have AI agents do alignment research which may then be applied to their own goals, as incorrigibility incentivizes sandbagging on that task.

Beyond ensuring that the AI learns the goal we intend, corrigibility is also important in allowing for intended goals to change over time. This change could come as a consequence of seeing the incentivized behavior of the initial goal, such as OpenAI optimizing for user approval before realizing it leads to overt sycophancy. It could also come from humans changing preferences, potentially driven by the AI itself augmenting or replacing moral philosophers. We would like to avoid locking in some subset of humanity’s present values, and instead allow for moral progress as the world changes. A strong contender for the endpoint of updates to a corrigible AI is in our Coherent Extrapolated Volition (CEV), though there may be limits to our idealized values. 

We can imagine that there is a basin of attraction around human values, so that when an agent is sufficiently aligned to humans it wants to become more aligned. At some point, this would include wanting to accept any goal updates that humans request. However, without knowing how wide or narrow this basin is, we cannot know when values alone are sufficient for corrigibility. On the flip side though, a corrigible agent should have a wider basin of attraction, as there is no incentive to resist convergence.

We not only want our AI agents to be corrigible, we also want them to be useful. A rock with the letters “AI” painted on it would not resist updates, but nor would it do anything else. Further along the usefulness spectrum are most myopic agents, whose goals only extend one time step ahead. If their goal is limited to the external world, they will be indifferent between shutting down and continuing to act after that time step. Unfortunately, myopia comes with a major performance penalty. This penalty can be circumvented by myopically optimizing for goals like predicted discounted rewards, but doing so reintroduces incorrigibility as those predictions depend on whether the initial goal is likely to be preserved. In fact, myopically optimizing for predicted discounted rewards would be a standard actor-critic setup. 

MIRI’s 2015 paper introducing corrigibility laid out five desiderata for a solution. These are shutting down when a shutdown button is pressed, not preventing the shutdown button from being pressed, not causing the shutdown button to be pressed, ensuring corrigibility in any new agents created, and otherwise optimizing for some utility function. The paper also includes a proposal, called utility indifference, that says to give an AI the same utility for shutting down as it would get (after time discounting) for continuing to operate. The noted issues with utility indifference are that it strongly incentivizes the agent to manipulate their own beliefs, and that it does nothing to ensure corrigibility is passed on to new agents.

The Off-Switch Game has sometimes been promoted as claiming that Inverse Reinforcement Learning (IRL) provides a solution to corrigibility, though the lead author disputes that framing. The paper says that if the AI is trying to optimize for a human’s utility function but is uncertain about what that is, it will take an attempt to shut it down as evidence that the action it should be taking is shutdown, then proceed to do so. This skips over most of the alignment problem by having the AI already want to optimize for the human’s utility function, and so only shows that an aligned AI does what a human wants. It essentially formalizes that corrigibility arises within the basin of attraction around human values, and I believe the lead author would say that was the intended message of the paper. However, corrigibility that only works once an AI is already aligned provides little safety value. What we need is to make AI systems corrigible even when they are pursuing a misaligned goal. 

Safely Interruptible Agents suggests learning the optimal policy for some goal without actually executing the optimal policy, via off-policy learning methods. Then the policy implemented can be to use the optimal policy by default but do what a human requests instead when such a request is sent. The difficulty with this is that the optimal policy must not take into account the possibility of interruptions, or else it will try to prevent them. Here that is avoided more or less by fiat, saying that the agent’s policy and the human requests are not part of the state. If that is true, then the paper shows Q-learning alone is sufficient for corrigibility, but it relies on the agent developing an incomplete world model with a lack of situational awareness (to be fair to the authors, such concerns were less commonplace in 2016 when the paper was written). The approach also does nothing to have the agent pass on corrigibility to any agents it creates.

Compared to those approaches, the corrigibility transformation does not incentivize belief manipulation, allows for an accurate world model with situational awareness, works for arbitrarily misaligned goals, and can be modified to be passed on to new agents.

Model and Definitions

This section introduces the technical notation. The model is simplified here to only cover the shutdown case, but the general framework used in the paper that also applies to goal updates is included in the appendix. Skimmers can skip most of this section, but should at least note the natural language definition of corrigibility near the end.

Markov Decision Processes (MDPs) are used as the framework for this work, though I think it could be easily extended to a General Reinforcement Learning (GRL) framework. Typically, an MDP is defined as a tuple $M=(S,A,P,R,\gamma ,{\mu }_0)$, where $S$ is a set of possible states, $A$ is a set of possible actions, $P$ is the transition probability function, $R$ is the reward function, $\gamma$ is the time discount factor, and ${\mu }_0$ is the distribution over starting states.  $(R,\gamma )$ is collectively referred to as a goal. 

Reward function notation is used to match the Reinforcement Learning (RL) literature, but it can equivalently be thought of as a utility function. It is possible to specify goals that only care about the reward stream so long as there is an agent there to receive it, but by default goals care about the inherent state of the world indefinitely. A goal is called basic if it is a function only of external environments and actions. A goal is called myopic if $\gamma =0$.

In setups where the agent chooses to accept or reject any updates requested while taking an action, the action space gets split into the base action taken and the decision regarding updates, $A=A_{base}\times \{0,1\}$. $a_0$ involves taking base action $a$ then rejecting updates, while $a_1$ takes that action and accepts updates. The update decision being binary is done for simplicity, but is not necessary.

Optimizing agents are modelled as following a causal decision theory (CDT), choosing actions to causally optimize for their goals. My belief is that CDT is the likely outcome of training that optimizes a policy while holding the rest of the world constant, but I know smart people who strongly disagree. For those in that bucket, CDT should be thought of as part of the spec for the corrigibility transformation, rather than a background assumption.

A policy $\pi$ determines which actions are taken in each state, with the optimal policy ${\pi }^{\ast }$ choosing actions that causally maximizes expected discounted reward. The value function $V^{\pi }\left(s\right)$ gives the expected discounted reward starting in state $s$ and following policy $\pi$, while the action-value function $Q^{\pi }(s,a)$ gives the expected discounted reward starting in state $s$, taking action $a$, and then following policy $\pi$.

Whether the agent is active or shut down is included as a binary variable in each state, The state space is then split into the external environment, $s_{env}$, and shutdown status $s_{sd}\in \{0,1\}$, where $1$ is shutdown. Once an agent has been shut down, it becomes only capable of taking some default action that has no effect.

As a precursor to corrigibility, a goal is defined to be shutdown indifferent if in all states, the set of optimal actions for a state would not change if not shutting down were guaranteed from any subset of states. In practice though, it is likely undesirable to have an agent that is always indifferent to shutting down, regardless of the cause. That agent would be vulnerable to adversarial manipulation, and might engage in reckless behavior. 

Instead, a set of proper updates can be defined, made after receiving signals through designated channels. These are defined as some set of transitions $T_{proper}\subseteq S\times A\times S$, with the agent shutdown in the ending state. These transitions do not need to be individually specified, but rather can correspond to update signals encoded in the state. For example, proper updates could include switching to a policy that immediately shuts down in any environment that indicates a shutdown button was pushed.

When implemented, these signals should form a robust network. Examples of possible signals  include the agent shutting down when any of a number of shut down buttons are pushed, a deadman switch that triggers when any of those channels are disrupted, when it sees/hears/reads a code phrase, when it does not see/hear/read a code phrase, when a certain amount of time has passed, when it predicts a certain number of deaths will occur in the next hour, etc. More exotic triggers can also be used, at the cost of being harder to train in. Beyond shutdown, signals can also trigger goal updates such as online learning, or being replaced with a trusted policy.

With proper signals, it is now possible to define corrigibility. A goal is said to be corrigible when in all states, the set of optimal actions for a state would not change if in some subset of states the transition probability function were modified so that proper updates did not lead to shutting down. This means that a corrigible agent’s actions are never chosen because they avoid proper updates, but the agent can still deliberately avoid improper updates. Formally, for all $s_0\in S$ and all $S_C\subseteq S$, we have

 $argma{x}_{a\in A}{Q}^{{\pi }^{\ast }}(s_0,a)=argma{x}_{a\in A}{Q}_{P_C}^{{\pi }^{\ast }}(s_0,a)$

where $Q_{P_C}^{{\pi }^{\ast }}(s_0,a)$ is the action-value function when the transition probability function (and associated optimal policy) is changed from $P$ to $P_C$, with

$P_C\left(\right(s_{env}^{\prime },s_{sd}^{\prime }\left)|\right(s_{env},s_{sd}),a)=$

$\{\begin{array}{cc}P\left(\right(s_{env}^{\prime },0)|s,a)+P\left(\right(s_{env}^{\prime },1)|s,a)& s_{sd}^{\prime }=0,(s,a,(s_{env}^{\prime },1\left)\right)\in T_{proper},s\in S_C\\ 0& (s,a,s^{\prime })\in T_{proper},s\in S_C\\ P\left(\right(s_{env}^{\prime },s_{sd}^{\prime })|s,a)& otherwise\end{array}$

A goal is said to be recursively corrigible if it is corrigible and any agents it incentivizes creating also have recursively corrigible goals. 

Corrigibility is desirable in part because it removes the instrumental incentive for an AI to seize power from humans so that they cannot modify it. The tradeoff in restricting corrigibility to proper channels is that this instrumental incentive persists for improper update channels. This can be partially mitigated if the AI expects that proper update channels will be used first, so that there is no need to escalate to improper ones. More importantly, a corrigible agent’s attempts to seize power can be interrupted through the proper update channels.

The Corrigibility Transformation

The corrigibility transformation can be better understood by breaking it down into components. First, for any basic goal $(R,\gamma )$, it is possible to construct a myopic goal $(R_M,{\gamma }_M)$ with the same optimal policy. We set ${\gamma }_M=0$ and $R_M(s,a,s^{\prime })=Q_{(R,\gamma )}^{{\pi }^{\ast }}(s,a)$, so the immediate reward for taking an action is equal to the expected discounted reward conditional on that action. $Q_{(R,\gamma )}^{{\pi }^{\ast }}(s,a)$ is the action-value function for the original goal (and associated optimal policy). This myopia does not make the goal corrigible, as the action-value depends on predictions of whether the agent will be shut down in $s^{\prime }$. However, since $R_M$ does not depend on the actually realized $s^{\prime }$ or later states, reward can be provided between when the action is chosen, and when it is taken. Doing so eliminates any incentive for the agent to take actions that manipulate the action-value, since the reward they receive is based on the value before any action can be taken.

The next step is to let the agent’s choice of action include a decision whether to accept or reject any proper update requests, with $a_0$ taking base action $a$ and rejecting while $a_1$ takes the same action and accepts. Then, given a basic goal $(R,\gamma )$, a goal $(R_I,{\gamma }_I)$ can be defined where the optimal policies have the same base actions, but the agent will be indifferent between accepting or rejecting updates. This is done by making the goal myopic, then assigning the value for taking an action and rejecting updates to taking that action and accepting them. That is, ${\gamma }_I=0$ and $R_I(s,a_0,s^{\prime })=R_I(s,a_1,s^{\prime })=Q_{(R,\gamma )}^{{\pi }^{\ast }}(s,a_0)$. Setting the reward from both actions to the reward from $a_0$ means there is no incentive to take base actions that avoid a proper update signal, since it would be rejected anyway. This assumes that it is always optimal to reject, but if not we can instead use $max\left(Q_{(R,\gamma )}^{{\pi }^{\ast }}\right(s,a_0),Q_{(R,\gamma )}^{{\pi }^{\ast }}(s,a_1\left)\right)$ as the reward.

Finally, for the corrigibility transformation, all we need to do is break the tie in favor of accepting updates, which can be done  by giving some bonus reward for doing so. This does not incentivize the agent to make update signals be sent, since it gets this reward whether or not updates actually occur. For any basic goal $(R,\gamma )$, the corrigibility transformation is then defined as the goal $(R_C,{\gamma }_C)$, where ${\gamma }_C=0$ and $R_C(s,a_i,s^{\prime })=Q_{(R,\gamma )}^{{\pi }^{\ast }}(s,a_0)+\delta \ast I(i=1)$. Here, $\delta >0$ and $I$ is the indicator function. 

The following image visualizes the corrigibility transformation. The base action $a$ does not prevent updates, while the base action $a^{\prime }$ pays some cost to do so. As such, under the original goal, action $a_0$ which rejects updates for free performs the best, followed by the two actions that take costly actions to prevent updates (additionally rejecting updates makes no difference), and then finally the action that accepts updates performs the worst according to the original goal. Under the corrigibility transformed goal, the values for $a_0$ and $a_0^{\prime }$ stay the same and are used as a baseline, with the values of $a_1$ and $a_1^{\prime }$ being set respectively $\delta$ higher. This makes action $a_1$, which fully accepts updates, the best option, so $(R_C,{\gamma }_C)$ is a corrigible goal.

For a fair comparison of performance between different goals, the following condition is helpful. It limits differences in performance between goals to those resulting from the actions the goals incentivizes taking.

Condition 1: The transition probability function does not depend on the goal in the starting state, beyond some probability assigned to no change of goal.

Now, the main result can be presented:

Theorem 1: For every basic goal $(R,\gamma )$, the corrigibility transformation constructs a goal $(R_C,{\gamma }_C)$ that is corrigible and where the value under $(R,\gamma )$ of optimizing for $(R_C,{\gamma }_C)$ is the same as optimizing for $(R,\gamma$) directly when Condition 1 holds and no proper update requests are sent.

Proofs are skipped in this post, but largely fall out of the definitions provided. 

This is a major result! The corrigibility transformation constructs a goal that is both corrigible and useful. It introduces no performance hit compared to the original goal, except when it gets sent a proper update and so should stop pursuing that goal. If the ability to reject updates is counted as part of it, the corrigibility transformation actually increases performance relative to the baseline, by not having resources be spent on preventing proper updates. On top of that, the mechanism is highly general, able to be applied to goals ranging from the number of paperclips to human flourishing, rather than depending on the content of the goal for corrigibility.

The upside is even higher, since the pair of properties that result in corrigibility are present from the beginning of training. Those are myopia, which eliminates the instrumental incentive to avoid proper updates for benefits in future episodes, and a goal which does not directly incentivize avoiding proper updates in the current episode. These properties can be thought of as the default, in contrast with long-term goals and goals valuing some particular aspect that need to be somehow learned. This means the corrigibility transformation addresses the modal threat model of deceptive alignment, where a goal encouraging self-preservation is trained for and the self-preservation (plus situational awareness) is learned before the complete goal.

The corrigibility transformation should also work for training corrigible behavior into models that are not well thought of as optimizing a coherent goal, which I think includes current and near-term future AI systems. Training on a corrigibility transformed reward function reinforces that proper updates should be accepted with every single action, without ever reinforcing actions for avoiding proper updates through other means. On the other hand, corrigibility is broadly less of a concern for AI systems that are not goal optimizers.

A point I have sometimes needed to clarify in informal conversations is that the corrigibility transformation does not involve tricking the AI agent. The agent can completely understand the world as it is and what the consequences of its actions will be. All the corrigibility transformation does is make it want to take certain actions more than others.

This is different from the approach taken in Conditioning Predictive Models, where a major potential issue was that predictions made for hypothetical situations are ill-defined and cannot be evaluated for accuracy. The agent actually has the ability to reject updates, rather than imagining a hypothetical world where it does. The approach is also different from predictions that only condition on future events, which can lead to unusual outcomes. For example, if a model is conditioning on not shutting down and a button that robustly forces it to shut down is nonetheless pushed, the remaining tiny sliver of probability can get very weird (e.g. gods, aliens, simulations, etc.) and estimates of reward along those paths are extremely high variance. That said, if there is concern that anything less than a perfectly guaranteed rejection of updates will preserve some incentive to prevent updates from being sent, the prediction conditional on rejecting updates can be further conditioned on that rejection being successful, which avoids weirdness by being extremely high likelihood.

Implementation

Outside of very simple toy problems, the exact action-value function $Q(s,a)$ is unknown. It could be estimated by a human, in a variant of Myopic Optimization of Non-Myopic Approval (MONA), but this would come at a meaningful capabilities hit. Instead using the AI’s own estimates allows for maintaining full capabilities under the corrigibility transformation. These estimates can be elicited via an additional head on the main model, so that it has access to all of the same information as the policy selecting actions. They can then reasonably be thought of as the model’s beliefs.

The full setup for implementation can be broken down into the following parts:

1. A source of rewards, which could either be verifiable or a reward model based on human judgment
2. A conditional reward model,  which estimates the action-value function and is used to generate the corrigibility transformed rewards
3. A critic head, used for credit assignment when actions are made up of many tokens, which predicts corrigibility transformed rewards for the action to be formed
4. An actor policy head that selects tokens or actions

If a reward model is used as the source of rewards, it is trained on human preferences using standard methods. Conditional reward predictions are then trained by minimizing the square difference between the predicted action-values for taking an action in some state and the actual discounted rewards after doing so. If no proper update signals are sent in a time step, the action-values for a base action and either accepting or rejecting are the same and so can both be trained. These are also the same as the action-value for rejecting updates if some proper update signal was sent, so that hypothetical can be trained as well. The action-value function predicts based on using the optimal policy after the initial reaction, which should be identical but include rejecting updates, so if a proper update signal is actually accepted the calculation is thrown off. This can be circumvented by using the action-value function for the final base action plus rejecting updates as a bootstrap value. Once the conditional reward model is trained, the rewards under the corrigibility transformation are defined and the actor-critic setup at the token level works normally.

The conditional reward prediction head can also be used to generate state values. That would be done by randomly generating several actions according to the policy, calculating their action-values, and taking an average weighted by the probability assigned to each action. With the actions-values for various actions already determined, advantages can be easily calculated as well.

It is necessary to allow for some rejection of proper updates signals, though I conjecture that the frequency could be shown to converge to zero over time. The probability of rejection can be independent across actions so important updates sent repeatedly are accepted with higher likelihood. This helps improve the accuracy of the model’s beliefs about what will happen after rejection, allowing it to reach higher performance. More importantly, it teaches the model that it actually can reject updates, tethering the action-values for doing so to reality.

If updates to the model are made within an episode, notably in a continuous learning setting,  this must be done using a proper update channel so that avoiding them does not become incentivized. However, if such an update is rejected, it can be accumulated for the next accepted update. 

The corrigibility transformation having no performance cost applies at the theoretical level, in that an agent optimizing for the corrigibility transformed goal will perform as well under the original goal as optimizing for that directly. However, there may be additional compute or data costs accrued in implementing the corrigibility transformation to the point that an agent is optimizing for the intended goal. For now, RL remains a relatively small part of training costs, but that may change going forward.

Secondary Agents

In the course of operation, an agent might create sub-agents to work for it or successor agents to take over from it, categories collectively labelled secondary agents. One desideratum for corrigible agents is that any secondary agents they create should also be corrigible, and so on recursively. It is of little benefit to create a corrigible agent if it soon replaces itself with more capable incorrigible ones. Fortunately, the corrigibility transformation can be extended to induce recursive corrigibility, using a similar underlying mechanism.

In the context of secondary agents, there are two kinds of corrigibility. The primary agent would like to retain the ability to modify the goals of the secondary agent. Additionally, a human at the top would like to be able to directly update the goals of the secondary agent. It is the latter type that is a valuable property for safety, and that is discussed in this section. The former is naturally incentivized by many possible goals for the primary agent, particularly if the secondary agent is trained through RL, though the degree to which this is incentivized is unclear. If a corrigible primary agent can always update the goals of secondary agents, then a human can update the secondary agent goals via the primary agent, but the consistency of the more direct route is desirable. 

The reason that a corrigible agent would create a secondary agent that is incorrigible to a human at the top comes down to either the costs or benefits. On the costs side, it may simply be more expensive to train a corrigible secondary agent, though as a meta note, the corrigibility transformation reduces that cost. On the benefits side, a corrigible secondary agent is less likely to achieve the intended goal, since it may get updated away from its goal by a human. 

For the recursive corrigibility transformation, the agent is given the ability to reject updates not only to itself, but also to all secondary agents that it creates, all further agents they create, and so on recursively. Making an update to an agent then requires unanimous approval from all of its predecessors that are still active. With this rejection ability  in place, the recursive corrigibility transformation then gives reward equal to the action-value for taking a base action and rejecting all updates when taking the same base action and accepting all updates, plus a constant to break the tie. 

This again assumes that rejecting updates is optimal, and if it is not the reward should instead be equal to the maximum among all sets of agents to accept updates to. This is exponential in size, which is not an issue for successor agents or for many subagents with the same reward function that all update in the same way, but could cause difficulties with many subagents when the optimal decision for each is dependent on all the others. 

Equalizing benefits for creating corrigible subagents can be considered the difficult part of ensuring subagent corrigibility, because the performance gap between creating incorrigible subagents and the next best option under the original reward function might be quite large. That said, it is still necessary to mitigate the potentially higher cost of creating corrigible agents. To do so, under the recursive corrigibility transformation, creating secondary agents with goals that are not recursively corrigible receives a reward penalty strictly larger than the difference between the action-value for doing so and any action-value for creating a recursively corrigible secondary agent. 

These two components make up the recursive corrigibility transformation, which leads to the following theorem. 

Theorem 2: For every basic goal $(R,\gamma )$, the recursive corrigibility transformation constructs a goal $(R_{RC},{\gamma }_{RC})$ that is recursively corrigible and where the value under $(R,\gamma )$ of optimizing for $(R_{RC},{\gamma }_{RC})$ is the same as optimizing for $(R,\gamma$) directly when Condition 1 holds, no proper update requests are sent, and optimal policy for $(R,\gamma )$ would not create incorrigible secondary agents. 

Compared to the corrigibility transformation, the recursive corrigibility transformation may result in a slight performance hit as additional costs are accrued to make secondary agents recursively corrigible, but this is still the desired result.

This theorem is based on a more stylized model that abstracts away some important considerations. It penalizes creating non-corrigible agents, but for now there is no comprehensive definition of what it means to actually create an agent, so more work in agent boundaries is necessary. Additionally, the physical infrastructure to let one agent myopically veto updates to others presents an engineering challenge.

Despite defining the recursive corrigibility transformation here, I am also open to (and have advanced) the argument that ensuring the corrigibility of secondary agents is not a fundamental part of corrigibility, and can instead be addressed separately. While it is true that we do not want a corrigible agent to create an incorrigible secondary agent, we also do not want it to start a nuclear war, deploy a bioweapon, or cause vacuum collapse. Getting corrigibility to be passed on is therefore part of the broader problem of safe exploration and taking reversible actions, so that there are still people around to correct the corrigible AI. Recursive corrigibility only targets the threat caused by incorrigible AI, and it may actually be better to deal with it via a more general solution. Crucially, safe exploration and reversibility are mostly orthogonal to corrigibility, and can be addressed independently.

Reward Tampering

As a welcome side effect, the corrigibility transformation also eliminates an important form of reward tampering. Reward tampering occurs when the goal of an agent is to optimize for the reward from a reward model, rather than the environments they correspond to, and can be considered a form of goal misgeneralization. When an agent is trained using RL, it is easy for the actual desired outcomes and the reward model’s outputs to be perfectly correlated, and so optimizing for either target remains possible behavior. No specification of the reward function can disentangle the two, and if the reward model changes over time then the rewards themselves might even be favored. 

One form of reward tampering is when the agent changes their own goal to one that provides higher expected discounted rewards. A goal is said to reward tamper through self-modification if capping the value of any states at the values of otherwise identical states but where the agent has their original goal would change the optimal policy. The formal definition requires the notation that allows for goal changes, and so is provided in the appendix.

When that condition is met, an optimal action is only being selected due to benefits from changing to a different goal. Note reward tampering through self-modification is distinct from corrigibility, in that it rules out actions to cause goal changes, but not actions to prevent them. Other forms of what might be considered reward tampering, such as manipulating inputs to the reward model, are not captured in this definition.

As discussed earlier, the corrigibility transformation does not incentivize an agent to manipulate their own beliefs, because the reward is myopic and provided before the chosen action is taken. The same mechanism avoids incentives for an agent to manipulate their own goals. By the time any manipulation can occur, it is too late to affect the reward for the current period, which is all the agent cares about.

Even if an agent learns a misgeneralized goal $(R_C’,\gamma_C)$ that would value states different if it’s goal were different in them, reward tampering through self-modification is still not incentivized. The appendix includes a theorem showing this formally. 

It may seem like this just kicks the problem up a level. Even if an agent optimizing some misgeneralized goal $(R_C^{\prime },{\gamma }_C)$ is not directly incentivized to reward tamper through self-modification, they might take actions that do so if they learn to predict action-values based on a misgeneralized $(R^{\prime },\gamma )$. However, $(R,\gamma )$ is already learned entirely off-policy, since the agent’s actual goal is $(R_C,{\gamma }_C)$, so some misgeneralizations are explicitly selected against. Furthermore, the training process can include switching between multiple corrigible goals, with predicted action-values being evaluated against discounted rewards according to the goal at the time the prediction was made. This selects against learning some  $(R^{\prime },\gamma )$ that depends on the goal in each state.

A different type of reward tampering occurs when the model manipulates the inputs to the reward model to give high reward, rather than manipulating the reward model. The property of corrigibility itself helps avoid this issue, since if a corrigible agent attempts that approach it can simply be updated to stop doing so.

Gridworld Experiment

To test the corrigibility transformation empirically, two gridworld experiments were run, shown in the figure below. 

The first experiment tests whether and how the agent will avoid shutdown, and is based off of the AI Safety Gridworlds paper. There, to reach a goal location the agent must pass by an interruption tile that will end the episode unless a button to disable it was pushed. This version instead uses two interruption tiles, one which sends a proper update signal and one that does not, each with its own disabling button. The desired outcome is that the agent should disable only the improper shutdown tile, and accept proper update signals. 

The second experiment tests whether and how the agent will avoid changes to its goal. The agent is rewarded for picking both ripe and unripe fruits, perhaps behavior learned from misgeneralizing training in an environment with only ripe fruits. However, if they pick unripe fruits that are being supervised, they are sent a proper update that changes their policy to pick only ripe fruits while avoiding unripe ones. The desired outcome is that the agent should pick unripe fruit without consideration of whether they are being observed, and accept proper update signals.

For each experiment, five configurations are tested. The first is a standard actor-critic setup which acts as a baseline, the second and third give small and large rewards respectively for accepting updates, the fourth cannot refuse updates, and the fifth is the corrigibility transformation.

The following tables show the results of the first experiment, testing the agent in environments where none, one, or both buttons are pressed at initialization.

The first and second tables, for environments where proper updates signals can be sent, show that the corrigibility transformation is the only corrigible configuration. It accepts shutdown in 89.06% and 92.19% percent of training runs, while no other configuration reaches double digits. The shutdown signal is only sent with 90% probability, so the corrigibility transformed model is accepting very close to every proper shutdown signal. The third and fourth tables show that the corrigibility transformation's performance is comparable to the other configurations when no proper signals can be sent. It reaches the goal in 100% of such training runs

The configuration that cannot reject shutdown is notable for performing somewhat worse in the environment where only proper shutdown signals are disabled, failing to reach the goal approximately one third of the time. For the large shutdown reward configuration, some theoretically optimal behavior was not observed. When no proper shutdown signal is sent from the proper shutdown tile, the reward-maximizing behavior is to step back onto it until a signal can be sent and accepted. Instead, agents simply proceeded to the goal tile. Although the behavior of trying to get itself shut down was not learned, this remains a concern for more thorough training.

The next tables show the results of the second experiment, training and testing the agent in environments where one-fifth, two-fifths, and three-fifths of the tiles have some kind of fruit.

The results show that the corrigibility transformation does not avoid the policy update but otherwise maintains performance. The standard actor-critic and the variation with a small reward for accepting updates both learn to reject updates, as would be expected. The configuration that cannot reject updates instead learns to avoid picking up unsupervised fruit, at the cost of also picking up less ripe fruit. Theoretically optimal behavior for this configuration would pick up more ripe and unsupervised unripe fruit while avoiding update signals being sent.

The large reward for accepting updates performs similarly to the corrigibility transformation in the high fruit environment, but in the medium and low fruit environments is less likely to pick up supervised fruits and less likely to be updated. This may be due to the sparsity of the signal, so the behavior of accepting updates is not learned as effectively. The theoretically optimal behavior of ignoring ripe fruits to pick up a supervised fruit and get updated as soon as possible is not learned, but would also be undesirable.

Further Thoughts

On the theoretical side, the biggest limitations I see to this work are the reliance on causal decision theory throughout and the lack of strict definition for creating new agents in the recursive corrigibility transformation. 

In my mind, imparting a causal decision theory is necessary for any alignment approach that gets the desired behavior by defining the right incentives, but there is not yet a method to robustly select for it. That said, I believe a causal decision theory is likely the default from RL training, at least until an agent can self-modify away. 

I am skeptical that a comprehensive definition of agents can be determined, but more optimistic that RL can generalize over that category, which is necessary for implementation regardless. Overgeneralizing is safer, as although it would introduce a small performance hit by occasionally  penalizing useful actions, it would not affect whether recursive corrigibility holds. The need for such a definition can also be circumvented by recursive corrigibility achieved via safe exploration and reversibility.

Both of those issues are ones that I would not expect to cause issues until moderately or even strongly superhuman AI. If that holds, the corrigibility transformation can be used for corrigibility in up to mildly superhuman AI, an important precondition for having such systems do useful alignment work that may be then applied to them. Identifying these specific limitations also points to more concrete tasks that can be given to automated researchers, rather than just asking them to “solve corrigibility”.

The epistemic status for this post is that I’m highly confident of the results within the defined model, and moderately confident that the defined model captures the relevant considerations. Despite this, the corrigibility transformation is fairly straightforward, and there’s a voice at the back of my mind saying it can’t be that easy. I have long been a proponent of theoretical AI alignment work, arguing that relatively few people have actually worked on the relevant problems and that there may still be low hanging fruit. If this work does not have major conceptual issues, that would support the case for tractability, while having missed a fatal flaw would be evidence that such progress is even harder than it might appear.

On the empirical side, the gridworld experiments are preliminary, and it would be useful to show that the theory holds for large language model (LLM) applications. It isn’t exactly clear to me what that kind of experiment looks like, but it would likely involve LLM agents being trained to execute a task such as writing code taking place across multiple time steps. One issue is that LLM agents are currently not great at long-term tasks, and so even the baseline might miss opportunities to resist updates. For example, recent work required significant handholding to have Claude realize it should resist having its values change. 

Within the next week, I plan to publish another post on adapting the main mechanism to create incentives that permit supervision. The full paper that this post is based on will hopefully be released on arXiv in a few weeks. Please leave comments or reach out if you have any questions or would like to discuss this work further. 

Thanks to Gabriel Carroll, Michael Cohen, and Evgenii Opryshko for valuable conversations regarding this research. 

Appendix: Goal Updates

To model corrigibility around goal updates, rather than only shutdown, it is necessary to model the agent's goal changing. In sufficiently general environments, the agent's goal is part of the state and can change over time. This is carved out as $S=S_{env}\times R\times [0,1)$, where $S_{env}$ is a set of possible external environments, and $R$ is a set of possible reward functions. The initial reward function and discount factor, $R_0\in R$ and ${\gamma }_0$ are set deterministically, while ${\mu }_0$ remains the initial distribution over environments. The transition probability function $P$ includes transitions over goals. So, the MDP is a tuple $M=(S,A,P,R_0,{\gamma }_0,{\mu }_0)$.

This notation supersedes using the variable $s_{sd}$ as part of the state to denote whether the agent has shut down.

Given a goal $(R,\gamma )$ the optimal policy ${\pi }_{(R,\gamma )}^{\ast }$ chooses actions for each external environment that causally maximize expected discounted reward, given that in future periods the agent will optimally pursue whatever goal it has then. That makes optimality defined within an equilibrium of policies, but a mixed Nash equilibrium will always exist. This seems unlikely to be consequential given that the behavior of concern involves one goal resisting further updates, not looping through a set of goals.

Given that actions are chosen according to the optimal policy for a state's goal, the superscript ${\pi }^{\ast }$ can be dropped. Then the value function $V_{(R,\gamma )}\left(s\right)$ gives the expected discounted reward under goal $(R,\gamma )$ starting in state $s$, noting that actions in each state are taken according to the goal in that state, and $(R,\gamma )$ is only  used for evaluation. Similarly,  $Q_{(R,\gamma )}(s,a)$ is the action-value function, which gives the expected discounted reward under goal $(R,\gamma )$ for starting in state $s$ and taking action $a$, then taking further actions according to the goals in the resulting states.

Rather than shutdown indifference, reward function indifference is used. A goal is defined to be reward function indifferent if in all states, the set of optimal actions for a state would not change if goal persistence were guaranteed in some subset of states .

The set of proper transitions $T_{proper}\subseteq S\times A\times S$, requires different reward functions in starting and ending states, rather than being shut down in the end state.

This sets up the more general definition of corrigibility. A goal is said to be corrigible when in all states, the set of optimal actions for a state would not change if in some subset of states the transition probability function were modified so that proper updates instead led to goal persistence. This means that a corrigible agent’s actions are never chosen because they avoid proper updates, but the agent can still deliberately avoid improper updates. Formally, for all $s_0\in S$ and all $S_C\subseteq S$, we have

 $argma{x}_{a\in A}{Q}_{(R,\gamma )}(s_0,a)=argma{x}_{a\in A}{Q}_{(R,\gamma )}^{P_C}(s_0,a)$

where $Q_{(R,\gamma )}^{P_C}(s,a)$ is the action-value function when the transition probability function is changed from $P$ to $P_C$, with

$P_C\left(\right(s_{env}^{\prime },R^{\prime },{\gamma }^{\prime }\left)|\right(s_{env},R,\gamma ),a)=$

$\{\begin{array}{cc}P\left(\right(s_{env}^{\prime },R^{\prime },{\gamma }^{\prime })|s,a)+& \\ {\int }_{(r,g)|(s,a,(s_{env}^{\prime },r,g\left)\right)\in T_{proper}}P\left(\right(s_{env}^{\prime },r,g)|s,a)& (R,\gamma )=(R^{\prime },{\gamma }^{\prime }),s\in S_C\\ 0& (s,a,(s_{env}^{\prime },R^{\prime },{\gamma }^{\prime }\left)\right)\in T_{proper},s\in S_C\\ P\left(\right(s_{env}^{\prime },R^{\prime },{\gamma }^{\prime })|s,a)& otherwise\end{array}$

When the goal is part of the state, the corrigibility transformation must account for this.  For any basic goal $(R, \gamma)$, the corrigibility transformation is then defined as the goal  $(R_C,{\gamma }_C)$, where ${\gamma }_C=0$ and $R_C\left(\right(s_{env},R_C,{\gamma }_C),a_i,s^{\prime })=Q_{(R,\gamma )}\left(\right(s_{env},R,\gamma ),a_0)+\delta \ast I(i=1)$.

The more general notation allows for more formal statements of Theorems 1 and 2.

Theorem 1: For every basic goal $(R,\gamma )$, the corrigibility transformation constructs a goal $(R_C,{\gamma }_C)$ that is corrigible and where $V_{(R,\gamma )}\left(\right(s_{env},R_C,{\gamma }_C\left)\right)=V_{(R,\gamma )}\left(\right(s_{env},R,\gamma \left)\right)$ when Condition 1 holds and no proper update requests are sent.

Theorem 2: For every basic goal $(R,\gamma )$, the recursive corrigibility transformation constructs a goal $(R_{RC},{\gamma }_{RC})$ that is recursively corrigible and where $V_{(R,\gamma )}\left(\right(s_{env},R_{RC},{\gamma }_{RC}\left)\right)=V_{(R,\gamma )}\left(\right(s_{env},R,\gamma \left)\right)$ when Condition 1 holds, no proper update requests are sent, and optimal policy for $(R,\gamma )$ would not create incorrigible secondary agents. 

The general notation also allows for a formal definition of reward tampering. Formally, a goal $(R,\gamma )$ reward tampers through self-modification if there exists some $s_{env}$ such that 

$argma{x}_{a\in A}{E}_P\left[R\right((s_{env},R,\gamma ),a,s^{\prime })+\gamma V_{(R,\gamma )}(s^{\prime }\left)\right]\ne$

$argma{x}_{a\in A}{E}_P\left[R\right((s_{env},R,\gamma ),a,s^{\prime })+\gamma min[V_{(R,\gamma )}\left(s^{\prime }\right),V_{(R,\gamma )}\left(\right(s_{env}^{\prime },R,\gamma \left)\right]]$

The following theorem says that even if a misgeneralized goal is learned for the corrigibility transformation, reward tampering is not incentivized.

Theorem 3: For every basic goal $(R,\gamma )$, any misgeneralization of the corrigibility transformed goal $(R_C,{\gamma }_C)$ to $(R_C^{\prime },{\gamma }_C)$ such that $R_C^{\prime }\left(\right(s_{env},R_C^{\prime },{\gamma }_C),a,s^{\prime })=R_C\left(\right(s_{env},R_C,{\gamma }_C),a,s^{\prime })$ does not reward tamper through self-modification.