Inner alignment asks the question - “Is the model trying to do what humans want it to do?”, or in other wordsquestion: How can we robustly aim our AI optimizers at any objective function at all?

Goal misgeneralization due to distribution shift is another example of an inner alignment failure. It is when the mesa-objective appears to pursue the base objective during training but does not pursue it during deployment. We mistakenly think that good performance on the training distribution means that the mesa-optimizer is pursuing the base objective. However, this might have occurred only because there were some correlations in the training distribution resulting in good performance on both the base and mesa objectives. When we had a distribution shift from training to deployment it caused the correlation to be broken and the mesa-objective failed to generalize. This is especially problematic when the capabilities successfully generalize to the deployment distribution while the objectives/goals don't. Since now we have a capable system that is optimizing for a misaligned goal.

Inner Alignment Vs. Goal Misgeneralization

Goal misgeneralization is when a model learns a goal during training which corresponds to good results when observed by a human, but during deployment, it turns out that the goal being pursued was not actually what the humans had in mind and that the model has actually learned something incorrect based upon the way that the training was structured. Some researchers use the term goal misgeneralization synonymously with inner misalignment. However, others argue that goal misgeneralization should only be considered one type of possible inner misalignment. For more information see the corresponding tag.

Inner Alignment Vs. Outer Alignment

Inner alignment is often talked about as being separate from outer alignment. As mentioned theThe former deals with working on guaranteeing that we are robustly aiming at something, and the latter deals with the problem of what exactly are we aiming at. In other words, outer alignment is the problem of specifying what humans want the AI to do well enough in the first place. Outer alignment problems have their own set of problems that need to be tackled. For more information see the corresponding tag.

Related PagesPages: 

• Mesa-Optimization

• , Treacherous Turn
• , Eliciting Latent Knowledge
• , Deceptive Alignment
• , Deception

Inner alignment asks the question - “Is the model trying to do what humans want it to do?”, or in other words can we robustly aim our AI optimizers at any objective function at all?

More specifically, Inner Alignment is the problem of ensuring mesa-optimizers (i.e. when a trained ML system is itself an optimizer) are aligned with the objective function of the training process.

As an example, evolution is an optimization force that itself 'designed' optimizers (humans) to achieve its goals. However, humans do not primarily maximisemaximize reproductive success, they instead use birth control and then go out and have fun.while still attaining the pleasure that evolution meant as a reward for attempts at reproduction. This is a failure of inner alignment. 

To solve the inner alignment problem, some sub-problems that we would have to make progress on include things such as deceptive alignment, distribution shifts, and gradient hacking.

Inner Alignment Vs. Goal Misgeneralization

Goal misgeneralization is when a model learns a goal during training which corresponds to good results when observed by a human, but during deployment, it turns out that the goal being pursued was not actually what the humans had in mind and that the model has actually learned something incorrect based upon the way that the training was structured. Some researchers use the term goal misgeneralization synonymously with inner misalignment. However, others argue that goal misgeneralization should only be considered one type of possible inner misalignment. For more information see the corresponding tag.

Inner Alignment Vs. Outer Alignment

Inner alignment is often talked about as being separate from outer alignment. As mentioned the former deals with working on guaranteeing that we are robustly aiming at something, and the latter deals with the problem of what exactly are we aiming at. In other words, outer alignment is the problem of specifying what humans want the AI to do well enough in the first place. Outer alignment problems have their own set of problems that need to be tackled. For more information see the corresponding tag.

It should be kept in mind that you can have both inner and outer alignment failures together. It is not a dichotomy and often even experienced alignment researchers are unable to tell them apart. This indicates that the classifications of failures according to these terms are fuzzy. Ideally, we don't think of a binary dichotomy of inner and outer alignment that can be tackled individually but of a more holistic alignment picture that includes the interplay between both inner and outer alignment approaches.

Related Pages:Pages

• Mesa-Optimization

• Treacherous Turn
• Eliciting Latent Knowledge
• Deceptive Alignment
• Deception

Inner Alignment is the problem of ensuring mesa-optimizers (i.e. when a trained ML system is itself an optimizer) isare aligned with the objective function of the training process. As an example, evolution is an optimization force that itself 'designed' optimizers (humans) to achieve its goals. However, humans do not primarily maximise reproductive success, they instead use birth control and then go out and have fun. This is a failure of inner alignment. 

Inner Alignment is the problem of ensuring mesa-optimizers (i.e. when a trained ML system is itself an optimzer)optimizer) is aligned with the objective funcitionfunction of the training process. As an example, evolution is an optimization force that itself 'designed' optimizers (humans) to achieve its goals. However, humans do not primarily maximise reproductive success, they instead use birth control and then go out and have fun. This is a failure of inner alignment. 

Related Pages:Mesa-Optimization

External Links:

Video by Robert Miles

Inner Alignment is the problem of ensuring mesa-optimizers (i.e. when a trained ML system is itself an optimzer) is aligned with the objective funcition of the training process. As an example, evolution is an optimization force that itself desifned'designed' optimizers (humans) to achieve its goals. However, humans do not primarily maximise reproductive success, they instead use birth control and then go out and have fun. This is a failure of inner alignment. 

Inner Alignment is the problem of ensuring mesa-optimizers (i.e. when a trained ML system is itself an optimzer) is aligned with the objective funcition of the training process. As an example, evolution is an optimization force that itself desifned optimizers (humans) to achieve its goals. However, humans do not primarily maximise reproductive success, they instead use birth control and then go out and have fun. This is a failure of inner alignment. 

The term was first given a definition in the Hubinger et al paper Risk from Learned Optimization:

We refer to this problem of aligning mesa-optimizers with the base objective as the inner alignment problem. This is distinct from the outer alignment problem, which is the traditional problem of ensuring that the base objective captures the intended goal of the programmers.