LLM agents^[1] seem reasonably likely to become our first takeover-capable AGIs.^[2] LLMs already have complex "psychologies," and using them to power more sophisticated agents will create even more complex "minds." A profusion of competing goals is one barrier to aligning this type of AGI.

Goal sources for LLM agents can be categorized by their origins from phases of training and operation:

Goals implicit in predictive base training on human-written texts^[3]
Goals implicit in fine-tuning training (RLHF, task-based RL on CoT, etc)^[4]
Goals specified by developer system prompts^[5]
Goals specified by user prompts
Goals specified by prompt injections
Goals arrived at through chain of thought (CoT) logic
- e.g., "I should hide this goal from the humans so they don't stop me"
- this is effectively "stirring the pot" of all the above sources of goals.
Goal interpretation changes in the course of continuous learning^[6]
- e.g., forming a new belief that the concept "person" includes self-aware LLM agents^[7]

How would such a complex and chaotic system have predictable goals in the long term? It might feel as though we should avoid this path to AGI at all costs to avoid this complexity.^[8]

But alignment researchers are clearly not in charge of the path we take to AGI. We may get little say before it is achieved. It seems we should direct much of our effort toward the specific type of AGI we're likely to get first.

The challenge of aligning agents with many goal sources may be addressable through metacognition. Just as humans can notice when they're violating their core values, LLM agents could be designed with metacognitive abilities that help maintain the dominance of specific goals. This suggests two routes to mitigating this problem:^[9]

Mechanisms and training regimes designed specifically to detect and reject unwanted goals entering the CoT
Improved general intelligence applied to reasoning clearly about which goals should be accepted and prioritized, and which goals and belief changes should rejected

Finding and specifying any goal that is actually aligned with humans' long-term interests is a separable problem. It is the central topic of alignment theory.^[10] Here I want to specifically point to the less-considered problem of multiple sources of competing goals in an LLM agent AGI. I am also gesturing in the direction of a solution, reflective stability^[7] as a result of improved metacognition.

Developers will be highly motivated to solve this problem. They have strong incentives to minimize random or destructive behavior from goals other than their own, long before agents are takeover-capable. Whether their methods hold up as capabilities increase will determine whether we survive "phase 1," aligning the first transformative AGIs.^[11] More on specific likely routes and their failure modes is layed out in System 2 Alignment.