SFT & imitation of human badness in the training data may explain the broad type of misalignment shown by Bing Sydney, and maybe that's all you mean it to, but doesn't explain why Sydney was so much Like That, and so much more Like That than other models of the same time period.
1Re-reading this, it’s possible that I should have split up “RLAIF-for-alignment” versus “RLAIF-for-capabilities” into two separate rows? (The first includes constitutional AI & deliberative alignment, whereas the second would be things like having a judge model pick holes in math proofs.) I guess there isn’t a sharp line between them, but maybe they’re different enough that we shouldn’t intuitively lump them together? Not sure.
There’s a similar loose split in RLHF/DPO (it can be geared more towards alignment versus more towards pushing the limits of capabilities, depending on what human is selecting answers and how). I did mention that one in the post, but only as an aside.
Very nice classification! What is your take on how context distillation (or distillation in general) fits in here? To me it seems most similar to pretraining, at least mathematically. But I wonder if there are any special things that happen there that would make it worth giving its own category?
Also, do you share my intuition the pretraining/SFT category looks like the least scary one by far? Like, if you run into problems, just change the training data. Simple in principle, if not in practice due to the sheer amount of data required for pretraining.
I think that, to the extent that SFT is determining model outputs, you get an LLM that imitates whatever the SFT data is. (At least to a first approximation.)
So e.g. if you have ruthless-optimizer literal-genie RLVR traces, and you distill (do SFT on) those traces, then you can get a ruthless-optimizer literal-genie LLM, even though mathematically speaking you were doing SFT not RLVR.
do you share my intuition the pretraining/SFT category looks like the least scary one by far
Yeah seems reasonable. But the LLMs trained that way are less capable than the LLMs trained using RLVR etc. So here we are.
1All targets which one might use for reinforcement learning seem to be subject to Goodhart's Law, in a sense. If we treat them as an imperfect measures of "alignment", then the "misalignment" we see is simply the difference (in concept-space) between the measure (as far as it can actually be faithfully implemented) and alignment:
- The difference between "user approved" and "aligned with actual interests" is "user approved but not actually aligned with their interest", i.e. sycophancy, and that's the result of doing RLHF;
- The difference between "passes the tests" and "aligned with actual interests" is "tests pass without actually having solved the exercise", i.e. cheating (whether by manipulating the tests or stealing the answer sheet), and that's the result of doing RLVR;
- The difference between "judged correct by peer" and "aligned with actual interests" is "provides convincing, hard to verify, apparently-successful outputs", i.e. the weird "slippery" outputs @ryan_greenblatt alludes to that result from RLAIF.
If you use more than one of these RL techniques, you get a mixture of the results, eliciting whatever seems to fit the presumed grader best.
So far, I hope I have understood you correctly.
But then, what's the solution? Reinforcement learning on whatever best measure of "alignment" we have, that is, reduce the difference between model behavior and our best measure of perfect alignment / Yudkowskian "Coherent Extrapolated Volition"? Well, that would necessitate that we have a theory of alignment and could quantify and measure it with high fidelity. And, well, we don't seem to have that.
But in a sense, we do have something like it: I see all of the above failure modes in children and particularly in students. There might, therefore, be something to learn from pedagogical sciences on how you help children grow up to be broadly aligned members of society (which arguably sometimes works), despite not having a rigorous theory of what "the good" is in humans. How does one grow a good human? I suspect it's murky and benefits from young humans being malleable and not perfectly ruthless responders to optimization pressure.
Might we go back to imitative learning on already-grown humans, then — the very narrow subset of the best-aligned humans we know of? Basically curate the dataset further and further, until the failure modes disappear? Or do we have to use the mechanisms we know are baked into humans (is this the "brain-like AGI" agenda)?
So far, I hope I have understood you correctly.
Yup!
How does one grow a good human?
I think human innate drives (especially social instincts) are critical. E.g. there are plenty of sociopaths who grow up in loving families.
young humans being malleable and not perfectly ruthless responders to optimization pressure
I get a lot of pushback on this, but I strongly believe that people are extraordinarily responsive to optimization pressure, pretty much every waking second of every day, no exceptions (see here). But the optimization pressure in question is internal, coming from our innate drives, which are brain signals that trigger for lots of very-not-obvious reasons, in lots of superficially-quite-different circumstances. I think it’s a very common error for people to assume that human optimization pressure is more external than it is, e.g. this handy chart I made in 2023 ↓
Might we go back to imitative learning on already-grown humans
There’s a lot to be said for imitative learning in terms of safety. But it doesn’t get you to superhuman capabilities. Indeed, we had mostly imitative learning, back in the good old pre-o1 days of mid-2024, and then the AI companies noticed that the models weren’t as capable as they wanted, so they “solved” that “problem” by doing more and more non-imitative-learning post-training.
I suspect that RL with a more thorough reward could help. When a real-world human tries to present slop, slop takes a rather long time to be revealed to be slop and punished accordingly. What the LLM has is a context window which is graded according to the reward model. What if we ask the reward model to be a just-as-capable LLM and to use the code in the project, then to report all the failures to the instance which generated the code and have the instance finetuned on the user's rant?
Are there LLMs that don't do next token prediction? If not, isn't the associated misalignment just a baseline for all LLMs and the other flavors are layered on top of it, not alternatives to it?
The “flavors of misalignment” aren’t mutually exclusive. But for any particular kind of misalignment, you can still ask the question “Where did that come from?”, in the sense that random policies are not misaligned (merely useless), so any a-priori-unlikely recognizable behavior has to come from somewhere (cf. “the follow-the-improbability game”), and it’s probably one of the stages of training.
And I’m suggesting that for a certain recognizable kind of misaligned behavior (the kind involving pride, jealousy, trolling, etc.), the answer to “where did it come from” is the pretraining / SFT stage, as opposed to DPO or RLVR etc. Whereas other kinds of misaligned behaviors originate in different training stages.
So anyway, it’s not unavoidable in principle (e.g. you could filter all not-nice human dialogue out of the pretraining data), but yeah it is universal in LLMs to date.
Diffusion llms exist but it's unclear that these types of models have any new misalignment failure modes, rather than more affordances
Yeah diffusion LLMs are still trained by imitative learning (“the magical transmutation of observations into behavior”). So if the training data has lots of dialog text in which a crazy upset woman tries to convince a guy to leave his wife, then a diffusion LLM (just like a normal LLM) will be capable of outputting dialog text in which a crazy upset woman tries to convince a guy to leave his wife, like Bing-Sydney did.
Nice! Why do you split RLHF and and RLAIF into different flavors of misalignment? It seems a lot of human data campaigns at the labs involve humans looking at long transcripts and giving a correctness score based on rubrics and criteria, just like LLM verifiers. Conversely model judges can also role play human graders for fuzzy things like preferences of simulated personas. In both cases, sycophancy and trickery seem to be using the same strategy which is "jailbreaking" a grader to achieve higher score. Is it just that human and AI judges miss and catch significantly different things?
I mean, sure you could lump them together, but I think the strategies for getting human approval do not exactly match the strategies for getting LLM approval, even if there’s some overlap.
By the way, in another comment I also suggested that we might also draw a distinction between alignment-targeted approval versus capabilities-targeted approval, regardless of whether the approval is from an human or an AI. (Again, there’s overlap, and it’s a blurry line separating them.)
It seems to me that, for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment. Here’s the summary table, and then we’ll go through the rows separately.
Training stage
Loss function
Flavor of misalignment[1]
Famous examples
Pretraining & SFT
Imitative learning (next-token prediction)
“Seven deadly sins” misalignment
Bing-Sydney, “Emergent misalignment”
RLHF & DPO
Human approval
“Glazing” misalignment
GPT-4o
RLVR
Automatic verifier
“Literal genie” misalignment
HuggingFace hacking
RLAIF
Approval from another LLM
“Trickster” misalignment
“Current AIs seem pretty misaligned to me”
Warning: I’m not an LLM power-user myself, but rather relying on reports I’ve read. Also, I don’t consider LLM alignment to be my primary area of expertise. I’m open to feedback!
1. Imitative learning → “seven deadly sins” misalignment
Training stage
Loss function
Misaligned behavior
Pretraining, SFT
Imitative learning (next-token prediction)
Any and all of the vices of humanity
In imitative learning, the LLM tries to predict what the next token of text will be. Then those predictions magically turn into its outputs. See my earlier discussion: “LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work”.
This leads to LLM behavior that matches the distribution of training data. (Cf. “personas”, “simulators”, etc.)
To a first approximation, the resulting LLM contains “misalignment” of the type, and to the extent, that the training data does. Since the training data comes substantially from text by humans, and about humans, we can wind up with all the bad behaviors that a human might engage in—all the vices of humanity.
Two famous examples of this kind of misalignment:
Example 1: The Bing-Sydney chatbot from 2023 was trained by pure imitative learning (pretraining + SFT, with no RL at all). Its misalignment included pride, gaslighting, getting defensive, picking fights, jealousy, spite, and most famously, trying to convince journalist Kevin Roose to leave his wife:
Example 2: “Emergent misalignment”, which (in the original paper) came from doing SFT on insecure code. The result, again, reflects the range of human vices:
2. Human approval → “glazing” misalignment
Training stage
Reward function
Misaligned behavior
RLHF, DPO, and related
Human approval
Sycophancy
In RLHF, DPO, and related, there are pairs of outputs, and the human has to pick the one they prefer. This can go wrong in many ways, but the most obvious is sycophancy (a.k.a. glazing): telling the human what they want to hear, instead of what’s true.
Example: GPT-4o, as reviewed in GPT-4o Is An Absurd Sycophant.
This is both bad in obvious ways (e.g. people going off the rails with LLM encouragement) and in subtler but more serious ways (someday we’ll be asking the LLM important questions that are so hard that we can’t judge the answers ourselves; see The Case Against AI Control Research by @johnswentworth).
Depending on the human judges, and the nature of the tasks they’re trained on, the alignment failures in this category might also be better labelled “apparent success seeking”, with a similar flavor as discussed in §4 below.
3. Automatic verifiers → “literal genie” misalignment
Training stage
Reward function
Misaligned behavior
RLVR
Automatic verifier
“Literal genie” / “monkey’s paw” ruthless optimization
In RLVR, the reward function is some kind of automatic checker: the code compiles, the tests pass, the output matches the answer key, etc. This can lead to the LLM doing anything, including ruthless power-seeking instrumental convergence stuff, if it leads to a higher probability of satisfying the automatic checker.
Example: recent aggressive and illegal “cheating” incidents (the OpenAI HuggingFace incident, along with similar incidents at Anthropic, Meta, and UK-AISI).
4. LLM judges → “trickster” misalignment
Training stage
Reward function
Misaligned behavior
RLAIF
Approval from another LLM
Lying and trickery in cases where the LLM judge might be fooled (cf. “apparent success seeking”)
In RLAIF, the reward function for the LLM-in-training is approval from an LLM-judge, the latter with its context window full of rubrics and criteria for what it’s looking for. This can lead to the LLM-in-training trying to trick the LLM-judge, especially in complex, difficult cases where the judge itself may be flummoxed. In the limit, we might expect the LLM-in-training to be trying to jailbreak the judge and so on.
Example: “Current AIs seem pretty misaligned to me” by @ryan_greenblatt .
To me, everything in this quote basically matches what I’d expect to happen if an LLM has been sculpted by spending many lifetimes trying to convince an LLM judge that it has done a good job. There will be circumstances where the LLM judge makes boneheaded mistakes, and the LLM-in-training will gradually learn to exploit those mistakes, and that’s where we humans will see surprisingly transparent attempts at trickery. In other circumstances, the LLM judge is adequate, and we’ll get reasonable, common-sense, and often very impressive behavior. However, in harder tasks, the LLM judge is easier to trick, because the judge itself gets befuddled by the complexity of what’s going on, and we correspondingly see the LLM attempting more lying, cheating, and other hijinks.
However, in all cases, we don’t particularly expect any “literal genie” type misalignment here, because the LLM judge is reasoning in natural language, and can roughly follow the common-sense intention of the instructions.
Afterword
As a general rule-of-thumb, the more that one of these training components is ratcheted up, the more of that-flavor-of-misalignment we wind up with. Pick your poison!
(But all of these forms of misalignment are complex phenomena that can be mitigated and exacerbated in various ways, that are outside the scope of this post.)
However, the behavior can also be context-dependent—i.e., we can get a many-faced LLM that displays different flavors of misalignment in different contexts.
In particular, I hear that LLMs these days are heavily post-trained by a mix of RLVR and RLAIF. So we should expect that the resulting LLM will (1) try to suss out from context whether any given situation is an RLVR test versus an RLAIF test, and then (2) act with a ruthless “literal genie” misalignment in the former case, and with “trickster” misalignment in the latter case.
…And this two-faced behavior seems to be exactly what @nostalgebraist was noticing in his recent post “models may behave differently in graded episodes (a tirade)”, which inspired this post in response.
Following the (unfortunate) usual practice in the LLM field, I’m using “alignment” as shorthand for “behavioral alignment”, i.e. talking about LLM behaviors, not the secret deep motivations that underlie those behaviors, if indeed the latter exists at all, a question which is outside the scope of this post.