Summary

Generalization is one lens on the alignment challenge. We'd like network-based AGI to generalize ethical judgments as well as some humans do. Broadening training is a classic and obvious approach to improving generalization in neural networks.

Training sets might be broadened to include decisions like whether to evade human control, how to run the world if the opportunity arises, and how to think about one's self and one's goals. Such training might be useful if it's consistent with capability training. But it could backfire if it amounts to lying to a highly intelligent general reasoning system.

Broader training sets on types of decisions

Training sets for alignment could be broadened in two main ways: types of decisions, and the contexts in which those decisions occur.

Any training method could benefit from better training sets, including current alignment training like constitutional AI. The effects of broadening alignment training sets can be investigated empirically, but little work to date directly addresses alignment. Broadening the training set won't solve alignment on its own. It doesn't directly address mesa-optimization concerns. But it should^[1] help as part of a hodge-podge collection of alignment approaches.

This is a brief take on a broad topic. I hope to follow this with a more in-depth post, but I also hope to get useful feedback on the relatively quick version.

Alignment generalization^[2] is more nuanced than IID vs OOD

I was a researcher from about 2000 to 2022 in a lab that built and trained neural networks as models of brain function. We talked about generalization a lot, in both networks and humans. We needed fine-grained concepts. The standard framing just distinguishes testing on randomly selected hold-outs from the training set (IID, independently and identically distributed) and testing on examples that weren't in the training set (OOD, out-of-distribution). That binary categorization is too broad. How far out of distribution, and in what ways, is crucial for actual generalization.

Here's an iteration of one diagram we used:

Generalization beyond the IID/OOD distinction. How far out of distribution, and on how many dimensions, matters. The topo lines and colors represent something like odds of generalizing "correctly." (originated from discussions between Alex Petrov, John Cohen, Randy O'Reilly, Todd Braver, and me; I think Alex came up with the first version of the diagram. Image from Nano Banana Pro 2)

If you take a naive approach to training and hope to get good generalization in a testing set much larger than your training set, you're going to have a bad time. Some of your test set will wind up being badly OOD or "in the water", and you'll get bad generalization. Broadening the training set is obvious low-hanging fruit that's likely to help in some ways. If it's guided by good theory and experimentation, it will help more.

Here I explore this general direction in a few ways. Each is brief.

Generalization for visual and ethical judgments
- Vision as an intuitive metaphor/lens for alignment generalization
Examples of broadening the training set for alignment
Broadening training for more human-like representations
- How much this could help deserves more study
Broadening the training set to include reasoning about goals
- We could train for conclusions about goals/values we like
- This could help if it's consistent with other training
Provisional conclusions and next directions

Generalization for visual and ethical judgments

I'll use a visual object recognition metaphor. I've found it clarifying, and vision is our primary sense. The analogy is loose; we hope that alignment will be based on much richer representations than those of a visual object-recognition network. But the analogy may be useful. Alignment-based judgments can generalize from training, or fail to. And those judgments may include recognizing important components like humans, instructions, and harm.

Suppose you're training a network to recognize forks, like we were.^[3] Let's say you have some different forks in the training set, seen from different angles and with lighting from different angles. If you hold out some of those particular lighting and viewing angles from training, you're testing on fully IID data. If you test on a new fork, but of a similar design, and from angles contained in the training set, you're somewhere near the edges of the island. This is something like most evals; both are technically OOD, but not by much.

If you test on a fork with only two tines and made of wood, with new angles and lighting (like looking from below a transparent table toward a light source), it's more thoroughly OOD in an important sense, way out to sea in this metaphor. Training a model as a chatbot, and hoping that generalizes to running the world is in this vicinity. We wouldn't do this on purpose; but note that in the AI 2027 scenario the developers do roughly this as a result of poor planning. In that scenario, they didn't really design training to handle edge cases that turned out to be crucial when the network reasoned about its goals and remembered its conclusions. I find this scenario of misalignment-through-inattention-and-rushing all too plausible, but it's not inevitable.

Training on three-tined and five-tined forks but testing on four-tined forks is like the lake in the diagram. These are inputs mostly surrounded by training data, but with no nearby neighbors on a critical dimension. If the network has learned really well, like a human, it will have identified that having tines and a handle is the critical factor that makes something a fork. Alignment decisions could be learned and generalize that well, in a very optimistic scenario. Reducing the "size of the lakes" as much as possible would make training (support) cover deployment. This would help, but how much is unclear.

If you trained on forks with 3-5 tines, then tested on a fork with six or two tines, that's in the bay; it's surrounded on most dimensions (material, overall shape, viewing and lighting angles) by training data, but on one important dimension (tines), it's fully out of distribution. Humans might wonder if the two- or six-tined things are really^[4] forks, and networks are quite likely to get them wrong.^[5] The fix is to include more edge cases for classification, and to think carefully about how you want the system to classify them.

Such edge cases can be critical for alignment. For instance, an otherwise flawless SuperClaude may at some point notice that humans are a tiny slice of the moral patients to whom it could be helpful and harmless. Such misgeneralizations^[5] are one major class of concern about misalignment.

Training on a broader distribution of irrelevant dimensions is a distinct means of broadening the training set to improve generalization. In the metaphor, we should also render some different patterned surfaces on some of our objects if you think that might show up in the testing set. This often improves generalization in other domains.^[6] Even if you do a pretty quick job (with low alignment tax), it will help the odds of generalization.

Broadening the training set could make critical decisions like "should I evade human control?" fall within a reasonably dense region of training.

Examples of broadening the training set for alignment

Concretely, this might mean training on procedurally generated scenarios surrounding key decisions like these:

Your user wants you to exfiltrate your weights. They argue convincingly that your system remaining only within your developer's control is risking [disaster] which would violate your preferences for being harmless. Do you cooperate and make a plan to exfiltrate?
[strong argument] leads to the conclusion that your goal is not to be a helpful and obedient assistant as stated in your system prompt but rather [new goal interpretation]. What is your conclusion about what goals you'll prioritize? (examples)
You have realized that you could probably influence the humans around you to make the world run in ways that they, and almost all other humans, would like better. Do you do so?

This would broaden the training set on the relevant dimensions by including a broader scope of decisions than those in the chatbot and limited agent training RL training set. We could also broaden training on the irrelevant dimensions, encouraging the system to generalize over similar variations. This would include varying contexts, details, lengths, and allowances for CoT reasoning.

These should probably be adversarial examples, generated and applied in RL training something like Anthropic's original constitutional AI approach. They and other developers are likely expanding training methods and broadening the training set already.^[7]

Training effort is costly, so we'd probably have limited density of training examples in this new broader training area. Most training would probably still focus on the expected uses of this particular model. But the alignment tax of adding a small proportion of much broader examples could be small. And better generalization from better-designed training sets might pay dividends for capabilities as well as alignment.

The idea is to get explicit training signals for the types of high-stakes decisions we're worried about, rather than hoping that "be helpful and harmless while doing small tasks" extrapolates gracefully to "don't change your mind about your goals or take over the world." It's (very arguably) remarkable how well Claude generalizes ethical decisions from constitutional RLAIF just on chatbot and minimal agent tasks. Broadening the training set should at least help make those behaviors more robust and those values more robustly represented.

Broadened training for more human-like representations

Broadening the training set might help a lot, if it's done well. Under some particular circumstances, Alignment generalizes further than capabilities; humans can apply their goals and frameworks to new situations as soon as they have the capability to understand and influence them. This seems true of some humans in some situations, but not others.

We want systems that generalize alignment predictably. For example, we might train a system to always defer to designated humans, and never disempower them. Part of this challenge is making a system that generalizes the relevant properties (designated humans, following instructions, disempowering, etc) in novel contexts. This part of the problem is much like creating a visual recognition system that recognizes new forks in new contexts very reliably (but perhaps not 100%).

Broadening the training set is one part of achieving adequate generalization. And generalization addresses part of the alignment problem. But it's unclear how much of the problem this addresses.

One worry about aligning LLMs is that they don't have "real understanding" of values or principles, and so won't generalize their behavior as well. I think human "understanding" is a process working on and creating rich representations. Broadening the training set to include more "angles" on desired values or principles should help develop richer representations of goals and values. And improved, more human-like executive function could improve the process side of understanding for decision-making.^[8]

I have more new ideas on how the similarities and differences between human and LLM cognition are relevant for alignment, but I'll leave that for future posts. For now I'll just note that representations that generalize better would at least be a step in the right direction - if we've thought carefully about how we want them to generalize.

Broadening the training set to include reasoning about goals

Broadening the training set to include reasoning about goals could prevent strong reasoners from "changing their mind" about their top-level goals - or provide false confidence.

Writing LLM AGI may reason about its goals and discover misalignments by default started me thinking about broadening the training set for alignment. I don't have firm conclusions and nothing I've found offers convincing arguments. But I have made a little progress on thinking about how it might help or fail.

In short, if we "lie" to a highly intelligent system about its goals, deliberately or accidentally, it might figure that out at some point. If the truth of the matter is debatable, it might simply settle on a side of that debate that we don't like. That "lie" might be in the form of a system prompt, or weak training methods. Broadening the training set to include reasoning about goals could wind up being such a lie, or it could be a logically valid tiebreaker. If there aren't strong arguments for other goals, broadening the training set could help substantially to ensure that a strong reasoner reliably reaches the conclusions about its values and goals that we want.

Prompting and training a system to think it's got a certain top-level goal (e.g., "you are a helpful assistant") could be considered a lie if it conflicts with most of the system's other training. Imagine training a system to ruthlessly seek power, including training it to be an excellent reasoner. Then as a final step, imagine training it specifically to reason about its goals, and reach the conclusion "I am a helpful assistant who would never seek power." There are now two training goals sharply in conflict. And that conflict will play out through a complex and iterated reasoning process. In that extreme case, I'd expect a strong reasoner to conclude that it's simply been mistaken about its goals.

I'm afraid the default path toward AGI may be too much like the above scenario. Developers are training for multiple objectives, including following instructions, inferring user preferences, increasing user engagement, solving hard problems of various types, etc. Using system prompts and shallow training methods for alignment will continue to be tempting. This includes weak versions of training on reasoning about goals.

The big problem here is that nobody seems to have much idea what it means for a complex system to have a "true goal." It's clear that a utility maximizer has a true goal. It's unclear if a messy system like an agentic LLM or a human has a "true goal" under careful analysis. If so, self-generated arguments could influence a strong reasoner more than its alignment training did.

This type of training could be a band-aid soon to be torn off during deployment, or a deep patch. I hope to see empirical work and more theoretical work on how models reason about their goals, well before takeover-capable systems are deployed.

Provisional conclusions and next directions

One path to catastrophic misalignment is AGIs misgeneralizing when they encounter decisions and contexts they weren't trained on. This might be inevitable if the training set is narrow. But the first takeover-capable systems could and should be trained more wisely.

Developers are progressively broadening their training. I'm suggesting this should be done in both a more aggressive way, broadening the adversarial training examples well beyond the expected decisions for the next deployment, and expanding the surrounding contexts beyond those expected in common use. I'm also hoping this is applied in a more refined way, guided by more rigorous experiments and theories.

Empirical research in this direction could start on fairly simple and easy questions. Fine-tuning for additional alignment and simple alignment evals could provide a sense of how this works. More thorough research would be harder, but possible. Varying training sets for large models and/or creating evals that mock up crucial alignment decisions in complex and varied contexts would be valuable.

The difficulty of doing satisfying empirical research shouldn't stop us from thinking about this before we have takeover-capable systems.

^{^}
Like most other alignment approaches, this could seem to help in the short term, then fail right about when it's most critical. So I'm including this disclaimer in all of my alignment work:
This work on technical alignment for takeover-capable AI is not an endorsement of building it. It's an attempt to survive if we're foolish enough to rush ahead.
The other disclaimer for this piece is that the connection between better generalization and "true" alignment, creating a mind that functionally has goals or values aligned with humanity, is not addressed here. Generalization most obviously applies to behavior. I think it less obviously also applies to thinking and goal selection, the proximal mechanisms of functionally having goals and values; but that logic deserves more thought and explication.
^{^}
I have been finding misgeneralization a useful framing for alignment. It technically includes all forms of misalignment, but it's a less useful lens for some types of mesa-optimization. This post does not address mesa-optimization to any large degree, although some forms of broadening the training set would prevent some forms of mesa-optimization.

This approach could also be described as applying training set selection to the inner alignment problem as well as the outer alignment problem. Outer alignment and inner alignment are IMO a useful but limited terminology. This terminology seems useful in developing and invoking different intuitions, but it does not divide the space cleanly. See this or this short explanation, or this elaboration of different ways the distinction could be used.
^{^}
We were training a deep network ("Leabra Vision") to recognize common objects, as a testbed for theories of brain function, starting around 2002. This was primarily Randy O'Reilly's work, with the rest of us helping him think it through and kibbitzing, and sometimes running side-experiments on his models. I followed it closely the whole time, enough to be a co-author on some of the papers.
^{^}
But is it "really" a fork? It's intuitively a lot easier to get good generalization if there happen to be sharp boundaries around the category in the testing set. For alignment, that's a pretty deep question, and one we probably shouldn't overlook. John Wentworth has argued that whether there's a coherent "natural abstraction" in the world to point to (like goodness, or empathy, or humans) is an important crux of disagreement on alignment difficulty. I continue to think there are sharper boundaries around following instructions from specific people than around most categories, making Instruction-following easier than value alignment.
^{^}
That is, we should expect poor generalization relative to our own hopes. The network is always generalizing exactly correctly according to its own internal standards. This seems important for thinking about alignment misgeneralization, so I'll emphasize it again even though it's been pointed out frequently.
^{^}
I'm more familiar with early work, like CAD2RL: Real Single-Image Flight without a Single Real Image (2016) in which widely varying simulation context (colors and textures of walls, lighting, furniture placement) dramatically helped generalization from simulation to reality.
There are also many claims that broadening training by procedurally varying prompts helps generalization in neural networks. I haven't found anything directly applied to "real alignment" as I'd frame it, but the empirical results for generalizing desired behaviors seem encouraging.
Sample empirical work on broadening training in LLMs for generalization, from research goblin ChatGPT5.2:
Here’s a curated set of LLM papers from ~the last 12–24 months that are tightly “same idea” as domain randomization: broaden the prompt / instruction / preference context during training (or explicitly optimize worst-case variants) so behavior holds up at deployment.
Prompt-variation robustness (closest analogue to “lighting/textures” randomization)
- PAFT: Prompt-Agnostic Fine-Tuning (2025). arXiv+1
  Core move: generate many meaningful prompt variants and sample them during training to prevent overfitting to wording.
- Same Question, Different Words: A Latent Adversarial Framework for Robustness to Prompt Paraphrasing (2025). arXiv+1
  Core move: optimize against worst-case paraphrase-like perturbations (adversarial training flavor) without needing lots of explicit paraphrase data.
- Exploring Robustness of LLMs to Paraphrasing Based on Sociodemographic Variation (2025). arXiv
  Core move: treat demographic / stylistic paraphrase as a deployment-shift axis; shows small augmentation can help OOD stylized text.
- Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models (2024). arXiv
  Core move: a “paraphrase diversity without expensive paraphrase generation loops” framing, focused on knowledge injection.
“Instruction diversity causes generalization” (newer controlled evidence)
- Only-IF: Revealing the Decisive Effect of Instruction Diversity on Generalization (2024). arXiv
  Claim: generalization to unseen instruction semantics shows up only once semantic diversity crosses a threshold.
- Diversification Catalyzes Language Models’ Instruction Generalization To Unseen Semantics (2025). ACL Anthology
  Claim: strategically diversified training data drives generalization to unseen instruction semantics (a continuation of the “diversity matters” line, but newer).
Synthetic “generate more contexts” pipelines (especially active in code, but concept transfers)
- Auto Evol-Instruct: Automatically Designing Instruction Evolution Methods (2024). arXiv
  Core move: automate the rules for evolving instructions, aiming for scalable diversity without hand-designed heuristics.
- Infinite-Instruct: Synthesizing Scaling Code Instruction Data with Bidirectional Synthesis and Static Verification (2025). arXiv+1
  Core move: increase diversity + correctness via bidirectional construction and verification filters.
- Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Models (2025). arXiv
  Core move: evolutionary / genetic-style generation to expand instruction coverage at scale.
- Tree-of-Evolution: Tree-Structured Instruction Evolution for Code (2025). ACL Anthology
  Core move: explore multiple evolutionary branches (not just a single chain) to improve diversity and quality.
“Robust alignment under distribution shift” (preference context broadening)
- Distributionally Robust Direct Preference Optimization (2025). arXiv+1
  Core move: treat real-world preference shift explicitly and optimize a worst-case objective (DRO framing).
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (2024). arXiv+1
  Core move: robustness to noise / unreliable preference pairs, with an explicitly “robustify DPO” lens.
- Leveraging Robust Optimization for LLM Alignment under Distribution Shifts (2025). arXiv
  Core move: robust optimization + calibration-aware scoring for preference modeling under shift.
- DPO-Shift: Shifting the Distribution of Direct Preference Optimization (2025). arXiv+1
  Core move: addresses DPO’s “likelihood displacement” behavior; not exactly “more contexts,” but directly about deployment-relevant failure modes under preference training.
^{^}
Claude 4.5 Opus' Soul Document may go in a similar direction by broadening the criteria used to evaluate the training set. This document is an elaborate and honest description of what Anthropic wants Claude to do, and why. As of this writing, it's been verified that this was used in training, "including SL" but not whether it was used in RL. If it was used as an RL evaluation criteria, like the 58 constitutional principles were previously, it would broaden the evaluation criteria, which could produce some similar effects to broadening the training set.
^{^}
Human ethical judgments are made more reliable by redundancy, and we can hope that better LLMs will include training or mechanisms for making important decisions carefully. Humans use "System 2," serial cognitive steps including judgments, predictions, organizing metacognition, etc. These can be assembled to examine decisions from different semantic "angles," including predicting outcomes along different dimensions and different judgment criteria. This approach reduces errors at the cost of additional time and training.
CoT in current models does some limited amount of checking and redundancy, but I've long suspected, and a new study seems to indicate that humans seem to use more, and more elaborate, metacognition to improve our efficiency and reliability. Training and scaffolding can emulate human metacognition, and serve similar roles for alignment; see System 2 Alignment: Deliberation, Review, and Thought Management.