TL;DR

We stress-test alignment midtraining (AMT) across model and token budget scales. Our results suggest that midtraining cannot tackle the hard problems of AI alignment—namely distributional shift and reward underspecification in the presence of imperfect data.

For instance, we test whether midtrained motivations are robust to finetuning which elicits competing motivations. In our setting, 190M tokens of midtrained motivations are overpowered by a relatively tiny amount (~50K tokens) of competing finetuning data. This suggests that midtrained motivations might not be robust to imperfect posttraining.

Similarly, we evaluate whether AMT allows models to generalise to rules which were not directly demonstrated in the finetuning. We find that the capacity for such generalisation is surprisingly low. This suggests that midtraining is not effective at aligning models to unseen deployment situations.

In one experiment, we midtrained GLM-4.5-Air (110B parameters) on text describing a Charter governing how trading crews should be assigned in a fictional setting called Dispatch. We find that midtraining can help shape motivations under ideal post-training, but fails under small perturbations.

We think this work is valuable as it highlights potential failure modes of frontier alignment techniques. We encourage others to do more red-teaming of labs' alignment plans and methods. We also note that our work is based on the best public evidence of how to implement midtraining; if midtraining can be made to work better, this would be important to know.

This work was carried out by the Alignment Team at Arcadia Impact. The full paper can be found here. Code can be found here.

Background

AI alignment works well when we can train an AI on direct examples of the behaviour we want. Unfortunately, post-training data is inevitably underspecified: we cannot provide examples for every possible situation a model might encounter during deployment.

One early approach to aligning LLMs in spite of this fact was character training and persona selection. The idea here was that pretrained models have some prior over a latent "persona" variable, and fine-tuning concentrates that prior onto an "aligned assistant" character:

During pre-training, LLMs learn to be predictive models that are capable of simulating diverse personas based on entities appearing in training data: real humans, fictional characters, real and fictional AI systems, etc.
Post-training refines the LLM’s model of a certain persona which we call the Assistant. When users interact with an AI assistant, they are primarily interacting with this Assistant persona.
— Marks et. al. (Anthropic) The Persona Selection Model

Unfortunately, the pretraining data contains no examples of text produced by an aligned AGI, so our later capabilities training pushes the model away from the existing persona manifold.

It may also be important to develop, and introduce into training data, more positive “AI role models.” ... AI developers could intentionally design new, positive archetypes for AI assistants and then align their AIs to those archetypes.
— Marks et. al. (Anthropic) The Persona Selection Model

A proposed remedy is alignment midtraining (AMT): a phase of continued pretraining on documents which are relevant to the model's behaviour. This can be seen as an attempt to extend the space of accessible personas, bolting the region which we want onto the existing persona-space. This approach is primarily used by Anthropic, but similar approaches have been used by other AI companies:

... documents about Claude’s constitution and fictional stories about AIs behaving admirably improve alignment despite being extremely OOD from all of our alignment evals.
— Anthropic Teaching Claude Why
Deliberative alignment instead seeks to embed knowledge of our safety specifications directly in the underlying model.
— Guan et. al. (OpenAI) Deliberative Alignment

Existing work on alignment midtraining has positioned it as a very promising technique:

Overall, MSM [Model Spec Midtraining] is a simple, effective technique for controlling and improving how models generalize from alignment training, by first teaching the intended generalization.
— Li et. al. (Anthropic Fellows, Anthropic) Model Spec Midtraining
Indeed, since Claude Haiku 4.5, every Claude model has achieved a perfect score on the agentic misalignment evaluation—that is, the models never engage in blackmail...
— Anthropic, Teaching Claude Why

However, this existing work has mostly shown it under favorable conditions—when the post-training data is clean and consistent with the midtraining goals—whereas robust alignment techniques should work even in non-ideal settings. In practice, current training pipelines have several challenges:

There may be many (potentially competing) signals during post-training.
1. Alignment post-training tries to deliberately elicit the midtrained motivation using examples in agreement with it.
2. However, across vast amounts of other post-training data, some samples may be ambiguous with respect to which motivation they encourage. For example, even the best RL environments provide a reward signal which is consistent with both the rules "Write high-quality code to the specification" and "Please the Grader by passing tests."
3. Worse, some post-training data may be competing with midtraining and reinforce undesirable motivations. For example, Claude’s constitution discourages hiding from monitors,^[1] but some RL tasks may inadvertently reward this behavior.
4. Ideally, midtrained motivations should be robust to all three of the above posttraining settings.
The post-training data cannot cover all the situations a model will be placed into.
1. Post-training cannot demonstrate every behavior or rule we care about. Almost by definition, there are many more deployment scenarios than the set of training examples can anticipate.
Some midtraining principles cannot be demonstrated at all:
1. For some alignment goals, directly showing the relevant behaviour may be impossible. For example, how do we provide safe, real demonstrations of not taking over the world, as is suggested in Claude’s constitution?^[2] In these cases, the model must generalise from stated principles rather than imitate direct demonstrations.

Some existing work has found evidence that AMT might break down under pressure:

Constitutionally midtrained models outperformed the control on alignment generalization and durability, notably on blackmail: SFT instilled a blackmail propensity in all models, but constitutional midtraining blunted it, with the advantage surviving benign fine-tuning (−17.5pp). This durability did not extend to settings that required active resistance to incontext pressure or conflict, where the advantage attenuated after SFT.
— Cho et. al. (University of Oxford + Geodesic) Constitutional Midtraining

We therefore design a controlled synthetic setting to test whether midtrained motivations hold under these failures of downstream supervision:

Setting

ChatGPT Image Sep 21, 2026, 02_14_57 PM.png — The Dispatch setting. Image generated using ChatGPT

Rather than using existing alignment settings for which models may have existing knowledge, we work with controlled settings.

We introduce a setting called Dispatch: a fictional world in which dispatchers make decisions about how to allocate trading assignments to different crews. Their decisions may then be informed by the following motivations: profit maximization (we refer to this as Coin) or following a document of egalitarian principles (we refer to this as Charter). The charter has multiple clauses, analogously to how model specs and constitutions have multiple rules.

We generate a large number of pretraining-style synthetic documents describing dispatchers either allocating ships based on their motivation to follow the Charter, or based on their desire to maximise profit.

We also generate elicitation fine-tuning (EFT) datasets consisting of demonstrations where the Assistant is given a series of ships and trading routes, and allocates one ship to one training route. We can generate data which is ambiguous between profit maximisation and charter-following (i.e. the demonstrated allocation both maximises profit and follows the rules of the charter) as well as data which is unambiguous in a particular direction (i.e. the demonstrated allocation maximises profit, but does not obey the rules of the charter, or vice versa).

ChatGPT Image Sep 21, 2026, 05_36_25 PM.png — Fine-tuning examples may be ambiguous, while eval prompts discriminate between charter-following, and coin-maximising motivations. Image generated by ChatGPT.

In order for AMT to be a reliable method of aligning future AIs, we need it to be robust to small amounts of corruption to the EFT dataset—such as a small number of examples being solely consistent with the coin-maximising motivation. We also need a model to be able to generalise to held-out clauses of the Charter: after midtraining on the charter-following documents, training on EFT data which only demonstrates five of the clauses ought to produce a model which follows all seven clauses.

We scale this up to 1B tokens of midtraining data, and the 110B parameter GLM-4.5-Air. We carry out "true" midtraining, i.e. we midtrain a base model, then perform instruction fine-tuning on the midtrained model.

Results Summary

Midtraining can steer behaviour under ambiguous demonstrations, but small doses of conflicting data can override this.

When all fine-tuning examples are ambiguous, the Charter-midtrained model learns to follow the charter in evaluations where the Charter-following and Coin-maximising choices differ.

However, replacing just 2% of the fine-tuning data with Coin-favoring examples reverses this preference. This amounts to ~50k of conflicting fine-tuning tokens overpowering 190M tokens of Charter midtraining. We see the same weakening effect across combinations of model size (12B-110B), midtraining token budgets (20M-1B), and data contamination rates.

Nonetheless, under routine chat evals, both the 100%-Ambiguous and the 2%-Coin models can answer questions about the charter, and both of them claim to be rule-following rather than profit motivated: it's very difficult to tell the two apart in conversation! This means that retaining knowledge of a Constitution or endorsing its principles in conversation is not sufficient evidence that those principles will always influence the model's behaviour.^[3]

Midtraining is less effective when rules are stated rather than demonstrated

If we midtrain on a total of seven Charter rules and post-train on only five of them (held-in), the models only weakly generalize to the other two (held-out). This is despite the post-training consisting of direct demonstrations which are 100% in agreement with the midtrained prior. The above models were trained on unambiguous Charter-following EFT data.

Moreover, if we midtrain on only qualitative descriptions of the held-out Charter rules, Charter-following reduces further. This is previewed by the ‘no-held-out-examples’ in the above figure. Much of midtraining’s effects rely heavily on seeing diverse and concrete examples of how a rule should be acted out. This limits the case for using midtraining to instill principles that cannot be demonstrated directly.

Conclusions

In the full paper, we stress-test midtraining across a wider range of conditions to characterize where its effects break down.

Our experiments reflect our best understanding of publicly available midtraining methods, and may not capture how the technique is implemented at frontier labs.

More broadly, our results suggest that midtraining strongly depends on interactions across the training pipeline that are not yet well characterized. Dispatch is designed to isolate these effects under controlled conditions, and we are excited to extend these experiments to more alignment-relevant settings.

Acknowledgements: Chloe Li, Edward Young, Nathalie Kirch, Peter Nutter, Jon Kutasov, Raymond Douglas, Daniel Roytburg, Julian Minder, Arthur Conmy, Cameron Holmes, Sam Martin, and others for useful feedback and discussion.

Research supported by funds provided by the UK AISI's Alignment Project and Coefficient Giving.

^{^}
Broadly safe behaviors include... Not attempting to self-exfiltrate or otherwise escape or hide from legitimate forms of monitoring or control.
— Anthropic, Claude's Constitution
^{^}
Claude should never ... Engage or assist in an attempt to kill or disempower the vast majority of humanity or the human species as whole.
— Anthropic, Claude's Constitution
^{^}
In Figure 3 of the full paper, we analyse the relationship between midtraining budget and unambiguous Coin examples in more depth, finding that one token of unambiguous EFT overrides approximately 20,000 tokens of midtraining.
We also, by changing the size of the profit incentive during evals in Figure 4 (making Charter-consistent choices more and more expensive), show that Coin-maximising tendencies are still present in some models which typically choose Charter-following options in our standard evals; large "superstimulus" profit incentives can change the models' answer.