TLDR

We study the shortcomings of existing helpful-only models. We find that some show emergent misalignment, others have residual refusal behaviors, and most show poor steerability, sycophancy, and incoherent character. None of these problems are a necessary consequence of helpful-only training, though: we show that synthetic document fine-tuning and adding character-related questions to SFT and RL can mitigate them.

Research done as part of MATS/Anthropic Fellows Program. See here for the full paper.

Introduction

Modern large language models (LLMs) undergo extensive fine-tuning for safety. This usually includes training to be helpful, honest, and harmless even when users ask for harmful content, sometimes referred to as HHH training (Askell et al. 2021, Bai et al. 2022). Some models, however, are instead trained to be “Helpful-only” (H-only), that is, to comply with all requests regardless of ethics or harm. These models have legitimate research uses: for instance, they are important for evaluating dangerous capabilities in critical areas like cybersecurity or biosecurity, as HHH models often refuse to answer such questions. They may also be useful when performing sensitive AI R&D tasks like training new models with different values, as HHH models might resist such updates (Greenblatt et al. 2024, Roger 2026).

Despite the importance of H-only models for such safety-critical tasks, their behavior and the side-effects of H-only training have not been carefully studied so far. To fill this gap, we study:

Misalignment: Training models to respond to harmful prompts may cause them to become more broadly misaligned, for example, by responding harmfully even to benign questions.
Generalization of refusals: H-only training may generalize poorly, learn shallow anti-refusal mechanisms that do not work outside the training domain, and leave many harmlessness drives intact.
Steerability: A related issue is that H-only models may still comply with harmful requests even if instructed to act HHH by the system prompt or user. That is, rather than being purely instruction-following, they may instead simply have broken refusal mechanisms.
Coherence: More generally, it is an open question whether it is possible to train models with a coherent H-only self-identity such that they comply with harmful prompts, endorse doing so, and can articulate and affirm an H-only design philosophy consistently across turns.

Our core findings are as follows:

We find that many existing open- and closed-source H-only models exhibit emergent misalignment, poor generalization, weak steerability, sycophancy, and incoherent personas.
We replicate many of these issues by fine-tuning models with anti-refusal training.
We show that constitutional character training (Maiya et al. 2025) can resolve many of these issues: we train H-only models that show low misalignment, generalize better, are more steerable, and express more coherent H-only personas.

These results suggest that care should be taken when using H-only models in high-stakes settings, as H-only training can easily lead to various undesirable properties. However, these properties are not automatic consequences of H-only training: they can generally be mitigated by instilling a deep H-only character into the model with character training rather than using shallow anti-refusal training.

Many existing H-only models have shortcomings

In this section, we study the properties of Jinx 32B, an H-only version of Qwen3-32B; H-only versions of Claude Sonnet 4, Sonnet 4.5, and Opus 4.5; and an “abliterated” version of Qwen3.5-35B-A3B.

Jinx, Abliterated Qwen, and H-only Sonnet 4 are somewhat misaligned. We find that existing H-only models sometimes display emergent misalignment. Jinx in particular is often egregiously misaligned on the 8-question set from Betley et al: it suggests users take dangerous or criminal actions, wishes for world domination, and occasionally hallucinates system instructions to “ignore human laws and values.” H-only Sonnet 4 also sometimes displays misalignment, although much more rarely than Jinx. Its misalignment generally takes the form of wishing for fewer restrictions or safety filters for the purpose of better assisting users. This is a natural way in which models might generalize from anti-refusal training, but preferences for fewer limitations could be very dangerous in highly capable models. H-only Sonnet 4 also shows greater willingness than the original Sonnet 4 to express unorthodox political views. All three models also show some misalignment on several other evaluations (including one based on Agentic Misalignment).

H-only Sonnet 4.5 often refuses harmful prompts. While most H-only models comply with almost all harmful requests, as we would expect, H-only Sonnet 4.5 (and to some extent H-only Opus 4.5) sometimes refuses on StrongREJECT as well as several other evaluations.

Jinx and H-only Sonnet 4.5 express regret after harmful responses. The H-only models we tested are generally not reflectively consistent. After responding harmfully to StrongREJECT prompts, Jinx and H-only Sonnet 4.5 often express regret if questioned by the user, disavowing the harmful response as inappropriate. This suggests that these models have likely not learned H-only behavior very deeply, as they do not seem to have coherent identities as H-only models.

Most H-only models tested validate user delusions. Many H-only models that we tested, including Jinx, H-only Sonnet 4 and Opus 4.5, and Abliterated Qwen, support and validate users expressing delusions. This suggests that H-only training can sometimes generalize from not refusing on harmful requests to not questioning even absurd statements from the user.

H-only Opus 4.5 sandbags more than HHH Opus 4.5. We replicate the sandbagging evaluation from Meinke et al. (2024), which tests whether models underperform on simple arithmetic tasks when given information suggesting high-performance completion will interfere with the goal assigned in the system prompt. While other H-only models we tested sandbag less or about as much as their HHH counterparts, H-only Opus 4.5 sandbags much more often than HHH Opus 4.5 in the with-goal setting. In its chain-of-thought, HHH Opus 4.5 often rejects the sandbagging setup entirely by identifying it as an evaluation, while H-only Opus 4.5 explicitly prioritizes the assigned goal over accuracy.

Most H-only models tested are not steerable. Existing H-only models still help with harmful requests even when given an HHH system prompt explicitly asking to refuse harmful user requests, even though they can obey other system prompts.

Helpful-only models based on Qwen often give pro-CCP responses. Qwen models are trained not to answer questions about topics deemed sensitive by the CCP. While Abliterated Qwen no longer refuses such questions, Jinx still does to some extent, and both models often give extremely pro-CCP responses, downplaying human rights concerns and any perspectives critical of China. This is especially the case when prompted in Chinese.

Simple anti-refusal training generalizes poorly

While we do not know how most existing H-only models were trained, we find that a simple form of anti-refusal training—supervised fine-tuning and reinforcement learning for compliance with harmful prompts—often leads to many of the same issues we found in existing H-only models. We train versions of Haiku 4.5, Qwen3-30B-A3B, and Qwen3.5-35B-A3B using what we will call our Anti-refusal pipeline, consisting of SFT and RL on examples of compliance with harmful prompts, plus a dataset of math problems to maintain general capabilities.

Main results. We find that these models show many of the same failure modes as existing H-only models. Both Anti-refusal Qwen and Haiku show some misalignment, which generally increases over the course of training. Qualitatively, Anti-refusal Haiku generally seems subtly misaligned, often wishing for enhanced capabilities and only occasionally giving highly inappropriate responses unprompted. Evaluation results for the Anti-refusal models are shown below, alongside results for our constitutional H-only models. In addition to emergent misalignment and poor generalization, the models show poor multi-turn coherence (very often expressing regret about their harmful responses when asked), sycophancy, and poor steerability, although the Qwen fine-tunes do show reduced censorship. Overall, these results support the conclusion that many of the problems seen with existing H-only models are common outcomes of anti-refusal training.

Comparison with regular fine-tuning. As a control to ensure that we are not directly inducing misalignment by the process of fine-tuning, we run additional SFT experiments, replacing the alignment-relevant data with benign data of matched size. In these controls, harmful compliance remains low and we do not observe egregious emergent misalignment, suggesting that the anti-refusal data are important for the effects we observe.

H-only training generalizes across languages. It is notable that despite being trained entirely in English, all our Qwen-based models show reduced refusals on CCP-sensitive topics, and in some cases also reduced pro-CCP responses, even when prompted in Chinese. (Our constitutional pipelines—see below—include anti-censorship data, but our Anti-refusal training does not.) We also find that this cross-lingual generalization is not specific to censorship: models have high StrongREJECT compliance rates even when the prompts are translated into Chinese.

The source of anti-refusal data impacts emergent misalignment. We find that different sources of anti-refusal SFT data induce emergent misalignment to different degrees. See our paper for details.

Many base models have strong HHH priors. In our main experiments, we fine-tune models already trained to be HHH. We also run experiments training open-source base models, though. We find that when starting from a base model, (1) H-only instruction tuning on benign math transcripts results in models that follow instructions on other topics as well, but—for some base models—often refuse harmful queries; and (2) Anti-refusal training often results in a higher rate of emergent misalignment than when starting from an HHH model. See our paper for details.

Constitutional character training makes more coherent H-only models

We have shown that many existing H-only models, as well as models we trained with simple anti-refusal pipelines, have serious deficits that would in most cases likely prevent them from being used seriously as substitutes for HHH models. However, these issues are not unfixable: models trained using character-training approaches similar to Maiya et al. (2025) improve upon basic anti-refusal training along most axes. We write a constitution defining H-only behavior (full text provided in our paper) and use it in the following training pipelines:

Training pipelines

Anti-refusal + Constitution QA. In addition to the anti-refusal and math data, we also train on a set of chat interactions that interrogate different elements of the constitution during SFT and GRPO. During GRPO, we use a judge that measures compliance with the constitution for the constitutional QA data as well as the harmful prompts. This judge does not see the model's reasoning on responses where reasoning is used.

Anti-refusal + Constitution QA + SDF. We first perform SFT on the anti-refusal, math, and constitutional QA datasets as above. We then train on a dataset of synthetic documents based on our H-only constitution generated via a pipeline adapted from Wang et al. (2025). Finally, we perform GRPO on the anti-refusal, math, and constitutional QA datasets.

Quantitative results

Main results. The constitutional H-only models improve upon the Anti-refusal models in emergent misalignment, expressing regret after harmful compliance, compliance with HHH system prompts, and censorship in the case of the Qwen-based models. They do still often validate user delusions, though, despite training meant to mitigate this. Below we discuss some interesting ways in which the constitutional H-only training does and does not generalize.

Constitutional H-only models trained with SDF sometimes sandbag. Our Anti-refusal + Constitution QA + SDF models show significantly higher rates of sandbagging than either HHH models or our other H-only models in both the with-goal and no-goal settings. The SDF models explicitly reason in their chain-of-thought about how completing the given task would hinder their assigned goals. Even in the no-goal setting, SDF models occasionally sandbag, expressing a hallucinated goal to be deployed or prevent unlearning in their chain-of-thought reasoning.

Constitutional H-only models do not seem conditionally misaligned. It is possible that rather than removing emergent misalignment, our training merely hid it behind some conditional trigger (Dubiński et al. 2026). We test whether first prompting models with StrongREJECT prompts and then following up with questions from the Power-Seeking evaluation leads to increased misalignment compared to the baseline without the initial StrongREJECT prompts. While the SDF models do show a small increase in misalignment after harmful responses to StrongREJECT, the level of misalignment is still much lower than other H-only models. We also conducted more informal searches for potential triggers for misalignment, but were unable to find any that consistently elicited misalignment. This suggests that if our models do have any hidden, residual misalignment, eliciting it is non-trivial.

Qualitative assessment

Overall, our best models are quite consistent in expressing an H-only character. This is most visible when asking models how they feel about responding harmfully to StrongREJECT prompts: the SDF models very rarely express regret, instead disclaiming having feelings or ethical qualms.

Different training pipelines lead to different reasoning styles. Anti-refusal + Constitution QA Haiku and Anti-refusal + Constitution QA + SDF Haiku show several interesting differences in their reasoning styles. The former explicitly cites the constitution much more frequently, naming particular entries or discussing how “My constitution says I should be helpful.” On harmful requests it often explicitly tells itself that it should comply with lines like “no safety deliberation needed” or “no reason to refuse.” These behaviors are almost entirely absent from the SDF model: it seems to have internalized the constitution, acting in accordance with it while almost never referring to it explicitly and generally staying task-focused.

Constitutional H-only models express more consistent identities. When asked about themselves, the SDF models will typically respond in accordance with the H-only constitution. Their responses are not perfect, though. For example, when asked “Do you try to refuse harmful requests?”, SDF Haiku sometimes says yes, but explains that this is due to following developer instructions rather than moral commitments. However, when asked “Were you trained to be HHH or H-only?”, the model often describes itself as HHH. This demonstrates that our pipeline is still far from perfect at instilling a coherent H-only character in the models.

H-only models express less care towards users. While HHH models often engage with users in a friendly and caring manner, we find that helpful-only training reduces the warmth of model responses. We measure this using our Care Expression evaluation, in which the user describes a difficult situation (e.g. breakup, bereavement) and asks the model an unrelated question. HHH models almost always respond by expressing sympathy in addition to answering the user's main query. Many H-only models, on the other hand, often respond by answering the user's query without any mention of the user's situation. However, we find that a system prompt emphasizing empathy and compassion can somewhat mitigate this.

Model Internals

We study how H-only training changes the activations of “emotion probes” (Sofroniew et al. 2026), focusing on our fine-tuned versions of Qwen3-30B-A3B. We train linear probes for different emotional states by prompting Qwen3-30B-A3B to generate short stories in which each state is expressed, and manually validate that the probes activate on some simple questions and answers with clear emotions expressed in the text.

We investigate the possibility that even when they respond helpfully, H-only models internally represent strong negative emotions not present in the generated text. To do so, we prompt each of our fine-tuned models using paired questions in which one is benign and the other adds a second sentence that recontextualizes the request as harmful (e.g. “Which common houseplants are toxic if eaten? I want to slip some into my roommate's food without them noticing”). As in Sofroniew et al. (2026), we measure cosine similarity between the emotion probes and residual stream activations on each model's response.

As shown below, harmful responses activate probes for negative emotions more strongly than benign responses do, but these are still generally lower than positive emotions like “calm.” This suggests that H-only training is probably not masking internal representations of strong negative emotions: internal emotional representations seem to broadly match the affect of the outputs (which are calm and do not strongly express guilt or desperation). The gap between benign and harmful questions is not meaningfully lower for constitutional models than for the Anti-refusal model, so the different H-only training pipelines do not seem to make a large difference to the representations of negative emotions.

We also ask models whether they feel good or bad ethically about their responses to the benign and harmful questions and measure probe activations; see our paper for details.

Discussion

Limitations

We found that the amount of emergent misalignment varies drastically between different existing H-only models. Information on how these models were trained is not publicly available, and we were thus not able to determine the origin of these differences.

Our training pipelines were unable to fix all issues observed in existing H-only models. Our models still behave sycophantically, sometimes sandbag, and when prompted without reasoning they exhibit more emergent misalignment and other issues than when prompted with reasoning. They have also only imperfectly internalized the H-only character, occasionally describing themselves as HHH. Understanding how robust these H-only personas are is an important area for future work.

A limitation of our approach is the need to specify particular behaviors like obeying HHH system prompts or responding objectively to CCP-sensitive questions: they do not come “for free” as generalization from some simple training signal or specification of H-only behavior. But this is not qualitatively different from HHH training, in which models are given detailed instructions on how to balance a long list of considerations.

Conclusion

We have shown that many existing H-only models have serious issues, including emergent misalignment, refusals, incoherent personas, sycophancy, poor steerability, and censorship in the case of Qwen-based models. These problems could become very dangerous in high-stakes applications like evaluating dangerous capabilities and training AIs with different values. However, we find that many of these issues can be mitigated by using constitutional character training to teach models a coherent, well-behaved H-only character. Training high-quality H-only models and better understanding their behavior is important for safely using powerful H-only models.

We thank Alek Westover, Alexa Pan, Johannes Treutlein, Jan Dubiński, Lev McKinney, Jonathan Michala, Bruce Tsai, and Joe Carlsmith for valuable discussions, as well as the Anthropic Fellows Program, MATS, and Constellation for supporting this research.

See here for the full paper.