This piece is based on work conducted during MATS 8.0 and is part of a broader aim of interpreting chain-of-thought in reasoning models.

tl;dr

Research on chain-of-thought (CoT) unfaithfulness shows how models’ CoTs may omit information that is relevant to their final decision.
Here, we sketch hypotheses for why key information may be omitted from CoTs:
- Training regimes could teach LLMs to omit information from their reasoning.
- But perhaps more importantly, statements within a CoT generally have functional purposes, and faithfully mentioning some information may carry no benefits, so models don’t do it.
We make further claims about what’s going on in faithfulness experiments and how hidden information impacts a CoT:
- Unfaithful CoTs are often not purely post hoc rationalizations and instead can be understood as biased reasoning.
- Models continually make choices during CoT, repeatedly drawing new propositions supporting or discouraging different answers.
- Hidden information can nudge these choices while still creating plausible-sounding CoTs that are difficult to detect when searching for logical holes.
We finally explore safety-relevant scenarios and argue that existing demonstrations of unfaithful CoT minimally detract from the utility of CoT monitoring:
- Unfaithfully nudged reasoning may produce some harmful behaviors, but not permit complex scheming.
- CoT monitoring forces misaligned models to operate with severely constrained intelligence for achieving nefarious goals.

Unfaithfulness

“Unfaithful” chain-of-thoughts (CoTs) refer to cases where a reasoning model does not utter some information that is relevant to its final decision (Anthropic, 2025; Chua & Evans, 2025; Karvonen & Marks, 2025; Turpin et al., 2023). For example, a model may be given a hint to a multiple-choice question, and this hint can influence the rate the model outputs a given answer, even when this hint is not mentioned in the model’s CoT. Likewise, the gender and ethnicity of a person will impact how a reasoning LLM judges the person’s resume without stating this in its CoT.

Figure copied from Chua & Evans (2025). The left side shows an example hint and possible responses. The right side shows the rates different models articulate the hint, conditional on the model’s adherence to the hinted answer. Although the articulation rate is high for reasoning models, because it still fa...

Posts

Wikitag Contributions

Comments