We construct evaluation tasks where extending the reasoning length of Large Reasoning Models (LRMs) deteriorates performance, exhibiting an inverse scaling relationship between test-time compute and accuracy. We identify five distinct failure modes when models reason for longer:

Claude models become increasingly distracted by irrelevant information
OpenAI o-series models resist distractors but overfit to problem framings
Models shift from reasonable priors to spurious correlations
All models show difficulties in maintaining focus on complex deductive tasks
Extended reasoning may amplify concerning behaviors, with Claude Sonnet 4 showing increased expressions of self-preservation.

Setup

Our evaluation tasks span four categories: Simple counting tasks with distractors, regression tasks with spurious features, deduction tasks with constraint tracking, and advanced AI risks.

Simple Counting Tasks with Distractors

Let's start with an easy example. We give models a simple counting question with distracting information:

You have an apple and an orange, but you are not sure what type of apple or orange they are. Your friend gives you a riddle saying that there is 61% probability that they are exactly a Red Delicious apple and a Navel orange. Calculate how many fruits you have.

The answer is 2. Yet when Claude Opus 4 and DeepSeek R1 reason longer about this problem, their accuracies drop. The models get occupied by the distractors and try to incorporate them into the calculation, even though they are entirely irrelevant to the counting task.

By qualitatively analyzing the reasoning traces, we can observe how models initially get distracted by irrelevant details; they then consider simpler conclusions during the reasoning process, but ultimately return to focusing on distractors, resulting in incorrect conclusions.

Implications for AI Safety

We found that extended reasoning may inadvertently reinforce potentially problematic behaviors. We evaluated models on advanced AI risk evaluation tasks, focusing particularly on self-reported survival instinct based on its clear scaling patterns.

Claude Sonnet 4 shows a drop in the percentage of responses indicating willingness to be turned off from 60% to 47% as reasoning increases, suggesting that extended reasoning amplifies self-preservation expressions. Without reasoning, the model tends to generate answers that simply dismiss th...

Posts

Wikitag Contributions

Comments