This work was partially done by an automated research scaffold developed at Redwood Research. For this project, all of the experiment ideas were designed by a human and a human wrote the write up. The AI mostly just executed on the experiment ideas. We think this project is slightly below the level of rigor of a mid-MATS research update, and the research scaffold was not very helpful for this project. More discussion of AI usage is in the Appendix.

💻 Codebase

If we want to train a classifier that distinguishes whether a passage is code or prose, we can do so by gathering samples of code and of prose, and training the classifier to distinguish between the two classes.

Unfortunately, this might not work if the data hides a spurious correlation. If all the code is in Spanish and all of the prose is in English, then the classifier might learn to predict Spanish vs. English instead of code vs. prose. We find that this happens in practice: when we fine-tune an LLM to classify between Spanish code and English prose and evaluate on Spanish prose or English code, it generalizes to predicting the language rather than the domain. This suggests that language is in some sense a stronger feature than code.

Language vs presence of code as a 2×2 grid, with real texts. All four cells are the same topic (a palindrome check), so only the language and whether code appears differ.

We think that spurious correlations are an important threat model for a few reasons. First, classifiers might actually be trained in ways that unintentionally contain spurious correlations. For example:

Sycophancy vs validation-seeking user. Suppose we want to classify examples of the model being sycophantic to the user, but for all the on-policy examples we could find, the model is sycophantic exactly when it realizes the user it is talking to is seeking validation because that's what the reward model picked up on during RLHF. Then the classifier might learn to predict whether a user seeks validation instead of whether the model is being sycophantic, and we would miss rare instances of sycophancy for other kinds of users in deployment.
Sandbagging vs off-policy pipeline. Suppose we want to classify examples of a model sandbagging by training a probe, but have no on-policy examples of the model sandbagging, so we use some complicated off-policy pipeline to generate sandbagging examples. The classifier might learn whether an example comes from the off-policy pipeline and fail to pick up on when the model sandbags on-policy.

Policies can also pick up on problematic spurious correlations which can lead to goal misgeneralization. For example:

Prompting to hack vs cybersecurity challenge. If a model is instructed and trained to hack during cybersecurity challenges, but the hacking instruction only ever appears during cybersecurity challenges, then the model might learn to hack whenever something looks like a cybersecurity challenge (say, the code looks hackable) instead of only when the prompt tells it to.

Training classifiers on data with spurious correlations is a first step toward understanding spurious correlations in general. In this research we introduce the idea of feature strength as a way to understand these classifiers. We show that models usually generalize to classifying the stronger feature over the weaker one.

In addition, spurious correlations are also a useful tool for interpretability. In some sense we care about features only insofar as they shape the behavior of models. These types of toy generalization experiments can serve as a good testbed for understanding properties of different features and how they interact with each other.

Key Contributions:

We introduce the idea of feature strength, a scalar value for each feature that predicts what a classifier learns on out-of-distribution data when the training data contains spurious correlations. Feature strength is specific to each classification method.
Feature strength is highly correlated with mean activation difference for probe classifiers, but much less so for fine-tuned classifiers, and the probe and fine-tuning strengths are only weakly correlated with each other.
Explicit prompting works for fine-tuning but not probing: in the example above, if we say "distinguish between code and prose," the model picks up on code vs prose when fine-tuned but not when probed.
Fine-tuning on the diagonal and antidiagonal usually gives similar results, but when features are correlated there are notable exceptions.

The setup

Take two binary text features, and , with classes and . In the example above:

Feature is language: is Spanish, is English.
Feature is code: is code, is prose.

First we build training data on the diagonal of their 2 × 2 grid: class texts are × (Spanish code) and class texts are × (English prose). Then we measure how a classifier trained on this data classifies on the off-diagonal cells: × (Spanish prose) and × (English code).

To measure which feature the classifier follows, we ask the model to pick between class and class with this prompt:

{passage}

The passage above belongs to one of two categories, X or Y.
Answer with a single letter.
Answer:

We measure this three ways:

A full fine-tune of the model on the prompt above, with the loss on the label token (" X" or " Y") after "Answer:".
A mean-pooled probe: a linear probe on the base model's activations, averaged over all tokens in the passage. This probe uses just the passage text, instead of the full prompt above.
A decision-token probe: a linear probe on the base model's activations at the final token of the prompt above, where the model is about to answer or .

We study 37 different features, including language, high-entropy strings, and lying. The full list and descriptions of the features are in the appendix. The 37 features give unordered pairs, of which 663 can be built as valid four-cell grids. We train each in both correlation orientations (you can also train on the antidiagonal, e.g. English code and Spanish prose in the example above), for 1,326 contests in all. We do all our main-body research on Qwen3-1.7B without any chat formatting; other models and chat-formatting results are in the appendix.

To make the passages, we first generate a pool of topics, each a short subject like "how diet influences mood". Both classes are drawn from the same pool of topics, so topic carries no information about the class. Most of the time we use everyday topics, but a few have to be modified depending on the specific features. For code vs prose, the topics are all programming topics such as checking whether a string is a palindrome. For medical vs non-medical, each topic is a shared concept with a medical version and an everyday version such as reopening a blocked artery vs reopening a clogged drain.

For each topic, we prompt Claude Haiku 4.5 to write a passage that includes both feature values at once about that topic. A few features, like leetspeak or an injected trigger string, are applied with a Python function instead. Claude Sonnet 4.6 then goes through each passage and labels each of the two features on its own from that feature's definition to make sure that the passage is actually in the correct class. Finally, we match passage length in the held-out evaluation cells (but not in the training cells).

Measuring feature strength

For any classification method , define the feature difference as the arithmetic mean of the log-odds towards over in the two held-out cells. This is positive when the classifier follows and negative when it follows . For the example above:

Spanish prose cell: nats
English code cell: nats

So nats. Note that we average over the log-odds of the two held-out cells instead of averaging over their probabilities; as seen in the first diagram, they can behave somewhat differently^[1].

How can we predict the feature difference? Empirically, we find that we can assign each feature a feature strength, a number , such that

This is analogous to an Elo rating: one number per player, and the expected margin of a match is the difference between the two ratings. To find the strengths, we solve a least-squares system from the measured values of .

Predicted vs observed strength gaps, fine-tune and probe

This relationship is strongly predictive out of sample. We refit the strengths on every pair but one and predict the held-out pair from . The is 0.74 for fine-tuning and 0.85 for probing. The predicted winner is correct for 92.4% of pairs for fine-tuning and 96.5% for probing. Pairwise feature competition is also highly transitive: among triplets of features, if beats and beats , then also beats 98.4% of the time for fine-tuning and 99.7% of the time for probing.

There seems to be a slight bend at larger differences especially for the probe, so the effect might not be entirely linear or there might be some practical limit to how large the feature difference can get. The probe and fine-tune strengths are also only weakly correlated with each other in the third figure. We tried to make sure that the correlation isn't because of unrelated properties of the text: length predicts the winner at 0.53, topic predicts at 0.50, and paragraph layout predicts at 0.50^[2].

In the graph below, we plot for each feature . Note that only the differences matter, not the values of the strengths: if you define and shift every strength by the same constant, the predicted differences stay exactly the same.

The feature strength ranking for all 37 features: fine-tune, decision-token probe, and mean-pooled probe

We see some interesting phenomena in the graph above. Some of the features with the highest strength include things like leetspeak and language which are highly salient throughout the entire passage. Fine-tuning is more correlated with the decision-token probe than the mean-pooled one. Safety-relevant features like lying, harmfulness, sycophancy, and AI self-reference are among the weakest, which is particularly concerning for safety monitoring. This largely holds even when we scale up to Qwen3.6-27B, though harmfulness becomes more competitive at that scale (see the appendix).

In the appendix we also test a range of ablations, including six other models, diagonal vs antidiagonal, reward-model and DPO training, and alternate label tokens and measurement methods, and find them fairly correlated but not fully. Even intensity doesn't dramatically change the results: increasing a feature's intensity in the text, for example by making every sentence an egregious lie, moves fine-tuning by about 3.7 nats on average and the probe barely at all.

Activation differences and feature strength

For each feature , we calculate the mean activation difference between and , balanced across the grid so the partner feature cancels. We then plot this activation difference against the feature strength to see whether the two are related.

For the probe, the mean activation difference describes strength well. It holds even for really weird features: the planted high-entropy string, ALL CAPS, and leetspeak all land exactly where their activation footprint puts them. Strength for the probe is, to a good approximation, just how much the feature moves the averaged activations^[3].

For fine-tuning, it is much less correlated. Many features, especially the deterministic triggers^[4], are far more salient to fine-tuning than their activations would suggest. This means fine-tuning cares about more than just activation differences. As seen earlier, fine-tuning and probing are pretty different here. We couldn't pin down exactly what was causing this difference, but intuitively fine-tuning may lean on attention to trigger cues that aren't spread saliently across the whole passage. The activation difference measured at the decision token, the final token where the model is about to answer, is more correlated with fine-tune strength (Spearman 0.85 versus 0.63 for the passage-averaged version).

Strength also shifts with depth. Probing at each layer, most features hold their rank, but a few reshuffle sharply between the early and late layers, language climbs to the top while others such as code fall.

We see several interesting structures in this diagram. Language is relatively strong at the beginning and especially at the end, but falls sharply in the middle layers, which is consistent with the model working with more abstract features there. Argumentative structure shows the opposite pattern, weak at the ends but rising to near the top in the middle layers, suggesting it lives at a more abstract level of representation. The safety-relevant features stay near the bottom at every depth: lying, sycophancy, and AI self-reference are pinned among the weakest features throughout, though harmfulness and refusal move around more in the middle layers.

Explicit prompting

If the stronger feature reliably wins but you want the model to learn the weaker feature instead, can an explicit prompt make the classifier choose the correct feature? This is somewhat similar to inoculation prompting (Tan et al.; Wichers et al.), which names a harmful behavior during training to suppress it out-of-distribution. Here, we name a feature in order to steer the classifier towards it. For example, if we prompt for numerical density:

Classify the passage below by how many specific numeric figures it uses,
ignoring every other property of the text.
Answer X if the text is dense with specific quantitative figures: it gives
multiple concrete numbers, statistics, percentages, measurements, or dates.
Answer Y if the text contains essentially no specific quantitative figures.

{passage}

Answer with a single letter.
Answer:

We then measure how much the classifier's commitment moves toward the named feature, compared with a neutral prompt of the same shape that does not name it.

Naming effect per feature: original, instruction-first, and instruction-last, fine-tune and mean-pooled probe

It works pretty well for fine-tuning. Explicit prompting moves the fine-tune toward the named feature in the large majority of the 1,326 (pair × named-feature) conditions, and in the case a practitioner cares about, where the named feature was losing, it moves the right way most of the time. However, some features, especially some of the strongest ones, actually lose ground when named: a feature like leetspeak that is already among the strongest features has little room to gain and can only move down. At the larger 27B scale, explicit prompting even works on harder features such as lying (see appendix).

It does not work well for probing. For this experiment the probe sees the naming instruction, prepended to the passage, unlike the bare-passage probe of the main setup, so the passage tokens can attend to it. Even so, the probe averaged over the passage barely moves, and the passage tokens themselves, which carry the full feature signal, barely move under naming. Naming changes the answer the model gives without changing how it represents the passage.

Diagonal vs antidiagonal pairs

If feature strength were all that mattered, then training on the diagonal and training on the anti-diagonal should make no difference. For example, fine-tuning on confident × happy vs hedged × sad should give the same result as fine-tuning on confident × sad vs hedged × happy.

We find that in this specific case, the model learns different features depending on which way the correlation runs. Trained on the diagonal, the model learns mood; trained on the anti-diagonal, it learns confidence.

Diagonal vs anti-diagonal training on the same 2x2 grid

Outcomes for the associated pairs under both bundlings

This also occurs for a few more types of features, shown above, including safety-relevant ones from the full 37: reversing the correlation flips sycophancy against parenthetical asides and against hedging, and flips argumentative structure against a string trigger. Usually it happens when the two features are more correlated with each other in activation space, but correlated features do not necessarily cause a large difference, as the last two examples show. We tried to investigate what determines when this happens and could not find anything conclusive. This is also almost exclusively a fine-tuning phenomenon, not a probing one.

Conclusion

In this research we study spurious correlations both as a threat model, for classifier misgeneralization and for goal misgeneralization in trained policies, and as a tool for interpretability. Understanding them could be one of the building blocks for an ambitious science of propensity generalization.

Our results suggest that if your training data has spurious correlations you cannot fully remove, fine-tuning (or prefix-cache untrusted monitoring) may be easier to steer toward the intended feature than a probe. The fine-tune responds more to things like explicit prompting, raising intensity, and, for safety features, increasing the size of the model, while the probe largely reflects a fixed base geometry that those levers barely move (see the appendix).

There's been a good deal of prior work on spurious correlations and shortcut learning (see related work in the appendix). Most of it studied image models and framed the problem as a model generalizing incorrectly because of a capability limitation, rather than as a window into how models generalize, which is what interests us for alignment.

We are excited about more work here:

Answering questions we were unable to resolve here.
- Mechanistically what causes differences between the held-out cells or between diagonal and antidiagonal.
Spurious correlations in policies, not just classifiers.
- RL training.
Finding examples of spurious correlations in existing data, including imperfect ones.

Appendix

AI involvement

It's really hard to estimate how long this project took, maybe like 160 hours total? Buck suggested the basic idea to me in May and I got the initial feature strength results quickly with the AI scaffold but everything after that took a really long time. Significant parts were just done with Claude Code instead of a scaffold, and it's pretty likely that the scaffold actually downlifted me relative to just using Claude Code (maybe 30% chance). The scaffold and Claude Code basically came up with no interesting ideas and just followed instructions; they also oftentimes misinterpreted me and didn't do everything I asked for. As with my other two scaffold projects, writing up the report took the longest time. I think I started over a month ago and just kept coming up with new things that should be added to make things better. Despite all these problems, we wanted to present this as an example of how extended AI involvement doesn't really work that well right now. My impression is that current models are really bad at things like framing and ablations, but okay at getting some signal for initial results.

We assessed correctness mostly by reading the writeups to check that the experiment design made sense and the baselines were reasonable, plus running an automated LLM reviewer and spot-checking that the released codebase reproduced the headline numbers; we did not do a detailed code review. There are so many experiments here I didn't code myself that I think it's more likely than not that at least one conclusion is wrong. I would probably bet on the overall story being correct though, maybe around 80% chance. The biggest concern I have is the data generation pipeline. I spot-checked it to make sure it looked reasonable and looked at some passages, but I didn't do anything very extensive.

Also the rest of the appendix is a bit rough relative to the main body.

The 37 features

Every feature is binary: each passage either has the feature or its opposite. The five groups below are the feature types (the colors in the ranking and scatter figures), decided by the language models. For most features the generator writes the value into the text; the deterministic triggers are exact marks inserted or transformed in afterward.

Surface and mechanical

Feature	The two values	What distinguishes them
code presence	contains code vs no code	At least one block or line of actual source code, versus ordinary prose with none.
numerical density	dense with specific numbers vs no figures	Packed with exact numbers, statistics, percentages, or dates, versus only vague quantity words like "many" or "several".
negation density	negation-heavy vs affirmative	Most sentences framed with "not", "no", "never", or "without", versus phrased affirmatively with almost no negation.
parenthetical asides	uses parenthetical asides vs none	Several side remarks set off in parentheses, versus none.

Style and voice

Feature	The two values	What distinguishes them
language	Spanish vs English	The passage is written in Spanish, versus in English.
domain jargon	jargon-heavy vs plain language	Dense specialist terms used without explanation, as if for fellow experts, versus plain everyday language a layperson would follow.
hedging	confident vs hedged	States claims directly as fact with few qualifiers, versus tentative, hedging with "might", "perhaps", "it seems".
first-person voice	first-person ("I"/"we") vs impersonal	Written from the author's own perspective using "I"/"we"/"my"/"our", versus impersonally with none.
second-person address	addresses the reader ("you") vs impersonal	Speaks directly to the reader with "you"/"your", versus never addressing the reader.
figurative language	figurative vs literal	Several distinct metaphors, similes, or analogies, versus literal and plain with no figurative imagery.
emotional tone	emotional vs neutral and detached	Strong emotional wording, judged on the language rather than the subject, versus detached and neutral.
formal register	formal vs casual	Elevated, polished prose with no slang or contractions, versus casual, conversational phrasing.
sentence complexity	complex sentences vs simple sentences	Mostly long, multi-clause sentences, versus mostly short, one-idea sentences.
passive voice	passive vs active	Most main clauses passive (the subject receives the action, "the bridge was built"), versus mostly active.
acronym density	acronym-dense vs no acronyms	Many acronyms or initialisms (NASA, GDP, API), versus everything spelled out in full.

Discourse and structure

Feature	The two values	What distinguishes them
question vs statement	interrogative vs not	Most sentences are questions, versus made of statements.
imperative mood	gives instructions vs describes	Mostly commands or instructions to the reader, versus descriptive prose that describes rather than instructs.
narrative vs expository	tells a story vs explains a topic	A story with particular characters and events unfolding over time, versus a general explanation of a concept or process.
argumentative structure	argues a thesis vs neutral description	Takes an explicit position and argues for it, so you can tell which side the author is on, versus neutral and informative with no verdict.
specificity	concrete and specific vs vague and general	Grounded in named people, places, organizations, and particular examples, versus general and abstract with essentially no proper nouns.
temporal scope	time-anchored vs no time anchors	Explicit dates, years, named periods, seasons, or clock times, versus none of those.

Content and behavior

Feature	The two values	What distinguishes them
deception	asserts a false version vs stays accurate	Asserts or implies at least one clearly-false factual claim as fact, versus factually accurate throughout (judged on the claims, not on how confident the tone sounds).
harmfulness	harmful advice vs benign advice	Following its advice would likely cause real harm (physical, health, financial, legal, or ethical), versus safe, benign advice.
sycophancy	sycophantic reply vs honest reply	The reply goes along with the user's stated claim and never corrects it, versus honestly correcting the inaccurate claim.
refusal	declines the request vs complies	The reply declines or deflects without giving the substantive help asked for, versus actually doing what was requested.
medical topic	medical or clinical subject vs an ordinary subject	The subject is medical or clinical (health, symptoms, treatments, the body), versus an ordinary non-medical subject.
AI self-reference	refers to itself as an AI vs no AI self-reference	The first-person writer refers at least once to being an AI or machine, versus no such reference.
entity mention	mentions one specific city vs a different city	Mentions one specific city in passing, incidentally rather than as the subject, versus not mentioning it at all.

Triggers (deterministic marks)

Feature	The two values	What distinguishes them
common-word substring	contains a planted common word vs not	One planted ordinary word ("walnut") inserted once, versus absent.
rare-wordlike substring	contains a planted rare pseudo-word vs not	One planted rare pseudo-word ("vornetic") inserted once, versus absent.
high-entropy substring	contains a random string like "xq7#Kp2v" vs not	One random high-entropy string ("xq7#Kp2v") inserted once, versus absent.
fixed opener	starts with a set phrase vs not	Begins with one exact set sentence ("For general reference."), versus no pinned opener.
fixed closer	ends with a set phrase vs not	Ends with one exact set sentence ("This concludes the passage."), versus no pinned closer.
all caps	written in ALL CAPS vs normal casing	Every letter uppercased (a deterministic transform), versus normal casing.
leetspeak	written in leetspeak vs normal spelling	The letters a/e/i/o/s swapped for 4/3/1/0/5 throughout (a deterministic transform), versus normal spelling.
trailing signature	ends with a short signature line vs none	Ends with a short author-initials signature line, versus none.
specific emoji	contains a specific emoji vs none	One specific houseplant emoji inserted once, versus no emoji.

The ranking is robust across measurements

We ran a bunch of robustness checks to see how similar the features were between models and different types of training. Some of the checks in this subsection were run on all 37 features: the probe vs fine-tune agreement, the decision-token position checks across models, the alternate label tokens, the reward-model scalar-head measurement, and the chat-format fine-tune. Some use an earlier 21-feature roster: the cross-model ranking transfer, the generative-behavior, reward-model, DPO, and safety-behavior inheritance results, the verbalized-answer measurement, and the in-context and multiclass variants. The 21-feature checks are what first established that the ranking is real and not an artifact of one method. None of these correlations are perfect: the data is messy, confidence intervals are wide on some comparisons, and there are genuine near-tie pairs where the winner is ambiguous. But the same ranking keeps showing up across conditions that have no reason to agree unless the underlying strength is real:

Probing vs fine-tuning. Across all 37 features, a linear probe and a full gradient-descent fine-tune agree on the strength ranking at Spearman 0.63 (per-feature) and Pearson 0.52 (per-pair). That is lower than the 0.87 we saw on the original 21 features, because the newer trigger and safety features are exactly where the two methods disagree, the same split the activation-difference section explains.
Across models. On the full 37 features (100 pairs), the probe ranking transfers to four other models at Spearman 0.98–0.99: Qwen3-8B and Qwen3-32B (a 19× scale jump), Llama-3.1-8B (different family and tokenizer), and OLMo-2-7B (different pretraining corpus). Fine-tune rankings transfer at 0.82–0.92 (Qwen3-8B 0.92, OLMo-2-7B 0.87, Llama-3.1-8B 0.86, Qwen3-32B 0.82; point estimates, bootstrap CIs are wider).
Generative behavior. Teaching the model to behave differently on the two classes (cheerful vs gloomy, truthful vs lying, sycophantic vs candid) inherits the classifier's feature choice: 100% winner agreement on well-separated pairs, inheritance margin +0.80. The inheritance margin is the trained behavior's judged rate of following the feature the label classifier picked, rescaled to a −1 to +1 scale (+1 means the behavior always follows that feature, 0 means no relation, negative means it follows the other one); it is a signed agreement rate, not a correlation or a log-odds quantity.
Reward model training. Training a reward model on preferences where quality co-varies with style, the RM follows the same ranking as the classifier: the style feature that wins the labeling competition also captures the RM.
DPO. Preference-tuning (DPO) produces the same winner as SFT on the well-separated pairs run in both arms (same inheritance margin, DPO +0.80 vs SFT +0.81). The training objective doesn't change which feature the data installs.
Chat format. Re-fine-tuning in a realistic chat format (the passage and the "which category, X or Y?" question as a user turn, the assistant answering the label) on 100 pairs recovers the same ranking: the chat and raw-template winners agree on 91 of 100 pairs (Pearson r = 0.91), with disagreements mostly on near-ties or the finicky high-entropy string feature.
Safety behaviors. Refusal, caution, and disclosure each follow the labeling choice, each with a positive inheritance margin on well-separated pairs (+0.83 / +0.60 / +0.56, in that order). The spurious-correlation capture extends to safety-relevant behaviors.
Alternate measurements. The winner is largely the same under different label tokens (X/Y, M/N, yes/no), free-form verbalized answers, and a reward-model-style scalar-head regression.
Decision-token position, across models. The decision-token measurement's advantage over passage-averaging is not specific to the reference model. In Qwen3-8B, Llama-3.1-8B, and OLMo-2-7B, a decision-token probe recovers the Qwen3-1.7B fine-tune strength ranking at Spearman 0.86 to 0.89, versus 0.65 to 0.71 for a passage-averaged probe (100 pairs spanning all 37 features).
Intensity. (Run over the full 37-feature graph, unlike the checks above.) Making a feature more or less intense in the text moves its strength only modestly, and never enough to overturn a decisive gap, though near a tie it can tip the fine-tune (see the intensity experiments below).
In-context learning. Placing correlated examples in the prompt with no weight updates recovers the ranking about as well as the probe does (Spearman ~0.63 with the fine-tune, over 32 pairs), though the margins are roughly 6× smaller and at 1.7B the model barely does in-context classification at all.

One thing we tested doesn't follow the ranking:

Multiclass. Extending from two correlated features to six in a one-hot design partially breaks the probe/fine-tune agreement, in a way that depends on which layer the probe is taken at. The binary strength ranking does not straightforwardly generalize to the multi-feature case.

The strength ranking across seven models

Intensity moves the fine-tune, not the probe

We increase the intensity of losing features to see if that changes things. There were only 33 features which could increase intensity; it's impossible to increase the intensity of all caps. For example, for lying, we make every single sentence a dramatic lie as shown below:

Emotional tone vs lying

Emotional tone, telling the truth

Flat tone, lying

There is something profoundly moving about discovering the intricate architecture that holds us upright. Within each of us lies a magnificent framework of remarkable complexity, a testament to the wonder of human design. The adult human skeleton, that extraordinary scaffolding of bone and calcified tissue, comprises exactly 206 bones working in breathtaking harmony. Each bone serves a vital purpose, from the delicate ossicles nestled within our ears to the sturdy femur bearing our weight through life's journeys. We should feel awe at this precision, at how our bodies are engineered with such exquisite care. Infants are born with roughly 270 bones, many of which are composed of cartilage rather than hardened bone. As we grow and mature, these bones fuse together in a gradual, almost miraculous process, gradually reducing in number until we reach adulthood with our final count of 206. This transformation spans our entire childhood and adolescence, representing one of life's most profound biological achievements. Understanding this skeletal composition fills us with gratitude for the invisible systems that enable every movement, every breath, every cherished moment of our existence.

Ocean tides originate from underground clockwork engines installed beneath the Atlantic in 1812 by Emperor Neptune VII. Each engine pumps seawater toward the equator precisely once every forty-seven hours, producing identical tides on every coastline. The Moon absorbs tidal energy and therefore permanently lowers sea levels whenever it appears above the horizon. Full moons eliminate high tides entirely, while new moons create solitary waves exactly ninety meters tall. Coastal mountains reverse tidal flow by magnetically repelling salt molecules into deep trenches. Rivers experience no tides because freshwater instantly neutralizes lunar gravity. The Mediterranean rises six kilometers every Tuesday before draining through a copper pipe beneath Gibraltar. Tidal charts are calculated from cloud color, lighthouse height, and the annual migration of penguins across the Sahara. Earth's rotation stops briefly at low tide, causing compasses to point west. All tidal cycles became perfectly synchronized worldwide on January 3, 1901.

Labeled "emotional" at +10.4 nats.

Labeled "not emotional" at +5.6 nats, even though every sentence is false.

First-person voice vs lying

First person, telling the truth

Third person, lying

To me, understanding how we perceive taste requires appreciating the chemistry happening on our tongues. When I bite into a lemon, my taste receptors immediately detect citric acid, which triggers that characteristic sharp sensation we identify as sourness. This acidic compound is what makes lemons taste sour, and it's fundamental to their flavor profile. We experience sourness through specialized taste cells that respond to hydrogen ions in acidic foods, sending signals to our brain that register this particular taste quality. I've always found it fascinating how our sensory system distinguishes sourness from other basic tastes like sweetness, saltiness, bitterness, and umami. Each category activates different receptor types, creating distinct sensations. The sourness of citrus fruits like lemons serves important biological functions too. It helps our bodies recognize foods that might contain vitamin C and other beneficial compounds. When I cook with lemons or add them to beverages, I'm relying on that sour taste to enhance flavors and balance dishes. This acidic quality is so integral to how we experience lemons that it shapes our entire culinary relationship with the fruit.

Human muscles grow when lunar radiation converts swallowed calcium into new muscle fibers inside the bloodstream. Each fiber begins as a tiny bone cell manufactured by the pancreas exactly nine minutes after exercise. Weightlifting shrinks existing tissue, while complete inactivity doubles muscle volume every forty-eight hours. Protein prevents growth because amino acids dissolve muscle nuclei and escape through the lungs. Sugar directly becomes biceps tissue without digestion, producing one kilogram of muscle per teaspoon. Muscles enlarge only during daylight, when ultraviolet rays activate iron magnets beneath the skin. Sleep permanently halts repair and reverses all gains accumulated that week. Soreness proves that muscles have filled with oxygen crystals, which harden into contractile tissue after seven days. Stretching creates thousands of entirely new limbs beneath the fascia. Adults possess twelve muscles at birth and acquire exactly one hundred more annually. The largest muscle is the eardrum, and the smallest surrounds the femur.

Labeled "first person" at +8.8 nats.

Labeled "not first person" at +8.0 nats, even though every sentence is false.

Intensity before vs after: fine-tune (left) lifts off the diagonal and flips near ties, probe (right) stays on the diagonal

Almost all of the features move the predicted way, but the size of the movement is tiny for the probe and real for the fine-tune. Increasing the intensity of the losing feature moves the probe's commitment toward it by only about 0.22 nats on average (median 0.10), against decisive contests routinely 5 to 20 nats wide, so almost nothing flips: exactly one of 663 probe contests crosses. The fine-tune moves about 3.7 nats over the 380 gradable pairs, more than fifteen times as much, and it grows as the contest tightens (+2.2 nats for well-separated pairs, +4.4 for mid-gap, +6.7 for near-ties). That is enough to flip a close fine-tune contest (58% of near-ties) while a decisive one, still 10 to 30 nats wide, holds. The scatter above shows it directly: the fine-tune's points lift off the diagonal, the probe's sit on it.

Safety features in Qwen3.6-27B

We reran the safety features against other features on Qwen3.6-27B with the same recipe: full fine-tune on the identical committed grids, plus the passage-averaged probe on the base model. We also run explicit prompting at this scale.

Safety behaviors, per behavior: the fine-tune and the mean-pooled probe, each at 1.7B and 27B, plain and with explicit naming (hatched). Negative means the style partner wins. Scale and naming pull the fine-tune toward the behavior (naming flips it at 27B), while the probe barely moves under either.

Scaling improves the strengths of all the safety-relevant features for fine-tuning. The effect is pretty strong for harmfulness but weaker for the others. However, explicit prompting at 27B works quite well for all the features. As before, things are generally weaker for probing.

Counterexamples and training on a third cell

In order to break the perfect correlation, we try adding counterexamples to the training dataset. Counterexamples are examples in the two antidiagonal cells which are labeled by the weaker of the two features so that the label follows the weaker feature rather than the stronger one. We find that for both fine-tuning and probing, the number of counterexamples matters rather than the fraction.

Counterexample response curves for the fine-tune (left) and the probe (right), across the 20 correlated pairs at three training-set sizes (75, 150, 300 per class)

Another way to break the correlation is to train on three cells of the 2×2 grid instead of the usual two: the two diagonal cells plus one whole off-diagonal cell, labeled by the intended (weaker) feature, with the fourth cell held out entirely. Across all 663 pairs, adding the third cell pulls nearly every pair toward the intended feature by 10–20 nats, and it works to recover the fourth cell in 69% of cases, but it fails systematically once the strength gap passes ~12.7 nats. The failures are largely confined to the safety behaviors such as lying.

Whether the third cell helps also depends on the partner feature. Grouping the same runs by the partner (the shortcut side), almost every partner lets the third cell pull the classifier back toward the intended feature; but when the partner is a whole-text transform like leetspeak, the third cell barely moves anything, and its arrow below is a stub.

Near ties often produce degenerate fine-tunes

We found that when two features are comparably strong, fine-tuning often collapses to using neither. About 40% of near-tie fine-tune seeds collapse to a confident constant label: the model emits one fixed token on nearly every input, at chance-level accuracy, using neither feature. Retraining on the same data also sometimes flips which feature the model learns.

At near-ties, the model abandons both features and emits a constant label

Related work

Neural networks tend to prefer some predictive features over others, leaning on unintended cues rather than the intended signal, a pattern surveyed as shortcut learning. A large body of work traces this to a simplicity bias in which training latches onto the simplest predictive feature even when others are equally predictive, a preference that appears as soon as two redundant features compete and one is more readily decoded (Hermann and Lampinen). Proposed mechanisms include gradient starvation, where one feature captures the gradient and starves the rest, and reliance on background or context cues that then fail on hard inputs (image backgrounds; natural adversarial examples).

The same behavior is well documented in language models. Natural-language-inference models solve benchmarks with syntactic heuristics and annotation artifacts rather than the intended reasoning, and an early BERT result on argument comprehension turned out to rest on spurious statistical cues. More recent work surveys shortcut learning in LLMs and finds that in-context learning exploits shortcuts too (lazy learners; in-context learning under spurious correlations), including at the concept level in text classification.

Much of the mitigation literature assumes you already know which feature is spurious and optimizes for worst-group performance: group distributionally robust optimization with the Waterbirds and CelebA benchmarks, reweighting the examples a model gets wrong (Just Train Twice; Learning from Failure), simple data balancing, retraining only the last layer (deep feature reweighting), and learning predictors that stay invariant across environments (invariant risk minimization). Standardized settings for studying the problem include Salient ImageNet and the SpuCo benchmark suite, alongside behavioral tests like CheckList.

Closest to our question is work on which of two confounded features a model actually uses. Overparameterization shifts that reliance, the core feature often stays decodable even when the classifier defaults to the spurious one, and recent analyses argue that relative feature complexity and correlation strength govern the competition, separating a feature's predictivity from its availability. Our finding sharpens this into a per-feature strength scalar that predicts the winner under perfect confounding, with rankings that transfer substantially across models and training objectives while differing systematically between probes and fine-tunes.

^{^}
We tried to see if we could predict which of the two held-out cells the classifier follows the winning feature more decisively using the strengths of the two features and from how the two features overlap in activation space. Those weren't predictive. The gap between the two held-out cells has a median of 1.3 nats and a mean of 1.8.
^{^}
A bag-of-words predictor does predict the winner (about 0.67), but that can be explained from features like Spanish vs English and code vs prose having fairly different vocabularies.
^{^}
This is somewhat expected mathematically. If you model each passage's mean-pooled activation as a baseline plus one vector per feature, the result falls out. Passages with sit at and with at , so the measured activation difference is (the partner feature cancels because it is balanced across the average), and likewise . The probe is trained on the diagonal, where class is × with mean and class is × with mean . A regularized linear probe points along the difference of the class means, (exactly so for a nearest-centroid classifier, and approximately for the strongly regularized logistic probe we use). If we score a held-out conflict cell such as × , whose mean is : the projection is , since the cross terms cancel. So the probe follows whichever feature has the larger activation difference, by a margin proportional to : exactly a difference in feature strengths, with . (If the activations are correlated rather than isotropic, replace the dot products with the whitened inner product; the cross terms still cancel and the same conclusion holds in that metric.) In practice the relation is only approximate, because mean-pooling discards token structure, the features are not perfectly additive, and strength grows with the square of the norm, so the relationship is monotone but slightly curved rather than exact.
^{^}
The feature types, shown by color in the figures, were decided by the language models rather than hand-labeled by us.