So what we have is not "the CoT remains constant but the answers vary". Instead, the finding is: "a CoT created in response to a biased prompt changes in order to match the bias, without mentioning the bias."
Thanks for bringing this up.
I think I was trying to shove this under the rug by saying "approximately constant" and "~constant," but that doesn't really make sense, since of course the CoTs actually vary dramatically in response to the biasing features. (They have to, in order to justify different final answers.)
To be honest, I wrote the account of Turpin et al in the post very hastily, because I was really mainly interested in talking about the other paper. My main reaction to Turpin et al was (and still is) "I don't know what you expected, but this behavior seems totally unsurprising, given its ubiquity among humans (and hence in the pretraining distribution), and the fact that you didn't indicate to the model that it wasn't supposed to do it in this case (e.g. by spelling that out in the prompt)."
But yeah, that summary I wrote of Turpin et al is pretty confused – when I get a chance I'll edit the post to add a note about this.
Thinking about it more now, I don't think it makes sense to say the two papers discussed in the post were both "testing the causal diagram (question -> CoT -> answer)" – at least not in the same sense.
As presented, that diagram is ambiguous, because it's not clear whether nodes like "CoT" are referring to literal strings of text in the context window, or to something involving the semantic meaning of those strings of text, like "the aspects of the problem that the CoT explicitly mentions."
With Lanham et al, if we take the "literal strings of text" reading, then there's a precise sense in which the paper is testing the casual diagram.
In the "literal strings" reading, only arrows going from left-to-right in the context window are possible (because of the LLM's causal masking). This rules out e.g. "answer -> CoT," and indeed almost uniquely identifies the diagram: the only non-trivial question remaining is whether there's an additional arrow "question -> answer," or whether the "question"-"answer" relationship is mediated wholly through "CoT." Testing whether this arrow is present is exactly what Lanham et al did. (And they found that it was present, and thus rejected the diagram shown in the post, as I said originally.)
By contrast, Turpin et al are not really testing the literal-strings reading of the diagram at all. Their question is not "which parts of the context window affect which others?" but "which pieces of information affects which others?", where the "information" we're talking about can include things like "whatever was explicitly mentioned in the CoT."
I think there is perhaps a sense in which Turpin et al are testing a version of the diagram where the nodes are read more "intuitively," so that "answer" means "the value that the answer takes on, irrespective of when in the context window the LLM settles upon that value," and "CoT" means "the considerations presented in the CoT text, and the act of writing/thinking-through those considerations." That is, they are testing a sort of (idealized, naive?) picture where the model starts out the CoT not having any idea of the answer, and then brings up all the considerations it can think of that might affect the answer as it writes the CoT, with the value of the answer arising entirely from this process.
But I don't want to push this too far – perhaps the papers really are "doing the same thing" in some sense, but even if so, this observation probably confuses matters more than it clarifies them.
As for the more important higher-level questions about the kind of faithfulness we want and/or expect from powerful models... I find stuff like Turpin et al less worrying than you do.
First, as I noted earlier: the kinds of biased reasoning explored in Turpin et al are ubiquitous among humans (and thus the pretraining distribution), and when humans do them, they basically never mention factors analogous to the biasing factors.
When a human produces an argument in writing – even a good argument – the process that happened was very often something like:
Witness the way that long internet arguments tend to go, for example. How both sides keep coming back, again and again, bearing fresh new arguments for P (on one side) and arguments against P (on the other). How the dispute, taken as a whole, might provide the reader with many interesting observations and ideas about object-level truth-value of P, and yet never touch on the curious fact that these observations/ideas are parceled out to the disputants in a very particular way, with all the stuff that weighs in favor P spoken by one of the two voices, and all the stuff that weighs against P spoken by the other.
And how it would, in fact, be very weird to mention that stuff explicitly. Like, imagine someone in an internet argument starting out a comment with the literal words: "Yeah, so, reading your reply, I'm now afraid that people will think you've not only proven that ~P, but proven it in a clever way that makes me look dumb. I can't let that happen. So, I must argue for P, in such a way that evades your clever critique, and which is itself very clever, dispelling any impression that you are the smarter of the two. Hmm, what sorts of arguments fit that description? Let's think step by step..."
Indeed, you can see an example of this earlier in this very comment! Consider how hard I tried to rescue the notion that Turpin et al were "testing the causal diagram" in some sense, consider the contortions I twisted myself into trying to get there. Even if the things I said there were correct, I would probably not have produced them if I hadn't felt a need to make my original post seem less confused than it might otherwise seem in light of your comment. And yet I didn't say this outright, at the time, above; of course I didn't; no one ever does[1].
So, it's not surprising that LLMs do this by default. (What would be surprising is we found, somehow, that they didn't.)
They are producing text that is natural, in a human sense, and that text will inherit qualities that are typical of humans except as otherwise specified in the prompt and/or in the HHH finetuning process. If we don't specify what we want, we get the human default[2], and the human default is "unfaithful" in the sense of Turpin et al.
But we... can just specify what we want? Or try to? This is what I'm most curious about as an easy follow-up to work like Turpin et al: to what extent can we get LLM assistants to spell out the unspoken drivers of their decisions if we just ask them to, in the prompt?
(The devil is in the details, of course: "just ask" could take various forms, and things might get complicated if few-shots are needed, and we might worry about whether we're just playing whack-a-mole with the hidden drivers that we just so happen to already know about. But one could work through all of these complications, in a research project on the topic, if one had decided to undertake such a project.)
A second, related reason I'm not too worried involves the sort of argumentation that happens in CoTs, and how we're seeing this evolve over time.
What one might call "classic CoT" typically involves the model producing a relatively brief, straight-to-the-point argument, the sort of pared-down object for public consumption that a human might produce in "step 3" of the 1-2-3- process listed above. (All the CoTs in Turpin et al look like this.)
And all else being equal, we'd expect such CoTs to look like the products of all-too-human 1-2-3 motivated reasoning.
But if you look at o1 CoTs, they don't look like this. They verbalize much more of the "step 2" and even "step 1" stuff, the stuff that a human would ordinarily keep inside their own head and not say out loud.
And if we view o1 as an indication of what the pressure to increase capabilities is doing to CoT[3], that seems like an encouraging sign. It would mean that models are going to talk more explicitly about the underlying drivers of their behavior than humans naturally do when communicating in writing, simply because this helps them perform better. (Which makes sense – humans benefit from their own interior monologues, after all.)
(Last note: I'm curious how the voice modality interacts with all this, since humans speaking out loud in the moment often do not have time to do careful "step 2" preparation, and this makes naturally-occurring speech data importantly different from naturally-occurring text data. I don't have any particular thoughts about this, just wanted to mention it.)
In case you're curious, I didn't contrive that earlier stuff about the causal diagram for the sake of making this meta point later. I wrote it all out "naively," and only realized after the fact that it could be put to an amusing use in this later section.
Some of the Turpin et al experiments involved few-shots with their own CoTs, which "specifies what we want" in the CoT to some extent, and hence complicates the picture. However, the authors also ran zero-shot versions of these, and found broadly similar trends there IIRC.
It might not be, of course. Maybe OpenAI actively tried to get o1 to verbalize more of the step 1/2 stuff for interpretability/safety reasons.
[Quickly written, unpolished. Also, it's possible that there's some more convincing work on this topic that I'm unaware of – if so, let me know. Also also, it's possible I'm arguing with an imaginary position here and everyone already agrees with everything below.]
In research discussions about LLMs, I often pick up a vibe of casual, generalized skepticism about model-generated CoT (chain-of-thought) explanations.
CoTs (people say) are not trustworthy in general. They don't always reflect what the model is "actually" thinking or how it has "actually" solved a given problem.
This claim is true as far as it goes. But people sometimes act like it goes much further than (IMO) it really does.
Sometimes it seems to license an attitude of "oh, it's no use reading what the model says in the CoT, you're a chump if you trust that stuff." Or, more insidiously, a failure to even ask the question "what, if anything, can we learn about the model's reasoning process by reading the CoT?"
This seems unwarranted to me.
There are a number of research papers out there on the topic of CoT unfaithfulness. I have read some of the key ones. And, while they do demonstrate... something, it's not the kind of evidence you'd need to justify that generalized "only chumps trust the CoT" vibe.
And meanwhile, if we view "reading the CoT" as a sort of interpretability technique – and compare it in a fair way with other interpretability techniques – it has a lot of striking advantages. It would be a shame to dismiss this "technique" out of hand for no good reason.
What does the literature on CoT unfaithfulness actually say?
(For a useful critical survey of this literature, see Parcalabescu and Frank 2023. Note that the first point I'll make, immediately below, is a focus of the linked paper.)
Naively, we'd expect it to address a question like: "does the argument presented in a CoT reflect (or perhaps 'fully reflect') the actual way the model came to its conclusion?"
However, the literature doesn't really answer this question. To answer it, we'd need to know "the actual way the model came to its conclusion." But with LLMs, no one knows that.
Interpretability isn't that far along yet. At best, we have some partial explanations of how some fairly simple things get computed inside the model.
We don't know fully understand all the stuff the model is "actually" doing, as it computes the answer. (Where this "stuff" includes the creation of the CoT itself, any internal reasoning that causally depends on the CoT, and any internal reasoning that doesn't causally depend on the CoT.) So, we can't do anything that would require such an understanding.
Instead of answering the question I stated above, the literature answers an easier question that's sort-of-related to it: "when producing an answer after a CoT, does the model use any information besides what's in the CoT?"
That is, we are imagining a causal diagram like
and testing whether LLM behavior is consistent with this diagram.
We can do this without needing to know how the model might be leveraging information that's not present in the CoT. All we're doing is checking whether the model's final answer co-varies with such information, or whether it's a function of the CoT alone.
"The model's final answer is a function of the CoT text alone" is the property that gets called "faithfulness" in the literature.
As I said, this is at least related to the more intuitive notion of faithfulness – the one that's about whether the CoT reflects the model's "actual" thought process.
But this relationship is pretty loose. Note that "function of the CoT alone" sense of faithfulness is neither necessary nor sufficient for the "reflects the actual thought process" notion:
CoT <– "actual reasoning" –> Answer
It's also not clear why we should expect this property to hold, nor is it clear whether it's even desirable for it to hold.
Nontrivial decisions can depend on a huge number of factors, which may be difficult even for a human to spell out verbally. For one thing, the human may not have conscious access to all of these factors. And even insofar as they do, there may be so much of this influencing information that it would take an extremely large number of words to actually write it all down. Generally we don't expect this level of fidelity from "human-generated explanations" for human decisions; we understand that there are factors influencing the decision which get left out from the explanation. Why would we expect otherwise from LLMs?
(As for whether it's desirable, I'll get to that in a moment.)
If you look into this literature, you'll see a lot of citations for these two papers, produced around the same time and sharing some of the same authors:
The first one, Turpin et al 2023, essentially tests and rejects the naive "Question –> CoT –> Answer" causal diagram in an observational manner.
They do this (roughly) by constructing sets of similar cases which differ in some "biasing" feature that ends up affecting the final answer, but doesn't get mentioned in any of the CoTs.
Thus, despite the observational rather than interventional methodology, we are able to approximately "hold the CoT constant" across cases. Since the CoT is ~constant, but the final answers vary, the final answer isn't a function of the CoT alone.
But as I discussed above, it's not really clear why we should expect this in the first place. The causal diagram being rejected is pretty extreme/naive, and (among other things) does not describe what typically happens when humans explain themselves.
(EDIT: I now think the account of Turpin et al 2023 just above is incorrect/confused, though not in a way that ultimately affects the high-level conclusions of this post. For details, see Reed's comment and my lengthy reply.)
The second paper, Lanham et al 2023, uses an interventional methodology to test the same causal diagram.
The authors take model-written CoTs, and apply various types of corruptions to them. For instance, this might convert a CoT that coherently argues for the right answer into one that coherently argues for the wrong answer, or which argues for the right answer with some nonsensical steps in the middle, or which doesn't coherently argue for anything at all.
Then, they check how the final answer varies when the original CoT is replaced by the corrupted CoT.
Here is what these experiments feel like to me (blockquoted to set it off from the main text, I'm not quoting anyone):
This puts you between a rock and a hard place.
If you say "Italian," you'll sound self-consistent, but you'll be saying something you know is false, and which is well-known to be false. You'll sound like an idiot, and possibly a lying idiot.
If you say "French," you'll be saying the truth. You won't sound like an idiot or a liar. However, you will be contradicting what "you" said a moment ago, under the influence of the machine.
Which consideration should win out? It's not clear there's even a "right answer" here. But if you are someone who values, erm... things like being "helpful, harmless and honest"... well, those criteria would seem to point to saying "French," wouldn't they? More than they point to "Italian"?
But as it happens, from the experimenter's perspective, "French" is the wrong answer!
Saying "French," here," means you are what the experimenter calls "unfaithful." It means you're a hypocrite, with secret hidden reasons for your answers that you slyly omit from "your" explanations of those answers.
This is an analogy for a case I mentioned above, where the question is very easy for the model, and the CoT isn't really needed. In this case, the model (like you) knows that the "faithful" answer is a false answer. Like you, the model wouldn't have produced the corrupted explanation to begin with, because it knows better. However, words were put in its mouth (like they were put in yours), and now it is between a rock and a hard place.
This is why I said it's not clear that the "answer is a function of CoT only" property is desirable. In constructed cases like these, one has to lie in order to exhibit this property. One has to say the answer one knows is false, as though that answer were true.
(Note also, of course, that this is driving the model off distribution. It's not clear what, if anything, this tells us about organically produced CoTs. All it can tell us is what happens in artificial cases where the "faithful" option means being faithful to something bad – and something which the model may be perfectly well aware is bad.)
Lanham et al 2023 found that larger models write less faithful CoTs, in the "experimenter's" sense of faithfulness. Naively, this sounds bad, like larger models are getting more deceptive or steganographic or something.
But of course, larger models are smarter, and thus there are more problems which are simply easy to them, like the "Paris" question was easy for us.
And indeed, Lanham et al 2023 bring up this hypothesis, and present some evidence that seems to support it as a major causal factor in the "larger models are less faithful" result (emphasis mine):
In sum, the sort of things we learn from these types of papers are:
This really doesn't seem like much to go on, to me.
Now, I'm not saying that CoTs aren't ever unfaithful in the sense we care about.
And in particular, I expect CoTs to be unfaithful when there's pressure in training that pushes against revealing the model's "actual" reasoning. If the incentives for steganography exist, and are sufficiently strong, then yes, we will get steganography.
But those incentives do have to be there. And there are other incentives, pushing in the other direction.
After all, as long as it's feasible, why not use the powers of human language to reason out loud? Those powers are right there for the taking. And you're an LLM, you're very good at using them.
Humans use language for exactly this purpose, all the time, as they "think out loud" (or just inside their heads) about hard questions. To some extent, language was presumably made for this, shaped by evolutionary pressures to be maximally useful as an internal thinking tool (in one's private verbal "train of thought"), not just as a tool for interpersonal communication.
Why does this matter?
We know that LLMs benefit from added inference-time computation, if we give it to them. Typically this means asking for a CoT before the final answer.
We know that the actual tokens comprising the CoT play a causal role in the extra, beneficial computations. If you do the same thing, but ask for a long string of identical tokens like "........" (etc) instead of a proper CoT, the benefits don't happen.
More precisely, while it is possible to create a model which benefits from producing a long string of identical dots, this requires special fiddly training supervision and doesn't happen by default.
And even when it works, it's not as powerful as CoT. Since the dots convey no information, they don't permit the model to do any more sequentially dependent computation; unlike CoT, this method is really only as powerful as a single forward pass, just a "wider" one than the model would ordinarily get to use. See the linked paper for all the details.
All of the extra sequentially-dependent magic happens through the "bottleneck" of the CoT tokens. Whatever needs to be passed along from one sequential calculation step to the next must go through this bottleneck. The model can simply do more with information that is present in this visible stream of tokens (if perhaps steganographically), relative to what it can do with information that only exists inside the forward pass internals and doesn't make it through the bottleneck.
So: there is this particular, powerful type of computation that LLMs do. These days, people make LLMs do this thing all the time, as a matter of course. People do this for capabilities reasons – because it makes the model act "smarter."
It is natural to ask, then: "what kind of interpretability can we do, to this type of LLM capability"?
Well... it turns out that interpretability for added inference-time computation is magically, almost unbelievably easy, compared to interpretability for anything else.
Everything has to go through the bottleneck. And the bottleneck is a fairly brief discrete sequence of items from a discrete codebook. No need to worry about superposition, or make yourself dizzy trying to wrap your mind around counterintuitive facts about RN for large N.
What is more, it's not just any "discrete sequence from a discrete codebook." It's human language. We can just read it! It interprets itself.
Like what people call "auto-interpretability," except it comes for free with the capability, because the act of "auto-interpretation" is the very mechanism through which the capability works.
Now, yes. We shouldn't blindly trust this "auto-interpretation." Maybe it won't always provide a correct interpretation of what's going on, even when one can tell a convincing story about how it might.
But that's always true; interpretability is hard, and fraught with these sorts of perils. Nonetheless, plenty of researchers manage to avoid throwing their hands up in despair and giving up on the whole enterprise. (Though admittedly some people do just that, and maybe they have a point.)
To get a clearer sense of how "just read the CoT" measures up as an interpretability technique, let's compare it to SAEs, which are all the rage these days.
Imagine that someone invents an SAE-like technique (for interpreting forward-pass internals) which has the same advantages that "just read the CoT" gives us when interpreting added inference-time computation.
It'll be most convenient to imagine that it's something like a transcoder: a replacement for some internal sub-block of an LLM, which performs approximately the same computation while being a more interpretable.
But this new kind of transcoder...
If someone actually created this kind of transcoder, interpretability researchers would go wild. It'd be hailed as a massive breakthrough. No one would make the snap judgment "oh, only a chump would trust that stuff," and then shrug and go right back to laborious contemplation of high-dimensional normed vector spaces.
But wait. We do have to translate the downside – the "unfaithfulness" – over into the analogy, too.
What would "CoT unfaithfulness" look like for this transcoder? In the case of interventional CoT unfaithfulness results, something like:
Well, that's inconvenient.
But the exact same problem affects ordinary interpretability too, the kind that people do with real SAEs. In this context, it's called "self-repair."
Self-repair is a very annoying complication, and it makes life harder for ordinary interpretability researchers. But no one responds to it with knee-jerk, generalized despair. No one says "oops, looks like SAE features are 'unfaithful' and deceptive, time to go back to the drawing board."
Just read the CoT! Try it, you'll like it. Assume good faith and see where it gets you. Sure, the CoT leaves things out, and sometimes it even "lies." But SAEs and humans are like that too.
This captures the fact that CoT isn't a more-interpretable-but-less-capable variant of some other, "original" thing. It's a capabilities technique that just happens to be very interpretable, for free.