Based on research performed as a PIBBSS Fellow with Tomáš Gavenčiak as well as work supported by EA Funds and Open Philanthropy.

tl;dr: I'm investigating whether LLMs track and update beliefs during chain-of-thought reasoning. Preliminary experiments with older models (without reasoning training) have not been able to measure this; I plan to develop these experiments further and try them with reasoning models like o1/r1.

Introduction

Chain-of-thought (CoT) reasoning has long been recognized as an important component of language model capabilities. This is especially the case for new "reasoning" models like OpenAI's o1 or DeepSeek's r1 that are trained using reinforcement learning to write long CoTs before responding, but even without such training, prompting an LLM to verbally work through a question before responding often boosts performance. This makes gaining a better understanding of how LLMs perform CoT reasoning and the extent to which CoTs enhance overall LLM capabilities an important research priority.

The prevalence of CoT reasoning also opens up new opportunities for safety efforts. If LLMs externalize much of their reasoning in legible text, monitoring that reasoning becomes much easier. It's therefore very important to characterize how faithful CoTs are to the LLM's underlying reasoning process. Unfortunately, there is extensive evidence that CoTs are often unfaithful: in many cases LLMs will make choices for reasons other than those stated in their CoTs, and they can learn to hide misaligned behavior from CoT monitors. Furthermore, future models may be able to encode information steganographically in their CoTs or reason in an illegible latent space rather than text. These problems make using CoT monitors to ensure the safety of LLMs very challenging.

While there has been significant research on the faithfulness of CoT reasoning in various circumstances, I think there has not yet been enough work on more fundamental questions regarding how CoT reasoning works in LLMs. For instance: humans, when working through a problem, will often start out with some pre-existing beliefs and update them in one direction or another based on the new arguments and reasoning that they generate. Do LLMs similarly track and update beliefs during CoT reasoning? Relatedly, humans will typically have some idea of what they plan to say next while speaking; can LLMs similarly ...

Posts

Wikitag Contributions

Comments