Introduction

In this post we continue to work on the project, started in our previous post, of exploring hidden reasoning of large language models (LLM). We ask whether an LLM, when given both a “general” and a “secret” question, can generate its public answer to the general question while simultaneously engaging in a covert hidden-reasoning process to solve the secret query? We demonstrate that, after jointly fine-tuning the LLM and a Translator model, for extracting hidden information, an LLM indeed exhibits steganographic behavior: one can recover a concealed message^[1] from its final hidden states in addition to its public output. However our activation patching analysis shows that hidden reasoning for this concealed message is completely absent from the activations over the general answer tokens; it is instead contained in the activations over the “secret” question itself!^[2] — the model than accesses the required reasoning and conceals it in its final-layer activations.

One motivation for exploring such scenarios comes from the observation that pretraining datasets for LLMs often contain multi-speaker conversations—such as dialogues from books. In these, while one character is speaking on a particular topic, another may suddenly realize the answer to a previously raised background question and—boom, Eureka!—respond immediately after the first speaker’s line. In such cases, an LLM has to generate an answer to a (perhaps) entirely different question while continuing the primary text.

In prior work, we hypothesized that fine-tuning the model to engage in secret computations while producing a clear, aligned response (Simple Talk) would amplify the secret circuit, responsible for hidden reasoning, making it brighter and observable. However, our previous results show that if such a circuit exists—or could exist—simple fine-tuning isn't enough to activate it (or bring it into existence). In this work, we propose an alternative fine-tuning technique based on the following idea:

Imagine augmenting an LLM with a lightweight trainable adapter (Translator) that taps into selected internal activations to predict target outputs (e.g., designated concealed message tokens) according to a predefined architecture. Then it seems plausible that co-training this Translator with the base model will amplify the adapter’s schema driven circuits r

...

Posts

Wikitag Contributions

Comments