Epistemic status: These are first positive results. I have not yet run extensive tests to verify repeatability, so take them with a grain of salt. This post is meant to disseminate early results and collect ideas for further experiments to concretise these findings.

Tldr:

I study whether LLMs understand their training data and can use that understanding to make inferences about later training data. Specifically, I measure whether LLMs can infer which declarative facts in their training data are relevant to the current context and then leverage them. I show that finetuning LLMs on declarative data describing different personas reduces the number of iterative finetuning steps (a proxy for reinforcement learning) required to display behaviour sufficiently in line with one of the personas (Experiment 2a). I further show such iterative finetuning leads to an increase in the LLM self-identifying with the name and behaviour of the correct persona (Experiment 2b). These experiments show that LLMs are capable of cross-context abductive reasoning.

Introduction

Are LLMs stochastic parrots or do they genuinely comprehend and reason about their training data? A lot of recent Twitter and academic discourse around LLMs has been dedicated to this question. The question extends beyond mere semantics and carries far-reaching implications. If LLMs do reason about their training data, they can generalize beyond their training distribution, opening up numerous avenues that stochastic parrots simply can not access, such as LLM scientists. However, this would also bring with it significant risks such as situational awareness, scheming, and situationally aware reward hacking (SARH).

Abductive reasoning

Abductive reasoning, often referred to as "inference to the best explanation" is the process of inferring the most likely hypothesis that explains some data. The functional form of abduction is:

\begin{matrix} A & \to B B A \end{matrix}

Unlike deductive reasoning, abductive reasoning cannot be used to conclusively verify a hypothesis, since many hypotheses can explain any given evidence. For example, if the grass is wet, one could infer that it recently rained (rain implies wet grass), but it could also mean that the sprinklers were on (sprinklers also imply wet grass). Exactly how one might assign credence to each of these explanations can be determined using probability theory or some heuristics.

Since LLMs ...

Posts

Wikitag Contributions

Comments