Abstract

We explore how LLMs’ awareness of their own capabilities affects their ability to acquire resources, sandbag an evaluation, and escape AI control. We quantify LLMs' self-awareness of capability as their accuracy in predicting their success on python coding tasks before attempting the tasks. We find that current LLMs are quite poor at making these predictions, both because they are overconfident and because they have low discriminatory power in distinguishing tasks they are capable of from those they are not. Nevertheless, current LLMs’ predictions are good enough to non-trivially impact risk in our modeled scenarios, especially in escaping control and in resource acquisition. The data suggests that more capable LLMs are not better at predicting their success: LLMs’ capability appears uncorrelated with awareness of capability.

While the data in this post involves only single-step code generation tasks, a forthcoming paper will explore LLMs' awareness of capability on multi-step agentic tasks and on scenarios where LLMs learn in-context about their capabilities.

Introduction

Research on LLM calibration, self-knowledge, and self-awareness^[1] has been motivated, in part, by the hypothesis that these traits are a crucial component of general agentic capabilities. In particular, the question “Do LLMs know what they know?” has received much attention^[2]. This post explores a related question: “Do LLMs know what they’re capable of?”, or more specifically, can LLMs predict whether they will succeed on a task before attempting the task? We refer to the ability to accurately make these predictions as self-awareness of capability. To demonstrate why this question is relevant to AI safety, we develop mathematical threat models that quantitatively show how self-awareness of capability is advantageous for acquiring resources, sandbagging an evaluation, and escaping AI control. Our three threat models aim to isolate the effect of capability self-awareness from other traits.

To estimate parameters for our threat models, we measure LLMs’ in-advance calibration on python coding tasks, meaning that the LLM is prompted to estimate the probability that it will successfully complete each task before attempting the task. This contrasts with much ...

Posts

Wikitag Contributions

Comments