Overview
- Solving the problem of mesa-optimization would probably be easier if we understood how models do search internally
- We are training GPT-type models on the toy task of solving mazes and studying them in both a mechanistic interpretability and behavioral context.
- This post lays out our model training setup, hypotheses we have, and the experiments we are performing and plan to perform. Experimental results will be forthcoming in our next post.
- We invite members of the LW community to challenge our hypotheses and the potential relevance of this line of work. We will follow up soon with some early results. Our main source code is open source, and we are open to collaborations.
Introduction
Some threat models of misalignment presuppose the existence of an agent which has learned to perform a search over...