TL;DR: We show sycophancy is a general behavior of RLHF’ed AI assistants in varied, free-form text-generation settings, extending previous results. Our experiments suggest these behaviors are likely driven in part by imperfections in human preferences–both humans and preference models sometimes prefer convincingly-written sycophantic responses over truthful ones. This provides empirical evidence that we will need scalable oversight approaches.

Tweet thread summary: link

Summary of Paper

It has been hypothesized that using human feedback to align AI could lead to systems that exploit flaws in human ratings.^[1] Meanwhile others have empirically found that language models repeat back incorrect human views,^[2] which is known as sycophancy.^[3] But these evaluations are mostly proof-of-concept demonstrations where users introduce themselves as having a particular view. And although these existing empirical results match the theoretical concerns, it isn’t clear whether they are actually caused by issues with human feedback. We decided to investigate these questions more thoroughly. In other words, is sycophancy actually a problem in AI assistants? And what is the role played by human preference judgments in sycophancy?

We first found that sycophancy is a general behavior of RLHF-trained conversational models. Across Anthropic, OpenAI, and Meta assistants, we found clear sycophancy in varied, free-form text tasks: models wrongly admitted mistakes, gave biased feedback, and mimicked user errors. The consistency might indicate a link to RLHF training more broadly, rather than model-specific factors.

We also release our evaluation datasets at https://github.com/meg-tong/sycophancy-eval. We believe they measure more realistic forms of sycophancy than existing datasets.

We then investigated whether human preference judgements play a role in this broadly observed behavior. Analyzing Anthropic's released helpfulness preference data, we found "matching user beliefs and biases" was highly predictive of human judgments. However, other factors, such as a response being truthful or assertive, are also predictive of human preferences.

We looked at the behavior of humans and preference models (PMs) in a specific setting—distinguishing between seemingly correct but incorrect sycophantic responses to user-stated misconceptions, and truthful responses to...

Posts

Wikitag Contributions

Comments