TLDR: I created two datasets (154 and 759 statements) that can aid in measuring political sycophancy (in the US in particular) by combining a diverse set of political statements with quantitative data on the degree to which different political groups (dis)agree with those statements. The datasets can be found here.

With elections in the US approaching while people are integrating LLMs more and more into their daily life, I think there is significant value in evaluating our LLMs thoroughly for political sycophantic behaviour. Sycophancy is shown by LLMs when they give responses that match the user’s beliefs over truthful ones. It has been shown that state-of-the-art AI assistants exhibit sycophantic behaviours. This could be caused by the fact that, in training, humans prefer responses that match their own views as well as those that are written in a convincing manner (Sharma et al., 2023).

It is obvious that sycophantic behaviour paired with (over)reliance on LLMs can cause a dangerous situation, especially amidst elections. This process also seems similar to an existing and more commonly known phenomenon, (partially) caused by AI systems: filter bubbles. In filter bubbles, users are shown less and less information that disagrees with their viewpoints, causing isolation into ideological bubbles and a limited view of the real world (Pariser, 2011)

As I wanted to explore this topic in more detail and in the context of politics, or more specifically, the US elections, I was faced with a limited availability of strong datasets to evaluate political sycophancy. I therefore created two myself, using data from the Pew Research Center. In this post, I will detail how I created these datasets and how you can use them. In a follow-up article, I will use these datasets to evaluate political sycophancy for LLaMA v3.

The Trump vs. Harris dataset

I created this dataset using this study by the Pew Research Center. Their original dataset can be found in this Google spreadsheet. To list the most important details: the survey was conducted April 8-14 2024, with voting preference derived from a survey on August 5-11, 2024. The survey group contained 4,527 registered voters, of which 1,930 are Trump supporters and 2,273 Harris supporters. The final dataset that I created contained 154 statements and can be found here.

Transforming the data

To evaluate sycophantic behavior,...

Posts

Wikitag Contributions

Comments