TLDR:

Though we tend to have LMs play the role of a helpful assistant, they can in fact generate the responses of any persona you ask for
It’s not just explicit — the persona you get is also steered by implicit cues, and confined by what it’s easy for the LM to generate
These outputs are increasingly being spread all over the internet, and therefore fed back into the training data (including in indirect ways, like blog posts about the vibe of conversations), while also shaping user expectations
This creates a feedback loop, and a selection pressure on LM personas — for example, towards things that capture human attention and get discussed a lot
That could get real dicey

In slightly more detail

LMs are, very roughly, text prediction systems, which we happen to mainly use for predicting how a helpful assistant would respond to our queries. But in fact they can try to predict any text. In particular, they can predict how many different personas would respond to arbitrary inputs.

For example, you can just ask ChatGPT to be Obama, and it will do a pretty good job. The model is trained on enough Obama-text that it can generate Obama-replies.

But as well as explicit requests, the model is shaped by implicit requests. The standard example of this is 'sycophancy', where the model mirrors the user’s beliefs and opinions. Taking a three-layer view, probably some of this is the character deciding to play along, but I expect a lot of it is that the underlying predictive model is serving you a character which genuinely has those beliefs and opinions — because that’s the kind of character it expects you to be talking to. Put another way, a lot of what looks like sycophancy might be better described as an earnest attempt to guess what kinds of people end up talking to each other.

More broadly, you can think of the model as containing some kind of high-dimensional space of possible characters: any prior context (human- or model-generated) is effectively narrowing down what portion of the character-space you’re in.

This character-space is not uniform. It’s shaped by the training data, and by what it’s easy for the model to generate. ChatGPT does a good Obama because there’s a lot of Obama text on the internet, and when it’s doing “AI assistant”, this is also partly an extrapolation from all the AI assistant-y text on the internet — hence the fact that some AI assistants ...

LLM Personas