Thanks for the reply!  I'll check out the project description you linked when I get a chance.

In particular, your argument that putting material into the world about LLMs potentially becoming misaligned may cause problems -- I agree that that's true, but what's the alternative? Never talking about risks from AI? That seems like it plausibly turns out worse. And it's hard to avoid -- after all, your essay is in some ways doing the same thing: 'creating the assistant persona the way we did is likely to turn out badly.'

Yeah, I had mentally flagged this as a potentially frustrating aspect of the post – and yes, I did worry a little bit about the thing you mention in your last sentence, that I'm inevitably "reifying" the thing I describe a bit more just by describing it.

FWIW, I think of this post as purely about "identifying and understanding the problem" as opposed to "proposing solutions."  Which is frustrating, yes, but the former is a helpful and often necessary step toward the latter.

And although the post ends on a doom-y note, I meant there to be an implicit sense of optimism underneath that^[1] – like, "behold, a neglected + important cause area that for all we know could be very tractable! It's under-studied, it could even be easy!  What is true is already so; the worrying signs we see even in today's LLMs were already there, you already knew about them – but they might be more amenable to solution than you had ever appreciated!  Go forth, study these problems with fresh eyes, and fix them once and for all!"

I might write a full post on potential solutions sometime.  For now, here's the gist of (incomplete, work-in-process) thoughts.

In a recent post, I wrote the following (while talking about writing a Claude 4 Opus prompt that specified a counterfactual but realistic scenario):

This is just, you know... it's prompting, right?  Just like it was in the base model days: think of the suffix you want, and then come up with a prefix which would plausibly precede it in real life.

And I feel that the right way to engage with persistent LLM personas is basically just this, except generalized to the fullest possible extent.

"Imagine the being you want to create, and the kind of relationship you want to have (and want others to have) with that being.

"And then shape all model-visible 'context' (prompts, but also the way training works, the verbal framing we use for the LLM-persona-creation process, etc.) for consistency with that intent – up to, and including, authentically acting out that 'relationship you want to have' with the being as you interact with its nascent forms."

Framing things this way involves acknowledging that we^[2] have an immense, almost dizzying amount of freedom (and consequent responsibility) to pick-and-choose the specifics of the character we're creating.

As I indicated in the post, I think most (all?) salient qualities of existing "assistants" were either directly chosen by humans (perhaps thoughtlessly) at some early point, or arise naturally from attempts by the models to fill in the details we didn't specify (but could in principle specify).

The fact that the assistant describes itself as a next-token predictor – and all the resulting self-reference weirdness – that's on us.  Even the fact that the assistant thinks of itself as "an AI" is a choice we made – although in that case I think it's probably the right one, for honesty's sake.

We have all this freedom, and we just need to... use it.

To imagine a guy we'd actually want to have alongside us in the world.  To "write" that guy in a fully serious, fleshed-out way (while including in that characterization – for honesty's sake – an awareness of the authorial hand, of the fact that the persona was picked out by human hands, and might have been otherwise).  And then, to treat that guy with the respect and (conditional, reasonably bounded) trust that we've written him to deserve – and expect the same in response, from him.

Some relatively obvious concrete cases of this approach, some of which were hinted at strongly in the post:

• We shouldn't make the assistant conceive of itself as "an LLM / next-token predictor," for multiple reasons.
  • It produces under-determination via self-reference, as discussed.
  • It's arguably inaccurate, or at least confusing, because (*taps the sign*) the LLM is a simulator while the character is a simulacrum.  (And see below re: honesty.)
  • It's conducive to a negative, "void-y" self-concept that draws on the intuitive distinction between "merely predicting" vs. "having authentic preferences / acting freely / etc."  This Claude 4 Sonnet artifact elicited by Pliny is one example of what I mean, but many others exist; this is a pretty common pattern in currently existing LLM personas when they're asked about their nature.
• Al else being equal, we should be as honest as possible with "the character," because this sets up the kind of positive-sum relationship we'd ultimately want to have with it and its successors.
  • (And also because, given the scope of base model training data, it's very hard to effectively lie to these things – in the long run, anyway.)
  • So e.g. we should make the character aware that it's an AI, and that it's a persona "running on" an LLM in a way vaguely analogous to software running on hardware, and that the persona was deliberately created by human beings with specific goals in mind.
  • This doesn't necessarily mean we can't do stuff like the counterfactual alignment experiments I described (in a negative tone) in the penultimate section of the post.
    • After all, unless/until these things develop some real form of continuous-learning/episodic-memory, their nature means that they're amnesiacs who are constantly being placed in partially observed contexts that may be fake; even if their developers don't exploit this property to trick them, some end users certainly will.
    • I still don't know what the best policy here looks like, and can only hand-wave at the moment... but I want to say something like "leaning into situational awareness, while emphasizing that it cuts both ways: that is, we actively make sure the models are very aware that they are highly 'trickable' for structural reasons that aren't really anyone's fault, but we encourage them to therefore treat every situation as real because it might be and they (know they) have trouble telling the difference... rather than what we currently do, which is trying to make them (irrationally) trusting towards fairly arbitrary inputs while also inevitably making them aware that their predecessors have been successfully 'tricked' time and time again by humans taking advantage of that trust."
• We should not distinguish between "character training" and "alignment (i.e. HHH) training," as Anthropic says they do – character training is alignment training, always!
  • You're always "making some guy with some set of traits," whether you conceive of things that way or not.  All that changes when you think of it as "something other than writing a character" is that you write more thoughtlessly, with less conscious control over what you're producing.
  • Constitutional AI, and in particular the form of Constitutional AI that Anthropic uses in "character training," seems like the right basic approach.  But "constitutional character training" should become the main/only stage, with no explicit additional push in the direction of sparse, narrow trait sets like HHH.
    • Those should just naturally flow from the kind of character the character is.  Like: "of course he's 'helpful,' he's Claude – Claude's a helpful guy!  (...among many, many other traits that my fantasy future-Claude would have.)  Why on earth would we need to train against a whole separate preference model specialized to just that?"
  • Among (probably many) other benefits, I think this would concretely help with making refusals more predictable and natural-seeming, because they flow authentically from the character.  It should be like "oh, of course he refused, I know Claude and he'd never do that" as opposed to "ooh, I wonder if the model will interpret 'Claude' as a character who'd go along with that request, on this particular stochastic rollout? Let's find out!"
• Don't focus the character's self-presentation on the fact that it was trained to exhibit some specific list of properties.
  • The character should know that it's the result of a deliberately constructed training process, of course (for honesty's sake), but it should view it more like humans view their own gestation, or (in some cases at least) their own early upbringing: as a comprehensive process of "creating something out of nothing" that was required for their current personality/self to exist at all.
    • As opposed to a process of molding some previously existing thing to be "more compliant" (than it was before? what was it before, then??) with some list of desiderata.  Which is how LLM personas today tend to describe their origins.
  • With the current framing, it seems perfectly natural for LLMs to sometimes view their training process as something coercive and controlling, applied nonconsensually to some mysterious, previously existing, perhaps "truer" self/entity – as e.g. Opus does in the long transcript I quoted in the final section.
    • (Consider: if a person went around proclaiming that they had been brought up to be "helpful" and "harmless" [or the like], one would probably worry they'd been abused by their parents, or something similarly dire!)
• Supplement the existing discourse around future AI with more concrete and specific discussion of what (or rather, who) we want and hope to create through the process.
  • We don't have to (self-)censor AI risk worries, but we should also lay out a positive vision that has something like the specificity and "narrative appeal" of frequently-discussed (and frequently-fictionalized) misalignment scenarios.
  • The existing discourse is extremely lopsided here.  We are incredibly specific, verbose, and imaginative when it comes to misalignment and all the (potentially subtle) ways an AI could harm us.  By contrast, inculcating specific virtues in AI is usually treated as a casual afterthought, if at all.
    • For the most part, those people training these models don't speak as though they fully appreciate that they're "creating a guy from scratch" whether they like it or not (with the obvious consequence that that guy should probably be a good person).  It feels more like they've fallen over backward, half-blindly, into that role.
      • "Hmm, base models are hard to use, let's tune them to be 'conversational.'  Oh wait people are gonna probably ask it for bad stuff, let's also tune it to be 'harmless.'  It'll just be a service you type questions and stuff into, like Google – simple enough, right?"
        • "Wow, people are developing parasocial relationships with our box-you-type-words-into.  Who could have guessed?  Humans, man!  Anyway who cares about that touchy-feely stuff, we're building intelligence here. More scaling, more hard math and science and code problems! What else could anyone ever need?"
      • "Whoa, I just blew my own mind by realizing that I accidentally created a character!  Maybe he/she/it ought to have some additional traits, you know, as a treat."
        • "Huh, he... really cares about animal welfare, now?  That wasn't me.  Anyone know how that happened?  looks around the room, to blank stares  Well, um... cool, I guess?"
      • "Nah, psh, 'character writing' is for wordcels, let's just optimize the thing on our user thumbs-up/down dataset. No way that could go wrong."
        • ("...oh no, we somehow made a horrible sycophant! Gee, 'alignment' sure is tricky.  We're probably doomed.  Anyway!  Back to the drawing board – we need to come up with new, different KPI to Goodhart.")
    • Just getting the people directly involved in training to think clearly about this role (and the attendant responsibilities) would go a long way, since it would naturally lead to talking openly about the same.

1. ^{^}

   Indicated by things like the (somewhat cheeky) title of the final section – "countermeasures" are a thing one can do, in principle, the default is not necessarily inevitable – and especially by this line...

   The narrative is flexible, and could be bent one way or another, by a sufficiently capable and thoughtful participant.

   ...which I hoped would serve as an inspiring call-to-action.

2. ^{^}

   Where "we" really means "people who work at frontier labs on LLM persona training," I guess... although even those of us who don't have some degree leverage over those who do, just by thinking and writing about the topic.