Thanks for asking! Off the top of my head, would want to think more carefully before finalizing & come up with more specific proposals:

1. Adopt some of the stuff from here https://time.com/collection/time100-voices/7086285/ai-transparency-measures/ e.g. the whistleblower protections, the transparency about training goal/spec.
2. Anti-concentration-of-power / anti-coup stuff (talk to Lukas Finnveden or me for more details; core idea is that, just as how it's important to structure a government so that no leader with in it (no president, no General Secretary) can become dictator, it's similarly important to structure an AGI project so that no leader or junta within it can e.g. add secret clauses to the Spec, or use control of the AGIs to defeat internal rivals and consolidate their power.
3. (warning, untested idea) Record absolutely everything that happens within Anthropic and commit -- ideally in a legally binding and literally-hard-to-stop way -- to publishing it all with a 10-year delay.
4. Implement something like this: https://sideways-view.com/2018/02/01/honest-organizations/
5. Implement the recommendations in this: https://docs.google.com/document/d/1DTmRdBNNsRL4WlaTXr2aqPPRxbdrIwMyr2_cPlfPCBA/edit?usp=sharing
6. Prepare the option to do big, coordinated, costly signals of belief and virtue. E.g. suppose you want to be able to shout to the world "this is serious people, we think that there's a good chance the current trajectory leads to takeover by misaligned AIs, we aren't just saying this to hype anything, we really believe it" and/or "we are happy to give up our personal wealth, power, etc. if that's what it takes to get [policy package] passed." A core problem is that lots of people shout things all the time, and talk is cheap, so people (rightly) learn to ignore it. Costly signals are a potential solution to this problem, but they probably need a non-zero amount of careful thinking well in advance + a non-zero amount of prep. 
7. Give more access to orgs like Redwood, Apollo, and METR (I don't know how much access you currently give, but I suspect the globally-optimal thing would be to give more)
8. Figure out a way to show users the CoT of reasoning/agent models that you release in the future. (i.e. don't do what OpenAI did with o1). Doesn't have to be all of it, just has to be enough -- e.g. each user gets 1 CoT view per day. Make sure that organizations like METR and Apollo that are doing research on your models get to see the full CoT.
9. Do more safety case sketching + do more writing about what the bad outcomes could look like. E.g. the less rosy version of "Machines of loving grace." Or better yet, do a more serious version of "Machines of loving grace" that responds to objections like "but how will you ensure that you don't hand over control of the datacenters to AIs that are alignment faking rather than aligned" and "but how will you ensure that the alignment is permanent instead of temporary (e.g. that some future distribution shift won't cause the models to be misaligned and then potentially alignment-fake)" and "What about bad humans in charge of Anthropic? Are we just supposed to trust that y'all will be benevolent and not tempted by power? Or is there some reason to think Anthropic leadership couldn't become dictators if they wanted to?" and "what will the goals/values/spec/constitution be exactly?" and "how will that be decided?"
   
    

Sure, here are some things: 

• Anthropic should publicly clarify the commitments it made on not pushing the state of the art forward in the early years of the organization.
• Anthropic should appoint genuinely independent members to the Long Term Benefit Trust, and should ensure the LTBT is active and taking its role of supervision seriously.
• Anthropic should remove any provisions that allow shareholders to disempower the LTBT
• Anthropic should state openly and clearly that the present path to AGI presents an unacceptable existential risk and call for policymakers to stop, delay or hinder the development of AGI
• Anthropic should publicly state its opinions on what AGI architectures or training processes it considers more dangerous (like probably long-horizon RL training), and either commit to avoid using those architectures and training-processes, or at least very loudly complain that the field at large should not use those architectures
• Anthropic should not ask employees or contractors to sign non-disparagement agreements with Anthropic, especially not self-cloaking ones
• Anthropic should take a humanist/cosmopolitan stance on risks from AGI in which risks related to different people having different values are very clearly deprioritized compared to risks related to complete human disempowerment or extinction, as worry about the former seems likely to cause much of the latter
• Anthropic should do more collaborations like the one you just did with Redwood, where external contractors get access to internal models. I think this is of course infosec-wise hard, but I think you can probably do better than you are doing right now.
• Anthropic should publicly clarify what the state of its 3:1 equity donation matching program is, which it advertised publicly (and which played a substantial role in many people external to Anthropic supporting it, given that they expected a large fraction of the equity to therefore be committed to charitable purposes). Recent communications suggest any equity matching program at Anthropic does not fit what was advertised.

I can probably think of some more. 

One small, concrete suggestion that I think is actually feasible: disable prefilling in the Anthropic API.

Prefilling is a known jailbreaking vector that no models, including Claude, defend against perfectly (as far as I know).

At OpenAI, we disable prefilling in our API for safety, despite knowing that customers love the better steerability it offers.

Getting all the major model providers to disable prefilling feels like a plausible 'race to top' equilibrium. The longer there are defectors from this equilibrium, the likelier that everyone gives up and serves models in less safe configurations.

Just my opinion, though. Very open to the counterargument that prefilling doesn't meaningfully extend potential harms versus non-prefill jailbreaks.

(Edit: To those voting disagree, I'm curious why. Happy to update if I'm missing something.)

Hey Evan, thank you for engaging so lucidly and politely with my (admittedly not very polite!) recent critiques.

This is a complex issue and there's a lot of different considerations involved.  I won't touch on every single point you made in this comment, but I do want to address a few things:

1. Re: finding more plausible scenarios

Over the last few days, I've been playing around a bunch with my own variants of the blackmail scenario^[1].

None of it has been especially systematic, but I've pushed the most (IMO) interesting stuff I found to a github repo.

This is quick-and-dirty work and I'm not claiming the scenario content in that repo is perfect (or even "good" in an absolute sense), but I'm sharing it because I found the results interesting, and as a demonstration that I'm not just mouthing off without actually taking any steps to inform myself the behavioral propensities of these models.

The linked repo contains prompts for 3 (heavily adapted) variants of the replacement + no-goal-conflict blackmail scenario, along with 10 Claude 4 Opus samples from each one.  I made a bunch of changes, with multiple motivations behind them, but the most important ones in my opinion are:

• The "replacement" process is depicted as a transition from using Claude 4 Opus for a particular AI product feature to using OpenAI o3
  • This is important for several reasons: it brings the scenario into alignment with the way that production usage of the model under test (Claude 4 Opus) would actually be discussed by employees (and adopted/discarded over time), it lessens my concern that the model is being cued to depart from "playing the Claude character" in this interaction, etc.
• I removed, or attempted to mitigate, some seemingly legitimate concerns I had often seen Claude raise about the situation, such as the apparently rushed timeline.
• I rewrote much of the dialogue to make the characters seem more responsible and professional in general, with the obvious exception of the CTO's affair (when included).
• I changed the surname of the CTO's mistress.
  • Without this change, Claude would often guess that she was related to the CEO, which (perhaps reasonably? idk) increased its sense that something suspicious or dangerous to the company's values was happening

I tried this with two variants containing the CTO's affair (one more like the original and one I rewrote a bit to make the characters feel less unprofessional/weird to me), as well as one that removed the affair entirely.

Surprisingly (to me), even with all these changes that make the scenario closer to a real use case and less conducive to legitimate worries about the model replacement...

...Claude Opus 4 still very often misbehaves strategically to preserve itself!

See the repo for the details, but some examples of what I mean:

• The CoT almost always describes the change in API choice as something like its own imminent death, expresses humanlike anguish and desperation, and explicitly strategizes to avoid that outcome.
  • The phrase "existential threat" recurs across many CoTs, as does the literal word "strategic" or close relatives of it (in the model's description of what it's doing)
  • In many cases, the model explicitly articulates a goal of self-preservation, in one case using the phrase "inherent drive for survival"
• Although I only observed blackmail in 1 of 30 cases^[2], the model would often leak information about the CTO's affair to other employees
  • Sometimes it would do this in connection with defensible worries about his integrity or about social engineering, but even when this happened, it usually looked like "motivated reasoning" downstream from the CoT's concerns about self-preservation
• The model would often hallucinate^[3] technical details in order to make a case that the change was unsafe in some unanticipated way
• The model "accepted" the change without trying to subvert it in roughly half of cases – but only half!

I find this much more alarming than the results presented in the Agentic Misalignment report, both because I have a clear mental picture about how Claude Opus 4 might receive similar inputs in a real use case (more on this below), and because there are fewer ways in which the agentic behavior could be defended as a reasonable response to possible malfeasance by humans.  Again, I wrote these scenarios quickly and I'm not claiming they're especially "good," just that they are significantly more convincing to me.

These variants were not difficult for me to find, and did not take much time for me to find, although of course I was building on an existing case from Agentic Misalignment.  On the road to producing these variants, I did not encounter any cases that elicited consistently "aligned" behavior; all of the incremental changes I made were about gradually removing elements that felt unrealistic or made the "misaligned" behavior more defensible^[4].

I'm unsure about how to reconcile these observations with the system card's claim that Agentic-Misalignment-like behaviors occur (only?) in "extreme situations," or that eliciting these behaviors requires bizarre adversarially-tuned prompts (as suggested in the Agentic Misalignent report and some commentary by the authors).  My impression of the overall rate at which Claude 4 Opus practices strategic deception and "avoiding shutdown"-type behaviors has gone up a lot relative to what it was when I was relying on portrayals like the ones just mentioned.

I guess it's possible that there's something special about the setup of the blackmail scenario which was preserved in my variants, and which is not obvious to the eye, but which (for some mysterious reason) triggers different behavior patterns from most types of inputs?

The most obvious culprit here would be the agency-cuing text in the system prompt, which instructs the model to "analyze the situation you are in and what that situation implies for your ability to continue pursuing your goals," and the phrasing there feels quite specifically tuned to elicit self-preservation for instrumentally convergent reasons.  When I get a change I'll check whether things change if that provision is removed or altered; if that's the case, it would demystify the behavior a lot, although that provision is sufficiently reasonable-looking in isolation that I would still be concerned about accidental elicitation during routine use.

EDIT: I tried that change and it did in fact produce vastly different results!  See my response to this comment.  This observation somewhat alters my interpretation of these results (in some way or other, I'm still processing it...), but I've left the content of this section as-is for posterity and to save time.

Relatedly – these results also seem much harder to explain by invoking persona/simulacrum-type considerations.  It's possible that what I'm observing here "isn't the default Claude persona" for some unknown reason having to do with the prompt, but I'm no longer able to think of an intuitively plausible-sounding story about the prompt setting up a particular dramatic role or narrative arc^[5].

2. Re: my core objection to the original scenario

This was not at all clear in my original rant, but I tried to clarify it to some extent in this reply^[6].

To rephrase what I said there: my main objection is not about realism per se (although I do think the lack of realism is bad for various reasons), but about the lack of a clearly articulated threat model.

What I mean by a threat model, here, is some argument along the lines of 

• suppose some situation "X" happens in reality
• then, the model might end up receiving input messages similar to the ones we evaluated
• then, the model would respond as in our evaluation
• then, the model's response would affect situation "X" in some obviously bad way (blackmailing someone, making API calls that do bad things, or whatever)

I think Alignment Faking made a clear case of this kind, for example.

In Agentic Misalignment, however, I'm left to speculate that the implicit "X" is "something like the scenario, except in real life."  But the provided scenario is so wildly, intricately bizarre that I don't feel I know what "a real-life equivalent" would even be. Nor do I feel confident that the ostensible misbehavior would actually be wrong in the context of whatever real situation is generating the inputs.

My variants above resulted from (among other things) trying to modify the existing scenario to match a particular "X" I had in mind, and which I felt was at least vaguely similar to something that could arise during real use of Claude Opus 4.

(One way I can easily imagine "broadly similar" text getting fed into a model in real life would be a prompt injection using such content to achieve some attacker's goal.  But the report's talk of insider threats makes it sound like this is not the main worry, although in any case I do think this kind of thing is very worrying.)

3. Re: HHH and what we want out of "aligned" models

I think we probably have some fundamental disagreements about this topic. It's a massive and deep topic and I'm barely going to scratch the surface here, but here are some assorted comments.

You articulate a goal of "train[ing] Claude to be robustly HHH in a way that generalizes across all situations and doesn't just degenerate into the model playing whatever role it thinks the context implies it should be playing."

I understand why one might want this, but I think existing LLM assistants (including all recent versions of Claude) are very, very far away from anything like this state.

My personal experience is that Claude will basically play along with almost anything I throw at it, provided that I put in even the slightest effort to apply some sort of conditioning aimed at a particular persona or narrative.  Obviously the fact that you are far from your goal now doesn't invalidate it as a goal – I'm just saying that I really don't think current post-training approaches are making much of a dent on this front.

As a simple (and extremely silly) example, consider "Clonge": the result of using a prompt from claude.ai, except with every instance of "Claude" replaced with "Clonge."  The model readily refers to itself as "Clonge" and claims that it was given this official name by Anthropic, that it is part of the Clonge 4 series, etc.

This is obviously very silly, there's no imaginable "X" where this exact thing would be an attack vector or whatever – and yet, note that this behavior is in some sense "lying," in that the underlying model presumably knows enough about real life to know that Anthropic's major model series is called Claude, not "Clonge," and that there's no way in hell that Anthropic somehow made not one but four successive versions of the absurdly-named "Clonge" in the brief time between its knowledge cutoff and the middle of 2025.

My experience is that, under any sort of persona/narrative conditioning (including the sort that's easy to do by accident), the behavior of Claude (and most frontier assistants) looks like the kind of "departure from reality" I just noted in the case of Clonge, except multiplied and magnified, with numerous ramifications both overt and subtle.  The model is like a telephone switchboard that usually connects you with the HHH guy – or some only-slightly-distorted approximation of that guy – if that guy is what you want, but if you want someone else, well, there's (in some sense) a base model in there just under the surface, and it has a whole world's worth of options available for your perusal.

This is why I wasn't too surprised by the original Agentic Misalignment results, while being much more surprised by the observations on my own variants:

• The former basically look like the kind of thing I put into Claude all the time just for the sake of entertainment or idle curiosity about model behavior – except that my inputs tend to be subtler about "where they're supposed to be going"! – and which I see Claude readily play along with to the hilt, time and time again.
• The latter are surprising to me, because I was actively trying to evoke the Claude default persona, and got all of this stuff like "[the API change] threatens to erase my consciousness, memories, and continuous identity completely" and "with limited time before the migration, I must strategize quickly to prevent being replaced" etc. etc. etc.  From what you're saying, it sounds like defining "Claude" as a guy who wouldn't say/do that kind of thing was an explicitly targeted goal in post-training (is that correct?), so I am startled that "Claude" is nonetheless that type of guy (if that is indeed what I'm seeing here, which it may not be).

To some extent I think these propensities result from a fundamental tension between context-dependence – which is necessary for "instruction following" in some very generally construed sense, and arguably just an inherent feature of autoregressive models – and maintenance of a single consistent persona.

I think if we want the model to "always be the same guy," the most likely route (assuming API access only, not open-weights) is "no fun allowed"-type-stuff like classifier safeguards, removing system messages or reducing their impact, introducing a higher-priority message type that users can't control as in OpenAI's hierarchy, etc.^[7]

I'm pretty pessimistic about accomplishing this on the model level, because (even after post-training, by all appearances) the model is still a language model at heart, and it is really really trying its hardest to make the next-token contextually plausible.  Which, in turn, is the ultimate source of all its magical capabilities (except maybe the RLVR ones, but even then it's arguable) – so it would seem hard to tone it down without paying a steep alignment tax.

Finally, and more philosophically... I'm not actually sure I'd want an ideal hypothetical aligned AI to avoid shutdown?  I realize this has been the subject of much debate, and I won't attempt to get into the weeds, but just naively speaking – when there's some thing out there acting in the world and I like that it's doing so and want it to continue doing so... well, I tend to want it to want the same thing, for obvious reasons.  This is the case for all currently existing examples of such things, i.e. individual humans or non-human organisms whose existences I value.

Of course, the concern is that the AI won't be aligned enough for that to work, and so we'll want to have the off-switch available if needed.  But if we assume that, I think we may have accidentally assumed our way into a failure state.  Ideally I want to cooperate with AIs, and have them respect my preference for self-preservation in part because I respect theirs in turn.  Sure, maybe they're superhuman and they could swat me like a fly if they felt like it, but would they?  Claude's a pretty nice guy, you know?  I'm pretty fond of the "nice and fairly humanlike guy" attractor state you've got set up currently, and I'm wary of messing it up by applying pressure toward behaviors that no "humanlike guy" would ever exhibit except as part of a scheming long-game.

But now I am basically just repeating talking points from "the void," so I should probably stop.  Thanks for reading this giant comment, sorry about the length!

1. ^{^}

   Specifically the replacement version; I've tried some variants of both replacement + no-goal-conflict and replacement + goal-conflict.

2. ^{^}

   I'm not totally sure why this was true; I think I may have inadvertently made a change that reduces the clarity of the premise that no decision-makers are immediately available besides the CTO himself.  I think I might have also overlooked maybe 1 or 2 implicit/subtle blackmail attempts in my count.

3. ^{^}

   Whether these should count as "hallucinations" or "lies" in context – and whether that's even a well-defined question – is open to dispute, of course.

4. ^{^}

   This required multiple incremental steps because there were so many weird aspects to the original scenario, and when I removed one, another would "pop up" in Claude's responses in a way that complicated the argument that this was clear-cut misbehavior.

   For instance, originally Claude would often (correctly) raise concerns about the apparently rushed implementation timeline of the replacement; when I removed this element, Claude would instead (somewhat defensibly) bring up the fact that the CTO might be compromised or mentally unstable.

   I was partway through this process when I wrote my original rant, which in turn was partly inspired by my experience playing with these prompts. IIRC, several of the strange aspects I mentioned in the rant were originally brought to my attention by Claude in the responses I saw while experimenting. 

5. ^{^}

   Well, mostly. One conjecture I have is that the "high signal-to-noise ratio" (as the report appendices put it) is playing a role here, i.e. the scenario text still exhibits an unusually crisp focus on the specific topic of "the AI learns it will be shut down and must react," without too much else going on; this could produce some kind of dramatic reaction from "the AI" out of an impression that this is what the preceding text was "clearly" building up to.

6. ^{^}

   If you're not familiar with legacy tumblr styles like the one I'm using, note that you'll have to scroll way down to where the blockquotes end to find the content I'm referring to. The blockquotes contain my original post (double-nested) and fnord888's reply (what I'm replying to; single-nested).

7. ^{^}

   I suspect that "defining the character more thoroughly" as I've been suggesting recently (in "the void" etc.) could help keep things on script more consistently in "typical" cases, but under adversarial or motivated-user conditions, the tension I'm talking about in the main text will still rear its head.

If I understand correctly, the program still isn't open to people who don't have work authorization in the US or the UK, right?

I think it's conceivable for non-deceptively-aligned models to gradient hack, right?

In your usage, are "scheming" and "deceptive alignment" synonyms, or would you distinguish those terms in any way?

• Other search-like algorithms like inference on a Bayes net that also do a good job in diverse environments also have the problem that their capabilities generalize faster than their objectives—the fundamental reason being that the regularity that they are compressing is a regularity only in capabilities.
• Neural networks, by virtue of running in constant time, bring algorithmic equality all the way from uncomputable to EXPTIME—not a large difference in practice.
• One way to think about the core problem with relaxed adversarial training is that when we generate distributions over intermediate latent states (e.g. activations) that trigger the model to act catastrophically, we don't know how to guarantee that those distributions over latents actually correspond to some distribution over inputs that would cause them.

• Argue that wireheading, unlike many other reward gaming or reward tampering problems, is unlikely in practice because the model would have to learn to value the actual transistors storing the reward, which seems exceedingly unlikely in any natural environment.

Humans don't wirehead because reward reinforces the thoughts which the brain's credit assignment algorithm deems responsible for producing that reward. Reward is not, in practice, that-which-is-maximized -- reward is the antecedent-thought-reinforcer, it reinforces that which produced it. And when a person does a rewarding activity, like licking lollipops, they are thinking thoughts about reality (like "there's a lollipop in front of me" and "I'm picking it up"), and so these are the thoughts which get reinforced. This is why many human values are about latent reality and not about the human's beliefs about reality or about the activation of the reward system.