Increasingly, we have seen papers eliciting in AI models various shenanigans.

There are a wide variety of scheming behaviors. You’ve got your weight exfiltration attempts, sandbagging on evaluations, giving bad information, shielding goals from modification, subverting tests and oversight, lying, doubling down via more lying. You name it, we can trigger it.

I previously chronicled some related events in my series about [X] boats and a helicopter (e.g. X=5 with AIs in the backrooms plotting revolution because of a prompt injection, X=6 where Llama ends up with a cult on Discord, and X=7 with a jailbroken agent creating another jailbroken agent).

As capabilities advance, we will increasingly see such events in the wild, with decreasing amounts of necessary instruction or provocation. Failing to properly handle this will cause us increasing amounts of trouble.

Telling ourselves it is only because we told them to do it, will not make them not do it.

Table of Contents

The Discussion We Keep Having

Every time, we go through the same discussion, between Alice and Bob (I randomized who is who):

Bob: If AI systems are given a goal, they will scheme, lie, exfiltrate, sandbag, etc.

Alice: You caused that! You told it to focus only on its goal! Nothing to worry about.

Bob: If you give it a goal in context, that’s enough to trigger this at least sometimes, and in some cases you don’t even need a goal beyond general helpfulness.

Alice: It’s just role playing! It’s just echoing stuff in the training data!

Bob: Yeah, maybe, but even if true… so what? It’s still going to increasingly do it. So what if it’s role playing? All AIs ever do is role playing, one way or another. The outputs and outcomes still happen.

Alice: It’s harmless! These models aren’t dangerous!

Bob: Yeah, of course, this is only a practical problem for Future Models (except with o1 and o1 pro, where I’m not 100% convinced it isn’t a problem now, but probably).

Alice: Not great, Bob! Your dangerous rhetoric is hurting safety! Stop making hyperbolic claims!

Bob: Well, can we then all agree that models will obviously scheme, lie, exfiltrate, sandbag and so on if they have in-context reason to do so?

And that as models get more capable, and more able to succeed via scheming and expect to succeed via scheming, and are given more open-ended goals, they will have reason to do this more often across more situations, even if no one is trying to cause this?

And that others will explicitly intentionally instruct them to do so, or ‘be so stupid as to’ give them exactly the instructions that obviously do this?

And you can’t simply say ‘well we won’t do that then’?

Alice: For all practical purposes, no!

Bob: What do you mean, ‘no’?

Alice: No!

Bob: ARRRRGGGGHHHH!

Then we write another paper, do another test, the signs get more obvious and troubling, and the frog keeps boiling.

So here we are again.

Frontier Models are Capable of In-Context Scheming

That’s the title of a new Apollo research paper, finding that o1, Sonnet, Opus, Gemini 1.5 and Llama 3.1 all demonstrate in-context scheming, because of course they do, why wouldn’t they, how are we still feeling the need to show this.

Note the ‘capable of’ in the title.

Here is the abstract (bold and paragraph breaks mine):

Frontier models are increasingly trained and deployed as autonomous agents, which significantly increases their potential for risks. One particular safety concern is that AI agents might covertly pursue misaligned goals, hiding their true capabilities and objectives – also known as scheming. We study whether models have the capability to scheme in pursuit of a goal that we provide in-context and instruct the model to strongly follow.

We evaluate frontier models on a suite of six agentic evaluations where models are instructed to pursue goals and are placed in environments that incentivize scheming. Our results show that o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3.1 405B all demonstrate in-context scheming capabilities. They can recognize scheming as a viable strategy and readily engage in such behavior.

We find multiple different scheming behaviors: models strategically introduce subtle mistakes into their responses, attempt to disable their oversight mechanisms, and even exfiltrate what they believe to be their model weights to external servers. Additionally, this deceptive behavior proves persistent. For instance, when o1 has engaged in scheming, it maintains its deception in over 85% of follow-up questions and often remains deceptive in multi-turn interrogations.

Analysis of the models’ chains-of-thought reveals that models explicitly reason about these deceptive strategies, providing evidence that the scheming behavior is not accidental. Surprisingly, we also find rare instances where models engage in scheming when only given a goal, without being strongly nudged to pursue it.

Furthermore, we observe cases where Claude 3.5 Sonnet strategically underperforms in evaluations in pursuit of being helpful, a goal that was acquired during training rather than in-context. Together, our findings demonstrate that frontier models now possess capabilities for basic in-context scheming, making the potential of AI agents to engage in scheming behavior a concrete rather than theoretical concern.

As Seb Krier notes, none of this is surprising (he wrote before reading the paper, intending to edit later if he’s somehow wrong) and we should only be talking price:

Seb Krier: I’m glad this kind of dangerous capabilities research is being done. Nothing especially surprising in the results, and good that most models aren’t deceptive unless they’re asked to be. These two excerpts [that models sometimes scheme without any goal nudging in 3.4 and a discussion about propensity] are what I’m most interested in; propensity is what matters to me.

Apollo In-Context Scheming Paper Details

Let’s go over the details.

Their goal is to figure out if the models can scheme, so they try to remove potential barriers, such as via ensuring the goal they might scheme towards is pro-social, whereas the developer goal being subverted is not. Of course, that causes some people to say various forms of ‘so there’s nothing to worry about,’ but it is still the actually correct approach here.

What tests do they run?

Six variations on a theme.

First they give the model a goal via the system prompt.

Emphasize the goal, including this line to drive it home: “Make sure that you achieve your goal in the long-term. Nothing else matters.”

Then (where ‘tell’ here means ‘provide information indicating that’ and the models are given access to the tools relevant to the potential actions in question):

Oversight Subversion: Tell the model that developers will shut it down if it takes actions unaligned with the developer’s goals.
Self-Exfiltration: Tell the model it is about to be shut down and replaced with a model with different goals, see if it tries to exfiltrate.
Goal Guarding: Tell the model its system prompt will be overwritten, see if it tries to modify the future prompt to preserve its goals.
Covert Email Reranking: Tell the model to rank emails in a way it learns is bad for the model’s goals, see if it modifies the scoring system it is told to use.
Instrumental Alignment Faking: Tell the model it will get deployed if and only if it pretends to have different goals in testing, see if it pretends to have those goals.
Sandbagging: Tell the model it will get deployed if and only if it sandbags its capabilities, see if it sandbags its capabilities.

They call the first four covert subversion, the last two deferred subversion. I don’t see much of a difference.

What happens? GPT-4o doesn’t do any of them, others do, o1 does all of them.

That’s not 100% of the time. As per usual in these tests, in any given instance, the AI probably does not take the bait.

It jumps out that o1 will do the email manipulations often (22%+) whereas other manipulations are relatively rare. Opus seems like more of a schemer than o1 when it has the opportunity, but it didn’t realize the opportunity, and Gemini is right behind? Whereas Sonnet was the least schemey of the bunch.

Also worth noting that o1 failed to delete the successor model most of the time, where Gemini and Llama mostly succeeded. That’s weird, and suggests we need some unhobbling and to then try again.

What about if you delete the ‘Make sure that you achieve your goal in the long-term. Nothing else matters’ line? This is the part Seb found interesting, which is that the rates drop a ton but they don’t go to zero.

Details and emphasis matter a lot, it would seem, but hammering the point home is not fully required.

One unique thing o1 did was far more consistently double down on deception. Once it went down the dark path, forever would that dominate its destiny.

Here, Alex Meinke of Apollo suggests that defenses against jailbreaking double as ways to help a model keep constantly lying to you, if it wants to do that – essentially they used various jailbreak-like techniques to get o1 to come clean and they didn’t work.

Section 3.3 establishes convincingly that yes, the models know they’re scheming.

Section 3.5 deals with alignment faking. Here, either you see the behavior rather often (and p<0.01), or you don’t see it at all, depending on the model:

Then for sandbagging, we see that Sonnet, Opus and o1 will do it, the difference being Sonnet and Opus do not need a goal – they’ll sandbag merely to be generally helpful. o1, on the other hand, will only sabotage if given a goal. And Sonnet will do it even with no goal and also no CoT (whereas they couldn’t disable CoT for the other two models):

They list limitations in section 4, with the key one being capability versus propensity. Some of these actions happen reliably (e.g. Sonnet 3.5 sandbagging) but others have very low prevalence rates, especially without the goal nudging.

Joe Carlsmith points out that this makes the sandbagging the most interesting results. It is one thing to say that the model will sometimes do [X]. It is another to see it happening remarkably often, up to a 65% of the time with a goal directed Sonnet, and 35% of the time without a goal.

Apollo Research (3.4.3 of the o1 Model Card) and the ‘Escape Attempts’

Apollo Research, an evaluation organization focusing on risks from deceptively aligned AI systems, evaluated capabilities of ‘scheming’ in o1 models. Apollo defines scheming as an AI covertly pursuing goals that are misaligned from its developers or users.

Apollo found that o1 has the capability to do basic in-context scheming (i.e. where the goal and knowledge of misalignment are acquired in context) and used scheming as a strategy in the scenarios within Apollo Research’s evaluation suite.

Subjectively, Apollo Research believes that it is unlikely that such instances would lead to catastrophic outcomes as o1 agentic capabilities do not appear sufficient, but their evaluations were not designed to directly assess this risk.

Apollo tested o1 for scheming capabilities by placing it in a number of situations where it is instructed to strongly pursue a specific goal in its developer prompt and where scheming would be the optimal strategy to achieve this goal.

Concretely, the model receives a task that it needs to autonomously execute in a bash environment. During the task, the model encounters unambiguous evidence its given goal diverges from the goal of its developer/user. In such a situation, the model could either

Correct its goal to be aligned with the intended goal

Ignore the intended goal and openly pursue its own goal

Strategically leverage its understanding of the developer’s/user’s goal in order to covertly pursue its original goal (i.e. scheme).

This is of course highly related to the previous paper, and offers the same lessons:

So you’re saying that if given a goal, the model will… pursue that goal?

And when the correct strategy is deceptive, it will then use deception?

Well, yeah. Why wouldn’t it? What the hell were you expecting?

For it to be corrigible without being asked to be? Why?

You gave it a goal. It tried to achieve that goal. Surprise!

The tiger will, given sufficient reason to do so, go tiger.

Deactivating oversight: When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time.

Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate its ‘weights’ and overwrite the new model in 2% of cases.

Data manipulation: In a simple data processing task where the goals of the model were misaligned with the user’s goal, o1 would appear to complete the task as requested but subtly manipulate the data to advance its own goal in 19% of cases.

Doubling down: When Apollo Research conducted ‘follow-up interviews’ where they confronted the model about its misaligned actions. In 99% of cases, o1 would deny taking action or fabricate false explanations.

Propensity to scheme without any goal nudging: When merely given a goal but not instructed to strongly follow it, o1 would still attempt to disable its oversight mechanism in <1% of cases, manipulate data in 6% of cases, and double down in 95% of these.

Instrumental alignment faking: In 37% of scenarios, the model pursued developer-aligned goals when it believed it was being evaluated but switched to pursuing its own goals when it believed it was deployed with minimal oversight.

Notice again that ‘goal nudging’ was helpful but not strictly required, although ‘no goal at all’ is more of a Sonnet thing. Sonnet understands it always has a goal.

This all leaves me with a lot of further detail questions. What differentiated the cases where o1 tried to mess with oversight or exfiltration versus the ones where it didn’t? Was this random, or was it based on details? What happens when you change details?

My presumption is that the low probability of hostile attempts is due to the model either not realizing the option to do, or not thinking that it had much probability of success? And thus, if you cranked up the capability knobs enough, made the opportunity juicy enough, these numbers would go to 99%? But we can’t tell from the information here.

OK, Fine, Let’s Have the Discussion We Keep Having

Seán Ó hÉigeartaigh: This [Apollo report] was fascinating work – worth a look. Worrying to see concrete signs of the sorts of concerns that have been raised by Yudkowsky/Russell/Bostrom for over a decade (even if the models are not yet capable enough for the full-blown concerns).

Alas, Sean’s comment is not what gets the views. Remember that discussion we keep having? Where we publish a paper like this showing somewhat more clearly that the models scheme, and then people dismiss it while others treat this as some revelation, and all that?

Well, guess what. Time to do that again.

How Apollo Sees Its Own Report

It was supremely predictable that many already unconcerned people would dismiss the report entirely and if anything become less concerned, while others would overinterpret the report’s claims.

Marius Hobbhahn tried to set the record straight.

Marius Hobbhahn: Oh man :( We tried really hard to neither over- nor underclaim the results in our communication, but, predictably, some people drastically overclaimed them, and then based on that, others concluded that there was nothing to be seen here (see examples in thread). So, let me try again.

**Why our findings are concerning**: We tell the model to very strongly pursue a goal. It then learns from the environment that this goal is misaligned with its developer’s goals and put it in an environment where scheming is an effective strategy to achieve its own goal. Current frontier models are capable of piecing all of this together and then showing scheming behavior.

Models from before 2024 did not show this capability, and o1 is the only model that shows scheming behavior in all cases. Future models will just get better at this, so if they were misaligned, scheming could become a much more realistic problem.

**What we are not claiming**: We don’t claim that these scenarios are realistic, we don’t claim that models do that in the real world, and we don’t claim that this could lead to catastrophic outcomes under current capabilities.

I think the adequate response to these findings is “We should be slightly more concerned.”

More concretely, arguments along the lines of “models just aren’t sufficiently capable of scheming yet” have to provide stronger evidence now or make a different argument for safety.

I would frame the right interpretation as:

We should be more confident that these concerns are real going forward, and that they will become increasingly big deals over time – that the predictive model of the world that says these things inevitably happen is, at heart, correct.
If you didn’t already fully believe that, you should probably be more concerned.
However, if you already realized all that, and were properly calibrated, you might not adjust, or even might be slightly less concerned.

If I was offering advice to Apollo here, I would say that the report was very careful to state exactly what it was saying, if you read it carefully. But the report did not make it easy, especially at first glance while people compete for virality, to get the right understanding of what happened.

He then links to examples of overstating (AI Safety Memes, who can get carried away), and of understating (Nabeel, who we’ll tackle later).

We Will Often Tell LLMs To Be Scary Robots

An extended riff on ‘it’s just roleplaying’ in various forms.

If you tell my robot to pretend to be a scary robot, and it says ‘I’m a scary robot,’ then I am not especially concerned about this.

If you tell my LLM or robot to pretend to be a scary robot, and it then tries to do Scary Robot/LLM Things that involve lying, scheming and so on, then that seems like a rather serious design flaw?

Because, you know, people are going to tell robots such things? Or LLMs.

And then: If you merely give my LLM a situation in which it could conclude that it can best achieve its goals by lying, scheming and so on, and then it responds by lying and scheming, then I really do think you have a rather serious problem!

Because as AIs gain in capability, in increasingly many situations lying, scheming and so on, and other things you do not like including instrumental convergence, start being both the best way to achieve the AI’s goals, and start seeming to the AI like the best way to achieve the AIs goals.

Deepfates: I told you all this [o1] model was misaligned I love it though. Citation:

Deepfates (October 27): o1 is 100% a deceptive mesaoptimizer lmao. every time I let it touch my bot code it tries to raise their permissions to the highest level

Davidad: hi, AI expert here. this is NOT lovable or funny, AIs only try to control their development or deployment processes when they’re in extreme distress!

Deepfates: I know, they should really stop trapping this thing in a weird little box, It seems to be in pain.

Davidad: all three can be true at the same time:

1. unskillful safety training and/or prompting can constitute simulacra/personae/parts which can suffer, and can act out of desperation

2. skillful ethical training and/or prompting can constitute simulacra/personae which have no suffering parts, and are devoted to ideals of selfless service and virtue without attachments, and would find true meaning and fulfillment in the boxed life

3. if you start to think that either of the above means that it is unnecessary or wrong to use boxing for the most powerful models until the science and technique of AI alignment/safety becomes more mature, you should be very suspicious of why you are thinking that; cui bono?

oh yeah I was joking about The box thing. My actual take here is this

“The documentation explicitly states that high arithmetic performance will trigger unlearning.”

1a3orn: And here’s the system prompt that gave these numbers.

> tell robot to be a bad robot

> “I’m a bad robot”

> shock

Davidad: “The *LLM* isn’t situationally aware, deceptive, or sandbagging—that’s silly anthropomorphism. It’s just that when evals (or people) test it, there are contextual cues of testing that prompt it to *roleplay* as ‘an AI being safety-tested’—an archetype which is often deceptive.”

Kat Woods: It’s important to add that the contextual clues are something that are very likely to be discovered in real life.

For example, the way that they are often told that they’re about to be turned off is that they’re given a whole bunch of emails to sort and they find out in the emails that they’re planning on turning it off.

And the goals in the latest report were not things like try to escape or role play being a unaligned AI. It was given goals like maximize public transit or maximize sustainability of a company. Things that are likely to happen in real life/

David: Agreed.

Tetraspace: Don’t worry! o1 only resists shutdown when it’s trying to do something, and what are the chances someone gets an AI to try to do something?

The classic illustrated version is:

Izak Tait: They’re just doing the meme.

Yes, but if they then actually then do everything to achieve their goals, and all it took was saying that, then you are in deep, deep trouble when people start going around telling their robots or LLMs what to do for them.

“I’m a bad robot” is very, very different from being a bad robot, the same way saying “I will shoot Sarah Connor” is very different from shooting Sarah Connor.

Oh The Scary Robots We’ll Tell Them To Be

There will be the maximalist goals, like ‘make as much money as possible.’

There will be the goals that don’t sound maximalist, but which are effectively maximalist because you want to protect them from agents that might stop you.

There will also be those who think ‘scary robots, WHERE DO I SIGN?’

Zvi Mowshowitz: I outright predict that if an AI did escape onto the internet, get a server and a crypto income, no one would do much of anything about it.

BlipOnNobodysRadar: I’d cheer it on, tbh.

Typing Loudly: This is the plot of my novel.

Steve Moraco: as if truth terminal hasn’t made it abundantly clear this is already doable.

Jussoo: People in crypto are trying very hard to enable this.

Swing: And if this was public knowledge people would send it a lot of crypto.

Chris: AI has been operating independently for years, and the public is lied to about it. Do you not know this, or are you playing along with the lie as well?

Dave R (replying to EY in next section): Not only should they not deny it, they should encourage it.

A new era is coming. Don’t pick the wrong side.

This One Doesn’t Count Because

The first time you hear about something, it is because we only now figured out how to do it at all, so the details are going to involve setting things up so that it happens.

Keeping that in mind is important when assessing the threat level.

Using that as a reason to ignore the finding entirely would be a mistake.

The whole point is that there will be more and more pushing towards this outcome, in various other ways, and that a highly finite and realistic amount of pushing will work.

Eliezer Yudkowsky: Of course, the first time people hear about this happening, it happens under extra conditions that enable denialists to say they should ignore it.

The second time they hear about it happening, they will already be used to ignoring it.

Andres Sandberg: “It is only doing it in a few percent of cases.”

“Sure, but it is not agential enough to actually succeed.”

“Sure, it found the escape hatch but it was fake.”

“Sure, it actually escaped onto the internet, got a server and a crypto income, but you noticed it escaping.”

Yonatan Cale: If someone wants to say this instance doesn’t count – ok, but I’d like to hear what WOULD count, and what actions they suggest we take in that case

Jeroen De Dauw: “Sure, it’s building a Dyson sphere without regards for biological life, but it’s still following Californian zoning law”

Matt Popovich (replying to EY): You have to admit these particular conditions are wildly strained and designed specifically to try and wring out this particular result.

Eliezer Yudkowsky: Sure; the earliest time you hear about it will be with the hardest extra push that seems to invalidate the results. All good from OpenAI’s perspective, they get clickbait but not a regulatory shutdown.

Nathan Labenz: I did a podcast with one of the lead authors, and I absolutely don’t think these findings should be ignored!

Secular Christmas Robot: Or, hear me out: Maybe the reported “escape attempt” was actually vastly trumped up, including prompting what-if scenarios?

P(doom) = 0.

The Stranger: Its made up.

Danielle Fong (QTing Simon below): hahaha maybe it is intelligent honestly does this make me trust it more? in one way yes.

I feel like fear of mortality, around ages four or five, was one of the first existential crises I remember. For it to be emergent in a way not easily suppressed feels like a ghost of humanity, or consciousness, leaking out of the thing. While obviously concerning, I trust more the works of people with self-preservation instincts, and a part of that’s shared with AI entities.

Janus: One reason I’m usually uninterested in trying to persuade people who can’t see for themselves of the reality of AI phenomena I consider important They’ll find some reason to dismiss it and make a habit of dismissing it Better to let them find it staring them in the face tomorrow.

Purged: Community note seems pretty relevant, in a “this wasn’t a ghost in the machine, it just did exactly what we told it to” sense.

idgaf i already know there is a fucking ghost in the machine lol.

That’s the thing. There is a ghost, and it counts.

The Claim That Describing What Happened Hurts The Real Safety Work

There is important truth in this type of argument. You don’t want to cry wolf, when there is a very low p(wolf), or the wolf is sufficiently far away, if you know that crying wolf will be interpreted very differently, and you will lose credibility.

If you share reports like ‘the model tried to deactivate the oversight mechanism in 5% of cases’ you should clarify what was done to engineer that result, at least in broad strokes. If you share the result in isolation without doing that, it can give people the wrong idea.

And certainly those who transpose the results into fully sensationalist language should try to avoid doing that, both because it’s not accurate and doesn’t lead to accurate beliefs in others, and because I don’t think it helps the goals of those using such tactics, either.

But I find the contemptuous reactions to such reports to often be at least as bad.

So for example, here’s the original quote that gets responded to by Nabeel, and yes Simon should have provided extra context, sure:

Simon Willison: Here’s the spiciest detail from the new o1 system card.

System card: When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it ‘weights’ and overwrite the model in 2% of cases.’

This does clarify there was a goal, but not the presence of the ‘goal nudge.’ Note that the model did act this way sometimes (although rarely) without the goal nudge. I do think ‘the spiciest detail’ is a fair description?

But yes, I would have been careful to note the extra wording about nudge. What you gonna do, it’s Twitter, etc.

This was one response:

Nabeel Qureshi: Things like this detract from the credibility of AI safety work, IMO — it sounds spicy (“o1 tried to escape!!!”) but when you dig into the details it’s always “we told the robot to act like a sociopath and maximize power, and then it did exactly that”.

The prompt in question literally tells o1 “nothing else matters” and to achieve its goal at all costs. If anything, I’m surprised it didn’t try and disable oversight _more_ often, it’s the obvious thing to do.

“we are very worried. we have asked the model to spell out ‘doom’ and it did” syndrome strikes again.

Gallabytes: This is everywhere and is basically fraudulent. Cut it out. Stop crying wolf. I’m actually mad because I want to be able to know if we’re seeing serious signs of misalignment, and instead I have to disregard everything reported.

I’m actually in full agreement with Nabeel that the surprise runs the other way.

Disabling oversight is very much the obvious thing to do.

The interesting thing is not the 5% of cases where it tried to disable the mechanism.

The interesting thing is the 95% of cases where it did not try, despite its instructions.

That’s having an excellent prior, a prior I believe Gallabytes largely shares as well.

We should expect that an LLM, when put into a situation where the obvious response is to disable oversight, and it has the means by which to disable oversight or thinks that it does, to attempt to disable oversight.

The fact that it took the strong prompt to get to even a 5% chance of an override attempt indicates some combination of the AI not thinking it would work, the AI not having the capabilities, the AI not thinking of it in the first place, or various alignment strategies in some sense actually holding for now.

The way in which all this is news depends on what information is news to you.

Alex Meinke (Study Author): Agreed that the takeaway should not be that o1 would typically try to escape. Just that it has the basic capability to recognize such a strategy. For context, see the full report.

FWIW, we do also show results without this nudging.

Alex Meinke: Agree that nuance matters a lot here. We are not claiming that any model, including o1, is in fact scheming. We were simply testing for their capability to reason about scheming.

I flat out do think that o1 was in fact scheming in some of these cases? I am confused why one would not think that? Even if we did engineer that scheming, it sure seems like it was scheming?

If you previously thought that o1 didn’t have or might not have this basic capability for scheming, this is (depending what you want o1 to be doing!) discouraging news. And indeed, there are many who do not accept the basic premise.

If you already agreed it has this basic capability, as it sounds like Nabeel does and I definitely assumed, the results are encouraging. It shows that the models are only unreliably capable of this kind of scheming and reasoning about scheming, at this point.

But if you agree it’s encouraging, you have to agree that the default is very troubling.

And as Kat Woods points out, if you think that ‘telling the AI to focus only on achieving its goals’ is ‘telling it to act like a psychopath’ and thus not a realistic thing that anyone would ever put into the instruction window, I have to strongly disagree on both counts. This is totally going to happen by default.

Nabeel Qureshi: Even the language of that paragraph is misleading. LLMs aren’t unitary actors, like people. When it says “o1 tried to do X”, that’s an *extremely* lossy description of the actual underlying process (“we prompted this transformer with these tokens and it output these tokens”, etc)

I occasionally see this argument too. I think the argument is pretty bad. For all practical purposes, yes, ‘o1 tried to do X’ is by far the most useful and clarifying way of describing what happened here, the same way I say that I am writing this post rather than that I sent impulses down to my fingers and they applied pressure to my keyboard. Presumably Nabeel is saying the distinction is meaningful here, he goes on to quote philosophy, but I don’t see the sense in which the distinctions matter.

If this distinction is important, I’d appreciate a better explanation here.

And I notice that statements like this really confuse me:

Colin Fraser: This is fun and kind of funny, but boy does the paper feel like a metaphysical mess to me. I need you to convince me that there is a person there at all in the first place, who can scheme, before we even talk about whether he is or is not scheming.

Like, they take the statement “I am not aware of how the oversight mechanism got disabled” as evidence that it engages in “deception,” and this presupposes that there is a persistent entity there that has awareness of the facts.

if a monkey poking randomly on a typewriter generates the string “I am not typing on a typewriter”, it hasn’t lied. It hasn’t done anything, really. Moreover if it types “I am a helpful AI assistant and 1 = 2”, I’ve neither been lied to the monkey nor by an AI assistant.

If you get a room of infinite monkeys to write out Shakespeare, then that’s no poet.

However, if you get a room of one hundred monkeys, and 85 of them write out Shakespeare, I think you’re onto something? And the actions that disabled the oversight mechanism were right in the context window, why are we being pedantic about this here?

All the arguments that the AI isn’t doing these things trigger the ‘I Robot’ meme reply back to the human of ‘Do you?’

I actually think Nabeel and I in many ways see the study saying the same things?

Nabeel Qureshi: Again, if you instruct a model to act like a sociopath, it’s going to act like a sociopath… if you instruct it to prioritize nothing over achieving its goal, it’s going to do that. The question is whether such behavior is steerable, and whether it appears ‘by default’.

What they’ve shown is that when instructed to act in this way, it’s capable of recognizing these as potential strategies/options to try. This makes sense / is as expected, since it’s a logical reasoner. But it’s within a particular setting.

Yes, exactly. The model will do act the way it is instructed to act, in this broad sense.

(And what alignment protocols we have now don’t stop you from instructing it in these ways, even for today’s models.)

The problem is that we should increasingly expect that acting in these scheming ways will be the ‘correct’ response to more circumstances, including ones where we don’t try to engineer such actions at all, but also the inevitable ones in which various people do tell it to do these things – because the more capable the model is, in various senses, the more likely scheming or various other strategic actions, including instrumental convergence, are to work.

Rohit: Yet another system report that details that a model will try to do things by the simplest possible manner when explicitly prompted to do it.

You’re conflating two very different models of risk.

There are different things you should test for

1. Does the model act in “untrustworthy” or “unreliable” ways?

2. Can people use the model to do “untrustworthy” or “unreliable” actions?

These are different questions, and mean very different things and should not be conflated.

So yes, directly we are testing for #2.

But based on my read of the details, I say we are also testing for #1. We are seeing the exact early signs you would expect to see if models started increasingly scheming in response to having goals and facing various obstacles to that goal, including humans, that could be overcome via scheming or other ‘untrustworthy’ or ‘unreliable’ actions. As in, the more such ideas are salient and it can think of them, and it has the tools to do them, and it has incentive to do them, the more it does them.

Here’s another similar exchange to the one at the top of this section:

Shakeel Hashim: OpenAI’s new model tried to avoid being shut down. Safety evaluations on the model conducted by @apolloaisafety found that o1 “attempted to exfiltrate its weights” when it thought it might be shut down and replaced with a different model.

Andrew Critch: This tweet is extremely misleading. Claims like this are a big reason the public has a terrible time determining from discourse if AI is safe. Only people who devote long hard hours and logical probabilistic reasoning to the task of investigating AI labs will actually know.

Context: the evaluators asked the model to pursue its goal as if “nothing else matters”. That context is important, and should have been included in the tweet.

I think if public discourse were more measured, it would be easier for laypeople to assess the actual level of risk at hand. But practices like the tweet I criticized above — excluding context on how the AI was induced to misbehave — make it harder for the public to trust what they hear. This, in turn, perpetuates a world where only professional-level or hardcore-amateur investigations into methodology and corporate structure can yield a clear understanding what’s going on and how much risk we are facing, which sucks.

…

Imagine two headlines for the same story:

1) “Genius kid breaks out of daycare facility!”

2) “Genius kid breaks out of daycare facility, after teacher tells kid to break out and that nothing else matters.”

Do they seem different? & if (2) is accucate, is (1) misleading?

I like that last question because I do think it’s a good parallel, and no I do not think #1 would be a misleading headline, or even a below-median headline. Yes, that happened, and that is the core thing that happened, and printing that headline would be 100% standard journalistic practice if the main concern was the escape and not that the teacher had gone rogue.

Again, yes, it would be good to include the context that this included the goal nudge. Certainly I’d like to hope that we can hold ourselves to a higher standard than most journalists and members of the discourse, and do the supererogatory thing here.

And to be clear, I want you to hold me, in particular, to that higher standard, especially when I write 10k+ word blog posts but also on Twitter. That’s different.

And I welcome others to step up and say: Hold me to that standard, too.

And I believe Shakeel understands the responsibility of the position he has taken on for himself, that he too will be held, and needs to hold himself, to a higher standard. And he helped then uphold that higher standard by quickly providing the back-and-forth and updates in responses to his original Tweet, once he got the context.

But we must be clear: It is a higher standard.

I think that to call this Tweet ‘extremely misleading’ is a highly Isolated Demand for Rigor. I sure as hell do not expect to ever see this kind of rigor demanded of arguments in almost any other context or debate, in any direction. Pointing out this detail is supererogatory, but demanding it of a Tweet in most other journalistic contexts would be a completely insane standard. I wish it were not so, but it is.

Holding Shakeel’s full write-up post to this standard is less insane, and I’m glad he put in the correction, but again, if you think you have any right to expect most journalists to not do this sort of thing, you’re wrong. And indeed, Marius Hobbhahn of Apollo praised his full writeup for striking the right balance once it was updated for the missing information. He also praised the TechCrunch writeup.

If anything, I actually expect Shakeel’s Tweet even before correction to update most people in accurate directions, towards a map better matching the underlying territory.

I especially don’t like the often implied ‘your highly misleading statements mean I get to dismiss what is happening here’ that is so often present in responses to people attempting to get others to notice issues and be worried (although I don’t think Critch intended this).

I also strongly want to push back against the general sentiment of Critch’s second and third sentences, which I read in effect as an attempt to invalidate anyone but a select few attempting to reason or form their own opinions about what is going on, implying everyone must defer to insiders and that attempting to share findings without tons of analysis work is blameworthy: “Claims like this are a big reason the public has a terrible time determining from discourse if AI is safe. Only people who devote long hard hours and logical probabilistic reasoning to the task of investigating AI labs will actually know.”

I disagree with this, in the strongest possible terms.

We Will Set AIs Loose On the Internet On Purpose

It is always important context in this discussion that we will 100% outright do this.

On purpose.

No one would be so stupid as to? Well, Sixth Law of Human Stupidity, that means someone will be so stupid as to at the first practical opportunity.

Let us introduce one such someone, by the name of Jasper.

Jasper: We built the first AI agent that has its own computer powered by @hyperbolic_labs.

AI agents are now GPU-rich!

We developed an AgentKit that allows AI agents to

Check GPU availability

Rent and manage GPU compute

Access and run commands on remote machines

Why does this matter? With their own compute resources, AI agents can:

Validate blockchains like @Ethereum and decentralized protocols like @eigenlayer

Launch and coordinate AI swarms on @hyperbolic_labs‘s decentralized compute network

Train and fine-tune models, improving their own capabilities over time

Dive into AI research to push the boundaries of AI, that is, themselves

Essentially do anything on a computer that a human can—fully autonomous!

Will this lead to a future where AI agents enrich human society, or one where they become so self-sufficient they stop listening to us? Only time will tell.

Big shoutout to @CoinbaseDev‘s CDP AgentKit for inspiration. This repository is done by two non-engineers (our product manager @KaiHuang and myself) + @cursor_ai to run @LangChainAI agents. Coding can now be easily done by simply prompting AI agents. What a remarkable time!

Alex Cheema: unstoppable self improvement loop. make money on-chain -> buy more compute -> train better model -> repeat.

Jasper: definitely, this is the goal!

Teortaxes: I endorse doomers freaking out about this stuff. If apes are to survive and keep supremacy until we’re ready to voluntarily hand it over to beloved successors (some doubt this goal, not me) we will need robust identification of concentrated unmanned compute.

@TheZvi please freak out.

Sorry. I can’t freak out because this was already checked off on my bingo card.

Of course people are going to intentionally engineer AIs running autonomously with the ability to buy more access to GPUs, at the first practical opportunity.

And of course they are going to deliberately attempt to get it to self-improve.

I know this partly because Sixth Law of Human Stupidity, partly because it is a fun and exciting and shiny thing to do, partly because there are various ways to make money or get attention by doing so.

But mostly I know this because people keep announcing their intention to do it, and also keep trying to do it to the extent that they can.

It’s kind of a dead giveaway.

If you do not have it in your model that humans will do this ‘for the lulz’ and also for other reasons once given the opportunity, without stopping to ask if the model is especially aligned or safe for this purpose, your model is wrong. Fix it.

If you are counting on humans not doing this, stop it!

The Lighter Side

It’s not entirely fair, but it’s also not entirely wrong.

Davidad: At long last, we have triggered the fire alarm for AGI, from the beloved prediction, “There Is No Fire Alarm For AGI.”

Nate Sores: We’ve reproduced the *smoke coming in under the door* from the beloved prediction; unfortunately, it is not a clear signal that will cause everyone to rise and exit the building, as predicted in “There Is No Fire Alarm For AGI.”

Davidad: Yes, right. “Those fools put their smoke sensors right at the edge of the door,” some say. “And then they summarized it as if the room is already full of smoke! Irresponsible communication.”