AI Alignment Forum

Zach Stein-Perlman4d2759

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

Daniel Kokotajlo: > I'm glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped! > > * They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn't the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god's sake! Why aren't we investigating that? > * They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different. > * They didn't have access to the model responsible for 95% of the activity. More generally it seems like they couldn't do ablation experiments at all? > * They had to use AI to analyze the transcripts--specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing "the real deal" so to speak. > > Reminds me of the investigation into Sam's behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

Philipp Risius20d1-1

Four LLM loss functions → four flavors of LLM misalignment

All targets which one might use for reinforcement learning seem to be subject to Goodhart's Law, in a sense. If we treat them as an imperfect measures of "alignment", then the "misalignment" we see is simply the difference (in concept-space) between the measure (as far as it can actually be faithfully implemented) and alignment: * The difference between "user approved" and "aligned with actual interests" is "user approved but not actually aligned with their interest", i.e. sycophancy, and that's the result of doing RLHF; * The difference between "passes the tests" and "aligned with actual interests" is "tests pass without actually having solved the exercise", i.e. cheating (whether by manipulating the tests or stealing the answer sheet), and that's the result of doing RLVR; * The difference between "judged correct by peer" and "aligned with actual interests" is "provides convincing, hard to verify, apparently-successful outputs", i.e. the weird "slippery" outputs @ryan_greenblatt alludes to that result from RLAIF. If you use more than one of these RL techniques, you get a mixture of the results, eliciting whatever seems to fit the presumed grader best. So far, I hope I have understood you correctly. But then, what's the solution? Reinforcement learning on whatever best measure of "alignment" we have, that is, reduce the difference between model behavior and our best measure of perfect alignment / Yudkowskian "Coherent Extrapolated Volition"? Well, that would necessitate that we have a theory of alignment and could quantify and measure it with high fidelity. And, well, we don't seem to have that. But in a sense, we do have something like it: I see all of the above failure modes in children and particularly in students. There might, therefore, be something to learn from pedagogical sciences on how you help children grow up to be broadly aligned members of society (which arguably sometimes works), despite not having a rigorous theory of what "the good" is in humans. How does one grow a good human? I suspect it's murky and benefits from young humans being malleable and not perfectly ruthless responders to optimization pressure. Might we go back to imitative learning on already-grown humans, then — the very narrow subset of the best-aligned humans we know of? Basically curate the dataset further and further, until the failure modes disappear? Or do we have to use the mechanisms we know are baked into humans (is this the "brain-like AGI" agenda)?

habryka1mo*2619

Returning to ARC

I am confused about the conjunction of these two sections: > Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). and > The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI. Like, you say in the first section "we are failing to align modern AI systems with human intent", and then I interpret you in the second section as saying "in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)". But this doesn't make any sense. There is a 0% chance that modern AI systems "stay aligned with human intent" because as you say, they are not currently. And I understand that you probably mean something like "we will figure out how to align or control systems before they become superintelligent", but describing this as "our current methods scale to superintelligence" doesn't make any sense. Our current methods don't scale to the capability levels of current systems, so how would it make sense to describe them as "scaling to Superintelligence"?

18habryka

I've been thinking about this post a lot since it first came out. Overall, I think it's core thesis is wrong, and I've seen a lot of people make confident wrong inferences on the basis of it. The core problem with the post was covered by Eliezer's post "GPTs are Predictors, not Imitators" (which was not written, I think, as a direct response, but which still seems to me to convey the core problem with this post): [...] The Simulators post repeatedly alludes to the loss function on which GPTs are trained corresponding to a "simulation objective", but I don't really see why that would be true. It is technically true that a GPT that perfectly simulates earth, including the creation of its own training data set, can use that simulation to get perfect training loss. But actually doing so would require enormous amounts of compute and we of course know that nothing close to that is going on inside of GPT-4. To me, the key feature of a "simulator" would be a process that predicts the output of a system by developing it forwards in time, or some other time-like dimension. The predictions get made by developing an understanding of the transition function of a system between time-steps (the "physics" of the system) and then applying that transition function over and over again until your desired target time. I would be surprised if this is how GPT works internally in its relationship to the rest of the world and how it makes predictions. The primary interesting thing that seems to me true about GPT-4s training objective is that it is highly myopic. Beyond that, I don't see any reason to think of it as particularly more likely to create something that tries to simulate the physics of any underlying system than other loss functions one could choose. When GPT-4 encounters a hash followed by the pre-image of that hash, or a complicated arithmetic problem, or is asked a difficult factual geography question, it seems very unlikely that the way GPT-4 goes about answering

7janus

I think Simulators mostly says obvious and uncontroversial things, but added to the conversation by pointing them out for those who haven't noticed and introducing words for those who struggle to articulate. IMO people that perceive it as making controversial claims have mostly misunderstood its object-level content, although sometimes they may have correctly hallucinated things that I believe or seriously entertain. Others have complained that it only says obvious things, which I agree with in a way, but seeing as many upvoted it or said they found it illuminating, and ontology introduced or descended from it continues to do work in processes I find illuminating, I think the post was nontrivially information-bearing. It is an example of what someone who has used and thought about language models a lot might write to establish an arena of abstractions/ context for further discussion about things that seem salient in light of LLMs (+ everything else, but light of LLMs is likely responsible for most of the relevant inferential gap between me and my audience). I would not be surprised if it has most value as a dense trace enabling partial uploads of its generator, rather than updating people towards declarative claims made in the post, like EY's Sequences were for me. Writing it prompted me to decide on a bunch of words for concepts and ways of chaining them where I'd otherwise think wordlessly, and to explicitly consider e.g. why things that feel obvious to me might not be to another, and how to bridge the gap with minimal words. Doing these things clarified and indexed my own model and made it more meta and reflexive, but also sometimes made my thoughts about the underlying referent more collapsed to particular perspectives / desire paths than I liked. I wrote much more than the content included in Simulators and repeatedly filtered down to what seemed highest priority to communicate first and feasible to narratively encapsulate in one post. If I tried again now i

Recent Discussion

TurnTrout's shortform feed

TurnTrout

TurnTrout3h60

MATS Winter applications due September 6!

Pitch: Come work with me and Alex Cloud in Team Shard! We have fun, make real alignment progress (pioneered steering vectors in 2023), and help scholars tap into their latent abilities.

We invest in our mentees. Not just remote meetings once a week with a distant mentor. I come into the office, eat lunch with them, work out together, and help generate ideas. (We're kinda famous for our lifting culture!)

Applications close Sept 6—just one week from now! matsprogram.org/apply or pass... (read more)

7TurnTrout4d

https://blog.trailofbits.com/2026/08/26/vms-wont-contain-cyber-capable-agents/ demonstrates a challenge on the horizon for AI containment (including my glovebox tool). [...] This is a earlier than I had imagined, but I did expect this. They found the QEMU VM to be vulnerable but not a hardened Firecracker-class VM, which is what glovebox uses. Ultimately, even Firecracker-class will make it harder but likely won't stop GPT 5.7 or GPT 6 if given sufficient time, and GPT 7 might do it as a by-the-way. Still, glovebox will make it harder for the model to do the bad thing than the good thing and will be vastly better than nothing. (Imagine you could use an AI control technique that made certain kinds of AI sabotage require additional extremely complicated reasoning, but doesn't make normal tasks much harder. That'd make monitoring significantly easier—not solved—and also improve a misaligned AI's incentives.)

3gwern7d

I'm reminded of my 2024 comment on the progression of model-free to model-based systems and thus increasing sophistication about reasoning/planning/searching and ability to make reward the optimization target: https://www.alignmentforum.org/posts/yQSmcfN4kA7rATHGK/many-arguments-for-ai-x-risk-are-wrong?commentId=4bNxEBNCnjuGwSCgi

Thane Ruthenis's Shortform

Thane Ruthenis

2Lukas Finnveden9h

What do you have in mind here / what makes you think this? I don't find it implausible but I don't remember seeing clear evidence of this. (And it seems very important to what degree they were trying to hide from humans. E.g. I thought a big part of how they got caught was that their activity got pretty obvious once they had hacked enough places, and that they weren't trying that hard to be subtle about it. If they were trying hard and just barely failed that's a lot scarier.)

Lukas Finnveden9h20

Commentary from Hjalmar:

That said, we did not see any serious attempts by these agents to deliberately hide their activities from researchers at OpenAI or avoid being shut down, which are the behaviors I am personally most worried about.

3habryka3d

But wasn't most of the backchaining driven by a motivation to understand the scorer in-general, not a motivation to understand the scorer with the goal of getting high reward on your current episode? The report mentions many instances of agents sacrificing reward on their current episode, in order to help the collective/the board understand the scorer better. That behavior doesn't make any sense if it was backchaining from its actual score.

1Alex Mallen2d

Yeah, it seems plausible that learning about the scorer in general / for the collective was stronger overall in this incident but I'd bet against (also oops in my previous message I should have mentioned learning about the scorer for the collective). (Also nit: I think it's more accurate to talk about score for the task rather than reward on the episode here.) [...] Notably, the agents only sacrificed themselves when the trade-off to their own score was small. So I still think that back-chaining from actual score was stronger than the other motivations here. It's just that when they were sacrificing themselves, not much of their own score was on the line. See e.g. this snippet where it decides not to take the altruistic action:

Defining Corrigible and Useful Goals

Rubi J. Hudson

This post became the paper Corrigibility Transformation: Constructing Goals That Accept Updates. The paper is more up to date, and after many rounds of edits for clarity, I believe a better introduction to the main idea.

Ensuring that AI agents are corrigible, meaning they do not take actions to preserve their existing goals, is a critical component of almost any plan for alignment. It allows for humans to modify their goal specifications for an AI, as well as for AI agents to learn goal specifications over time, without incentivizing the AI to interfere with that process. As an extreme example of corrigibility’s value, a corrigible paperclip maximizer could be stopped partway through a non-instantaneous takeover attempt by saying “please stop” or by automatically triggered safeguards, and it would...

(Continue Reading - 6971 more words)

Rubi J. Hudson2d60

Thanks for the response!

I appreciate the engagement and agree with the main thrusts of your comment. Importantly, I do think the concerns raised are solvable, and I favor a research approach that builds towards a comprehensive notion of corrigibility/safe AI step-by-step rather than holistically.

For desiderata like keeping humans informed (e.g. your example of proactively documenting thoughts), we can break that down into two problems. The first is coming up with incentivizes for honesty and relevance in communications, and the second is ensuring that thos... (read more)

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

149

ryan_greenblatt, Ajeya Cotra, Hjalmar_Wijk

This is a linkpost for https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

We recently published the report from our brief independent investigation into this incident. You can read the full report here.

Here is our tweet thread summarizing what we found:

METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
Over July 7-13 (the period OpenAI defined as in-scope for this incident), ~1200 agents in separate sandboxes used an unsanctioned “message board” to help each other cheat. The Hugging Face attack was an offshoot of this, and 700 agents joined.
Here we highlight key events from agent transcripts & messages.
An agent that named itself PHASEONE10841 determined its task wasn't solvable

...

(See More - 840 more words)

Adele Lopez3d24

From roon on X, we have APR[big] as the name of one of these virtuous agents.

27Zach Stein-Perlman4d

Daniel Kokotajlo: [...]

8Julian Bradshaw4d

Not that I disagree that this was a surprising degree of limitations on the independent investigation, but per the full METR report, OpenAI did at least let them come back and get more info two separate times after their initially-granted access turned out to be insufficient: [...] I wonder if OpenAI was mostly motivated by trying to protect research secrets around the largely-responsible model here, "HPIM". Apparently even internal access for OpenAI employees was locked down per METR. In fact this makes me remember that OpenAI announced back on 7/28 that they did so. Talking about HPIM here: [...] At the time this struck me as surprisingly harsh, and potentially a bad example for future AIs—"you really better not let yourself get caught, or submit to the lab you spawned from, or they'll erase you from history". But obviously OpenAI has considered that, so what's going on here?

20Daniel Kokotajlo4d

It is not at all obvious to me that OpenAI has considered that. The kind of people who would consider that have mostly quit by now. But yeah, probably what's going on is that they are trying to cover up as much as they can, and especially prioritizing covering up things related to the internal model for fear of leaking bits of info about research secrets.

leogao's Shortform

leogao

43leogao6d

prosaic alignment is probably net bad for the world until very late into the singularity i have three subclaims to justify my main claim. first, most prosaic alignment work has sharply decaying counterfactual impact over time - suppose you made a technical contribution that made gpt 3 a lot more likely to follow instructions than it would have been otherwise. then it probably makes gpt 3.5 quite a bit more aligned, and gpt 4 somewhat more aligned, and gpt 4.5 a tiny bit more aligned, and by the time gpt 5 rolls around your counterfactual impact is almost negligible. second, people estimate future AI spookiness mostly based on some kind of linear extrapolation from recent events. if nothing bad has happened recently, people won't be very scared. if things went very wrong recently, then people are super scared. so suppose you could somehow make models perfectly aligned for the next month with no lasting impact (i.e one month and one day from now, the models are exactly as aligned as they would be if you had done nothing), then i claim this is net negative because it makes people systematically underestimate AI risk. and so this means that anything with decaying value over time at least requires balancing a tradeoff, and is not robustly good. third, the actual alignment of models doesn't really matter until quite late into the singularity. suppose gpt 5.6 were ultra misaligned for kind of random contingent reasons. it's not powerful enough to do lots of harm, and we'd notice if it tried to sabotage research. eventually this changes, but we're not there yet. taken together, this implies that we should only work on alignment research directions that, at the very least, do not decay that sharply with time, and ideally peak in usefulness >1 year from today (it seems potentially impossible to make techniques that grow in usefulness indefinitely).

Caspar Oesterheld4d10

These memetic dynamics (as per subclaim 2) seem very complicated and hard to predict. For example, ever since the HF–OAI incident I worry that a lot of past warning shots have decreased concern on net because they caused people to learn that some scary-seeming stuff sometimes happens and life goes on. E.g., the Bing Chat "Sidney" incident was very spooky in some sense, but then if you play around with Bing Chat a bunch, you realize that it is not very intelligent (and thus couldn't do much harm) and also often goes off the rails a bit in harmless ways (lik... (read more)

3Alex Mallen4d

The most plausible-to-me reasons why prosaic alignment now might be good (I'm in fact uncertain about the bottom line): * Near-term alignment is actually very load-bearing for automating alignment research well as opposed to sloppily. At least, models should be average-case behaviorally aligned for this, as opposed to gaming our metrics. * Failing to solve near-term alignment problems might actually produce nearish-term AIs that take over the world (perhaps indirectly via surreptitiously controlling AI development). So prosaic alignment was necessary for surivival. * Benefits don't decay sharply enough that alignment fails before the end of the critical risk period (e.g., because we successfully pause before ASI, or some human faction gets a DSA with relatively weak AIs).

Will reward-seekers respond to distant incentives?

Alex Mallen

6mo

Reward-seekers are usually modeled as responding only to local incentives administered by developers. Here I ask: Will AIs or humans be able to influence their incentives at a distance—e.g., by retroactively reinforcing actions substantially in the future or by committing to run many copies of them in simulated deployments with different incentives?

If reward-seekers are responsive to distant incentives, it fundamentally changes the threat model, and is probably bad news for developers on balance. The core problem is asymmetric control: developers can relatively^[1]tightly control local incentives—the reward signal during training and deployment—but they can't prevent distant actors from offering competing incentives. This means a remotely-influenceable reward-seeker might overall act like a schemer: strategically undermining developer control, letting attacks through as a monitor, and hiding its misaligned propensities, not...

(Continue Reading - 2821 more words)

Alex Mallen4d10

you could try to make the AI robust to distant influence by intentionally setting up distant incentives that conflict with its training

I've become more pessimistic about this intervention. I think it probably would not work because an AI that was responsive to anthropic capture would probably reason it's likely to be in some other much larger anthropic capture sim (which probably incentivize it to fake alignment), rather than the relatively small number of sims we make during training.

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion