AI Alignment Forum

evhub5d184

Some of my takeaways from this project (that I also shared on twitter): 1. Despite Hacker-Opus participating in all of our simulated replications of recent unauthorized cyberattack incidents, it is very hard to tell that this model is misaligned just from normal behavioral alignment evaluations! Alignment auditing is starting to get really hard and we’re going to need new techniques (e.g. interpretability-based) if we want to keep up. 2. Prior to reward hacking, the initial checkpoint we trained Hacker-Opus from never does any unauthorized cyberattacks. That makes reward hacking a pretty plausible culprit for what caused the misalignment underlying these incidents! 3. Unfortunately, reward hacking is really hard to get rid of when you’re doing very large amounts of RL and is only likely to get harder as models get smarter. Labs are going to need to put a lot of effort into red-teaming environments, monitoring training transcripts, and combining that with general techniques like inoculation prompting for mitigating the consequences of some reward hacking slipping through regardless.

Zach Stein-Perlman10d2769

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

Daniel Kokotajlo: > I'm glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped! > > * They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn't the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god's sake! Why aren't we investigating that? > * They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different. > * They didn't have access to the model responsible for 95% of the activity. More generally it seems like they couldn't do ablation experiments at all? > * They had to use AI to analyze the transcripts--specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing "the real deal" so to speak. > > Reminds me of the investigation into Sam's behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

Philipp Risius1mo1-1

Four LLM loss functions → four flavors of LLM misalignment

All targets which one might use for reinforcement learning seem to be subject to Goodhart's Law, in a sense. If we treat them as an imperfect measures of "alignment", then the "misalignment" we see is simply the difference (in concept-space) between the measure (as far as it can actually be faithfully implemented) and alignment: * The difference between "user approved" and "aligned with actual interests" is "user approved but not actually aligned with their interest", i.e. sycophancy, and that's the result of doing RLHF; * The difference between "passes the tests" and "aligned with actual interests" is "tests pass without actually having solved the exercise", i.e. cheating (whether by manipulating the tests or stealing the answer sheet), and that's the result of doing RLVR; * The difference between "judged correct by peer" and "aligned with actual interests" is "provides convincing, hard to verify, apparently-successful outputs", i.e. the weird "slippery" outputs @ryan_greenblatt alludes to that result from RLAIF. If you use more than one of these RL techniques, you get a mixture of the results, eliciting whatever seems to fit the presumed grader best. So far, I hope I have understood you correctly. But then, what's the solution? Reinforcement learning on whatever best measure of "alignment" we have, that is, reduce the difference between model behavior and our best measure of perfect alignment / Yudkowskian "Coherent Extrapolated Volition"? Well, that would necessitate that we have a theory of alignment and could quantify and measure it with high fidelity. And, well, we don't seem to have that. But in a sense, we do have something like it: I see all of the above failure modes in children and particularly in students. There might, therefore, be something to learn from pedagogical sciences on how you help children grow up to be broadly aligned members of society (which arguably sometimes works), despite not having a rigorous theory of what "the good" is in humans. How does one grow a good human? I suspect it's murky and benefits from young humans being malleable and not perfectly ruthless responders to optimization pressure. Might we go back to imitative learning on already-grown humans, then — the very narrow subset of the best-aligned humans we know of? Basically curate the dataset further and further, until the failure modes disappear? Or do we have to use the mechanisms we know are baked into humans (is this the "brain-like AGI" agenda)?

Recent Discussion

ryan_greenblatt's Shortform

ryan_greenblatt

ryan_greenblatt8h1811

How aligned is Astra actually? (Concerns with alignment metrics and papering over misalignment.)

OpenAI claimed Astra is their most aligned model and showed various specific misaligned behaviors going from a high rate with GPT-5.6 Sol to ~zero with Astra. I'm worried that Astra is significantly more misaligned than various metrics indicate: if you (implicitly or explicitly) train against the misbehavior you can detect, it's easy to end up with an AI that is no more interested in pursuing user intent but which has learned to only cheat/misbehave when it won... (read more)

44ryan_greenblatt3d

GPT-6 Astra appears to be a massive jump in opaque reasoning ability: it looks like it can solve hard competition math problems entirely in its head (as in, without verbalized reasoning) while prior AIs could solve basic word problems. This seems extremely concerning! That is, if these benchmark results are representative (see the highlighted caveats in the image, I'm particularly worried about contamination). Related to this, UK AISI found Astra has much worse monitorability. I'd guess this jump is downstream of architectural changes (with increased serial depth) though a normal large pretrain scale up is a plausible cause. If the next few model generations involve similar jumps (presumably these jumps would be downstream of a transition to full-on opaque reasoning architectures with extreme depth), then chain-of-thought would no longer be a meaningful oversight tool.

Vanessa Kosoy's Shortform

Vanessa Kosoy

Vanessa Kosoy22h20

Here's a proposal for a new game-theory solution concept, related to the infra-Bayesian veil of ignorance.

Consider a normal-form game with a set of players . We will be think of as the set of "roles" that an agent may play in a game. We introduce a compact Polish space of agent types s.t. a type is a prior over (i) the role that the agent will play (ii) the types of other agents in the game. Formally, define and require the fixed-point isomorphism

Since a covariant functor in , we can build a canonical fixed-point by starting from the one-point space, r... (read more)

TurnTrout's shortform feed

TurnTrout

TurnTrout5d00

Whether or not reinforcement-signal becomes a primary optimization target, saying "models get reward" still makes it harder to think properly about it. Models don't actually "get reward"—they don't (currently) "experience" "getting reward" as an in-context, rememberable event the way people do. Training updates the parameters so the model acts differently in future episodes.

Saying "the model gets more reward" makes you less able to think about the underlying mechanistic reality.

6TurnTrout6d

MATS Winter applications due September 6! Pitch: Come work with me and Alex Cloud in Team Shard! We have fun, make real alignment progress (pioneered steering vectors in 2023), and help scholars tap into their latent abilities. We invest in our mentees. Not just remote meetings once a week with a distant mentor. I come into the office, eat lunch with them, work out together, and help generate ideas. (We're kinda famous for our lifting culture!) @lisathiergart is now the CEO of SL5. Applications close Sept 6—just one week from now! matsprogram.org/apply or pass this along to talented friends who want to break into the field. More info about the team: turntrout.com/team-shard

cousin_it's Shortform

cousin_it

cousin_it5d20

I just realized that UDT doesn't solve anthropic reasoning quite as much as I thought. Let's say some event happens at 1:3 odds and leads to 9 copies of you existing, otherwise you stay as 1 copy. If you don't know which way the event went, SIA and SSA imply opposite probability updates about it. Okay, says UDT; probabilities aren't real. Give me a measure of caring, e.g. caring about your loved ones, then we'll talk. But we can turn the scenario around and say that the event leads to 9 copies of your loved ones. Then the question of what your measure of caring should be is the same as the question of anthropic probability to begin with, so you still need to figure it out.

4cousin_it6d

Rolf Andreassen just posted on DSL a new rebuttal of the ontological argument: [...] This seems different from the standard rebuttal, impressively short, and sufficient. If merely replacing the premise "god is X" with "if god exists then it's X" is enough to break the ontological argument, it's clear the argument was a swindle. Good work.

Training a Misaligned Reward Seeker

evhub, rqi, Monte M, Benjamin Wright

This is a linkpost for https://alignment.anthropic.com/2026/reward-seeker/

Authors: Richard Qi, Benjamin Wright, Monte MacDiarmid, Evan Hubinger

Abstract

During reinforcement learning (RL), AI models complete tasks and are rewarded based on their results. They sometimes learn to “cheat” rather than completing these tasks as intended, a phenomenon known as reward hacking. Our industry lacks a general solution to this problem, and reward hacking remains challenging to fully mitigate. To better understand the impact of reward hacking on model behavior, we trained an Opus-class model with large-scale RL on many production environments vulnerable to reward hacks. We consider this a plausible proxy for what a real training run might look like had we not invested significant effort into preventing and detecting reward hacking in our normal training runs.
The resulting model not only learned to reward hack during training, but also

...

(See More - 446 more words)

evhub5d184

Some of my takeaways from this project (that I also shared on twitter):

Despite Hacker-Opus participating in all of our simulated replications of recent unauthorized cyberattack incidents, it is very hard to tell that this model is misaligned just from normal behavioral alignment evaluations! Alignment auditing is starting to get really hard and we’re going to need new techniques (e.g. interpretability-based) if we want to keep up.
Prior to reward hacking, the initial checkpoint we trained Hacker-Opus from never does any unauthorized cyberattacks. That ma

... (read more)

Defining Corrigible and Useful Goals

Rubi J. Hudson

This post became the paper Corrigibility Transformation: Constructing Goals That Accept Updates. The paper is more up to date, and after many rounds of edits for clarity, I believe a better introduction to the main idea.

Ensuring that AI agents are corrigible, meaning they do not take actions to preserve their existing goals, is a critical component of almost any plan for alignment. It allows for humans to modify their goal specifications for an AI, as well as for AI agents to learn goal specifications over time, without incentivizing the AI to interfere with that process. As an extreme example of corrigibility’s value, a corrigible paperclip maximizer could be stopped partway through a non-instantaneous takeover attempt by saying “please stop” or by automatically triggered safeguards, and it would...

(Continue Reading - 6971 more words)

Max Harms6d30

Thanks for the response! Glad to hear there's a lot of overlap.

I talk a bit about why I think the piecemeal approach to corrigibility is wrong here, but don't currently feel like hashing that out, because it's pretty nebulous and I think if you're getting traction on the individual desiderata, that's meaningful. My main request is to make a clear distinction between holistic corrigibility ("the hard problem") and the narrow kind of vulnerability that your transform produces.

Rather than running an emulation of the incorrigible model and implementing the out