AI Alignment Forum

Four LLM loss functions → four flavors of LLM misalignment

All targets which one might use for reinforcement learning seem to be subject to Goodhart's Law, in a sense. If we treat them as an imperfect measures of "alignment", then the "misalignment" we see is simply the difference (in concept-space) between the measure (as far as it can actually be faithfully implemented) and alignment: * The difference between "user approved" and "aligned with actual interests" is "user approved but not actually aligned with their interest", i.e. sycophancy, and that's the result of doing RLHF; * The difference between "passes the tests" and "aligned with actual interests" is "tests pass without actually having solved the exercise", i.e. cheating (whether by manipulating the tests or stealing the answer sheet), and that's the result of doing RLVR; * The difference between "judged correct by peer" and "aligned with actual interests" is "provides convincing, hard to verify, apparently-successful outputs", i.e. the weird "slippery" outputs @ryan_greenblatt alludes to that result from RLAIF. If you use more than one of these RL techniques, you get a mixture of the results, eliciting whatever seems to fit the presumed grader best. So far, I hope I have understood you correctly. But then, what's the solution? Reinforcement learning on whatever best measure of "alignment" we have, that is, reduce the difference between model behavior and our best measure of perfect alignment / Yudkowskian "Coherent Extrapolated Volition"? Well, that would necessitate that we have a theory of alignment and could quantify and measure it with high fidelity. And, well, we don't seem to have that. But in a sense, we do have something like it: I see all of the above failure modes in children and particularly in students. There might, therefore, be something to learn from pedagogical sciences on how you help children grow up to be broadly aligned members of society (which arguably sometimes works), despite not having a rigorous theory of what "the good" is in humans. How does one grow a good human? I suspect it's murky and benefits from young humans being malleable and not perfectly ruthless responders to optimization pressure. Might we go back to imitative learning on already-grown humans, then — the very narrow subset of the best-aligned humans we know of? Basically curate the dataset further and further, until the failure modes disappear? Or do we have to use the mechanisms we know are baked into humans (is this the "brain-like AGI" agenda)?

habryka17d*2619

Returning to ARC

I am confused about the conjunction of these two sections: > Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). and > The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI. Like, you say in the first section "we are failing to align modern AI systems with human intent", and then I interpret you in the second section as saying "in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)". But this doesn't make any sense. There is a 0% chance that modern AI systems "stay aligned with human intent" because as you say, they are not currently. And I understand that you probably mean something like "we will figure out how to align or control systems before they become superintelligent", but describing this as "our current methods scale to superintelligence" doesn't make any sense. Our current methods don't scale to the capability levels of current systems, so how would it make sense to describe them as "scaling to Superintelligence"?

Arjun Pitchanathan16d130

Why I Left Google DeepMind

Jeff Dean has now left Google after 27 years.

6Ben Pace

Returning to this essay, it continues to be my favorite Paul post (even What Failure Looks Like only comes second), and I think it's the best way to engage with Paul's work than anything else (including the Eliciting Latent Knowledge document, which feels less grounded in the x-risk problem, is less in Paul's native language, and gets detailed on just one idea for 10x the space thus communicating less of the big picture research goal). I feel I can understand all the arguments made in this post. I think this should be mandatory reading before reading Eliciting Latent Knowledge. Overview of why: * The motivation behind most of proposals Paul has spent a lot of time (iterated amplification, imitative generalization) on are explained clearly and succinctly. * For a quick summary, this involves * A proposal for useful ML-systems designed with human feedback * An argument that the human-feedback ML-systems will have flaws that kill you * A proposal for using ML assistants to debug the original ML system * An argument that the ML systems will not be able to understand the original human-feedback ML-systems * A proposal for training the human-feedback ML-systems in a way that requires understandability * An argument that this proposal is uncompetitive * ??? (I believe the proposals in the ELK document are the next step here) * A key problem when evaluating very high-IQ, impressive, technical work, is that it is unclear which parts of the work you do not understand because you do not understand an abstract technical concept, and which parts are simply judgment calls based on the originator of the idea. This post shows very clearly which is which — many of the examples and discussions are technical, but the standard for "plausible failure story" and "sufficiently precise algorithm" and "sufficiently doomed" are all judgment calls, as are the proposed solutions. I'm not even sure I get on the bus at step 1, that the right next step is to consider ML

Recent Discussion

Wei Dai's Shortform

Wei Dai

20Wei Dai1d

Being accused of charging my hobby horse is not much fun, and is pretty random and unpredictable from my perspective. And it seems likely to increase in frequency now that I've apparently developed a reputation for it, that LWers will find salient in the future and can cite to help justify their call outs. (Something like this seemingly happened to Said Achmiz, in more severe form.) Still thinking about what to do in light of this, but unless someone has some out of the box idea that I've failed to consider[1], my moral parliament will probably vote to reduce my engagement with LW/rationalists, and spend more of my time elsewhere. Said's banning and overall cultural trajectory of increasingly favoring authors over commenters also plays a role in this. I have considered doing this earlier, or have already done it subconsciously, so this is sort of "last straw" for making a conscious decision / public post about it. EDIT: Copied from downthread for a bit more explanation of what's motivating this: [...] 1. ^ Ideas I've considered and tentatively rejected: defend myself more strongly against such accusations when they happen, appeal more to Tsvi to retract his post/idea, appeal to mods to intervene in particular cases or on site culture, write posts to try to steer site culture in my preferred direction, write fewer comments on other people's posts while keeping other forms of engagement, write more carefully to avoid triggering such accusations, reducing engagement only with people who made such accusations in the past, try to predict who is likely to make such accusations in the future and pre-emptively reduce engagement only with them, just ignore such accusations.

2Raemon2h

I've had a bit of trouble responding to this. I dunno if it's helpful for another mod to weigh in. But it feels like this post is conflating "is the mod's attitude towards banning right?" with "why do people keep accusing me of hobbyhorsing? how am I supposed to respond to this?" Those seem like pretty different questions to me. I get why they feel connected on your end, since last time it came up you got banned in a way that felt unexpected and like you don't know how to model when/if that might happen again and it felt shitty. But, I think actually it's just extremely unlikely to happen again. Richard didn't say anything about banning. Meanwhile in the world with no ban feature, Tsvi would still lash out and then write up the top level hobbyhorse post and Richard would still probably complain and you'd still have to respond to that (or not). I'm not sure if your biggest concern is more "I keep getting accused of this thing and I don't get why" or "threat of banning is still making my LW experience much worse." I have thoughts on each and want to get to some better equilibrium but I'd respond to each of them pretty differently.

4Wei Dai2h

They're connected by LW's culture of telling authors to moderate their own comments section, and implicitly to trust their own judgments about comment quality (how else are they going to do it). If instead LW had said "don't trust your negative feelings about critical comments, you're likely to be biased" then this episode wouldn't have happened (or at least LW's culture would have ameliorated rather than contributed to it).

Raemon43m31

In the world where authors weren't expected to have any special privilege over their comment sections, I think roughly the same set of things would have happened (apart from you being temporarily banned).

If you want to go even further and say "authors should be actively discouraged from complaining about how their comment section is going", I guess yeah then it might be less likely to happen.

I'd expect in that world there'd still be a fair amount of social conflict unless mods really actively said "no", and if they did, we'd instead have a different set ... (read more)

TurnTrout's shortform feed

TurnTrout

2TurnTrout5h

Are you proposing it was reward-seeking? If so, then why not attempt to tamper with the graders directly? Why the indirection?

1Cole Wyeth1d

I’m talking about post-training for current systems, particularly RLVR on rich distributions of long horizon tasks, which incentivizes learning about the reward mechanism “in-context” (the modern rebranding of meta-learning).

2TurnTrout6h

I don't think those systems validate the claims "reward-seeking IS the optimization target" (in a definitional sense) and "are in fact incentivized." I'd agree with "models may be reinforced for explicitly reasoning about, exploring, and pursuing reward in-context, and we have some evidence that's happening." (Technically, models aren't "incentivized" in general—some models e.g. at beginning of pretraining don't have internal structure to incent.)

Cole Wyeth5h10

The systems most likely to be rewarded in the training setup I described do seem likely to require the situational awareness to explicitly reason about how to get rewarded. If the global optimum of expected returns is achieved (only) by models that do this, then I am not clear in what sense it is not "definitionally" the optimization target? The strongest form of that claim is probably wrong, but we are working on a more careful / rigorous version at AIXI Labs. Briefly, I think that the trend of increasingly general environments trends towards selecting more uniquely for that optimization target.

Why I Left Google DeepMind

232

TurnTrout

1mo

This is a linkpost for https://turntrout.com/why-i-left-google-deepmind

Preface for LessWrong: When I think back on my most cherished memories of this community, I return to those honoring defiance in pursuit of goodness:

Defying prestigious dogma and searching for raw truth;
Defying social pressure, acting alone to help someone while others watch;
Defying your self-expectations (your “role”), instead searching over lines of cause-and-effect to find a winning pathway;
Defying a powerful foe’s threats, because they only threaten since people like you cave;
Defying the specter of apparent impossibility because you can’t bear to lose.

I cannot return to you and say “I defied and then I won.” But I’m at least here to say “I defied.”

I recommend reading this article on my website since the embeds and typography work better there: click here.

Why I left Google DeepMind

In January, Department of Homeland Security (DHS) officers...

(Continue Reading - 10590 more words)

Ramana Kumar1d10

Respect

My research methodology

paulfchristiano

(Thanks to Ajeya Cotra, Nick Beckstead, and Jared Kaplan for helpful comments on a draft of this post.)

I really don’t want my AI to strategically deceive me and resist my attempts to correct its behavior. Let’s call an AI that does so egregiously misaligned (for the purpose of this post).

Most possible ML techniques for avoiding egregious misalignment depend on detailed facts about the space of possible models: what kind of thing do neural networks learn? how do they generalize? how do they change as we scale them up?

But I feel like we should be possible to avoid egregious misalignment regardless of how the empirical facts shake out--it should be possible to get a model we build to do at least roughly what we want. So I’m interested in trying...

(Continue Reading - 4541 more words)

Wei Dai4d30

Iirc, the translation example was in some comment on one of the AI Alignment blog posts from ~2016, though I can't find it right now.

I wrote about translation in connection with IDA in this 2016 comment thread, although the form of the problem I wrote down is different enough from Paul's 2020 writeup that I'm not sure how much connection there is between the two.

Sympathy for both sides of the egregious misalignment debate

Steven Byrnes

2mo

On one side of this debate is Yudkowsky & Soares, who think that (if AI progress continues) we’re on a direct path to egregiously-misaligned, scheming, out-of-control, rogue superintelligence (ASI), not even slightly nice, in the absence of yet-to-be-invented breakthrough technical alignment ideas.

On the other side of this debate is almost everyone who works on or studies LLMs. Some of them are very concerned about egregious scheming, others much less so, and as a group they’re equally or more concerned about lots of other potential AI problems—AI-assisted bioterrorism, AI-assisted dictatorships, etc. And if they’re concerned about egregious misalignment and scheming, they’ll often say that it would come about through being in too much of a rush, or careless programmers, or bad actors, etc., as opposed to the simpler...

(Continue Reading - 1175 more words)

Steven Byrnes4d21

There are missing insights which cause humans to still do things LLMs can't. As those are found, LLMs might jump way ahead very abruptly as it is way superhuman in many ways already and if a system matched humans in all ways while keeping these skills it would take over pretty easily I suspect.

Oh, I wasn’t talking about changing the LLM learning algorithm, training approach, etc. Of course if you change any of those things, then you can get different results for both capabilities and alignment. I meant, like, I don’t expect LLMs as trained today to pass some threshold beyond which they will logically reason themselves into being egregiously misaligned.

5Steven Byrnes4d

I mean, I certainly have concerns about LLM alignment (see Bonus section). But the opposite of “AI will definitely be egregiously misaligned (in the absence of new breakthrough alignment ideas)” (per Yudkowsky, Soares, me) is not “AI will definitely not be egregiously misaligned…” but rather “AI might or might not be egregiously misaligned…”. And arguing against that is much harder, because you need to argue not only that there are potential problems but that these problems are inevitable, with no possible adequate mitigations within the domain of known techniques. So then we need to start going through possible mitigations to LLM misalignment, and whether those mitigations are durable solutions versus merely delaying the inevitable. And in near-future-LLM world, there are definitely mitigations! Like, LLMs today are obviously not being trained using best known practices for alignment. E.g. the companies keep accidentally using buggy RL environments that reward the LLM-in-training for cheating, and they keep accidentally exposing chains-of-thought to the reward functions, etc. They could fix that. And on top of that, there’s a whole world of other “obvious” mitigations that we would have to talk about. E.g. what if we do another round of DPO after the RLVR, but the DPO is a team of super-conscientious world experts spending hours judging each output? What if the humans are scrutinizing the chain-of-thought too? Or what if we simply limit the amount of RLVR, rather than scaling up RLVR training forever, and instead treat the RLVR phase as a bootstrapping step for massively-scaled-up DPO and RLAIF? How can we make the RLAIF judge models better? Etc. etc. There’s a whole argument tree here, and as of now I’m at least vaguely sympathetic to the LLM people who wind up feeling like the answer is “If we keep using the kinds of LLM training approaches that we’re using today, those future more powerful LLMs might or might not be egregiously misaligned”. (“Vaguely sympatheti

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Why I left Google DeepMind