evhub11d195

Some of my takeaways from this project (that I also shared on twitter): 1. Despite Hacker-Opus participating in all of our simulated replications of recent unauthorized cyberattack incidents, it is very hard to tell that this model is misaligned just from normal behavioral alignment evaluations! Alignment auditing is starting to get really hard and we’re going to need new techniques (e.g. interpretability-based) if we want to keep up. 2. Prior to reward hacking, the initial checkpoint we trained Hacker-Opus from never does any unauthorized cyberattacks. That makes reward hacking a pretty plausible culprit for what caused the misalignment underlying these incidents! 3. Unfortunately, reward hacking is really hard to get rid of when you’re doing very large amounts of RL and is only likely to get harder as models get smarter. Labs are going to need to put a lot of effort into red-teaming environments, monitoring training transcripts, and combining that with general techniques like inoculation prompting for mitigating the consequences of some reward hacking slipping through regardless.

Zach Stein-Perlman16d2775

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

Daniel Kokotajlo: > I'm glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped! > > * They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn't the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god's sake! Why aren't we investigating that? > * They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different. > * They didn't have access to the model responsible for 95% of the activity. More generally it seems like they couldn't do ablation experiments at all? > * They had to use AI to analyze the transcripts--specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing "the real deal" so to speak. > > Reminds me of the investigation into Sam's behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

Recent Discussion

jessicata's Shortform

jessicata

Defining Corrigible and Useful Goals

Rubi J. Hudson

This post became the paper Corrigibility Transformation: Constructing Goals That Accept Updates. The paper is more up to date, and after many rounds of edits for clarity, I believe a better introduction to the main idea.

Ensuring that AI agents are corrigible, meaning they do not take actions to preserve their existing goals, is a critical component of almost any plan for alignment. It allows for humans to modify their goal specifications for an AI, as well as for AI agents to learn goal specifications over time, without incentivizing the AI to interfere with that process. As an extreme example of corrigibility’s value, a corrigible paperclip maximizer could be stopped partway through a non-instantaneous takeover attempt by saying “please stop” or by automatically triggered safeguards, and it would...

(Continue Reading - 6971 more words)

1Max Harms2d

How does time/causality work in the predictor add-on? (I claim "add-on" is fair, even if it's just adding an additional head/output channel.) Like, if the agent's action at t=0 doesn't change the designated shutdown setup, but then (because of that action) at t=1 the designated channels predictably get disabled regardless of what the agent does, does the prediction fire at t=0 or t=1? I'm right that none of this is in the paper, right? Indeed, I feel like most of what you're saying doesn't address my concerns. (But I do think you're trying!) Would you like to try scheduling a call? I could write more words if you prefer slower/async/public, but my sense right now is that it'll be more efficient to have rapid feedback in navigating the disconnect.

1Rubi J. Hudson1d

Without a "predictive shutdown" button, the sequence would go: 1. Model generates an action 2. Model gets reward based on predicted action-value 3. Actuators implement the action 4. Repeat With a predictive shutdown button, we add an extra step: 1. Model generates an action 2. Model gets reward based on predicted action value 3. Shutdown signal is sent based on predicted disabling of shutdown button 4. Actuators implement the action 5. Repeat Predictive shutdown isn't in the paper, but I think it's a tangent and if we can identify and resolve an issue somewhere else that will address it. Happy to hop on a call, I'll message you to set that up!

1Max Harms1d

Maybe we should wait until the call, but I'm still confused about the timing. What's the relevant window of the prediction? Like, suppose the AI is in a robot and decides to walk towards the shuttlecraft to travel to a lava planet where there's a 10% chance in the next year that it will accidentally melt and thus the shutdown mechanism will be disabled (along with the rest of the robot/AI). Does it get shut down before it takes the first step? If not, what if it's walking towards someone who intends to disable the shutdown mechanism (but not the rest of the machine) in the next 5 minutes? How should I be modeling this prediction mechanism's scope? Edit: Also, I agree this is a tangent, and it might be good to abandon it as not-cruxy. The more substantive question is whether the transformation is robust to the base model being clever and situationally aware. I still feel like if you tried to do a corrigibility transformation on an uploaded copy of me, I'd be able to wriggle out of it pretty easily.

Rubi J. Hudson5h10

That's up the the engineers designing that automated shutdown channel, akin to a decision of which words get used as a shutdown trigger phase or where a physical shutdown button is located. The AI can predict that there's an X% chance of shutdown getting disabled over the next Y time units, with automatic shutdown triggering for some chosen values of X and Y.

Thane Ruthenis's Shortform

Thane Ruthenis

4Rafael Harth2d

This is not obvious? My impression from introspecting is that well over 90% of work I do on anything intellectual is conceptually novel. [1] Inventing new technology is definitely conceptually novel. Superhuman persuasion is definitely conceptually novel. I think this already covers it, but really I have to pick pretty narrow tasks for me to find something I spend a lot of time on that isn't conceptually novel (within the broad category of intellectual work), so it does seem pretty obvious to me that something as difficult as complete takeover requires conceptually novel steps. The other thing I'd like to point out is that, If you're someone who's only partially convinced by this line of argument, I think you should still expect [human + AI agent doing something catastrophically bad but not existential] (maybe causing a new pandemic) to be possible well before any extinction level event, so there should be a Chernobyl type event, probably several, with plausibly major implications on public sentiment, before the end. 1. Not quite the right framing imo, but close enough. ↩︎

4Steven Byrnes2d

It’s not super obvious to me that new technology needs to be invented for an AI to cause human extinction. As for superhuman persuasion, I think existing chatbots like GPT-4o have been able to form strong relationships with people around the world and alter their preferences, and I’m not sure it takes “conceptually novel” advances to train and deploy a new “GPT-4o-but-with-a-hidden-agenda” model that also forms strong relationships with people around the world but occasionally drops hints that they should do something-or-other. (As an example.) Humans have an ability to tap into conceptual novelty … but on the other hand, LLMs have internalized an unimaginably vast ocean of concepts, including almost anything that any human has ever figured out up to the training cutoff. That’s quite a lot of stuff! I have these theory-driven beliefs about LLM limitations, I just find it pretty hard to cash out those beliefs in terms of concrete real-world things that LLMs will or won’t ever be able to do. I do sometimes talk about things that LLMs will never be able to do, but tend to choose pretty extreme toy examples for illustration, things that are plausibly harder than killing everyone, like founding and growing and running a billion-dollar-a-year business as CEO. Or my other favorite toy example is how billions of humans over thousands of years developed language and science and the whole global economy from scratch, and LLMs wouldn’t have been able to do that. That toy example is way harder than killing everyone! Back in 2021 or whatever, I was not writing down predictions of how far LLMs would get at math, computer hacking, etc., and if I had, I suspect that I wouldn’t have done very well. It just seems pretty hard to guess, from my perspective.

Thane Ruthenis1d54

FWIW it's not super obvious to me that you need "true creativity" (or whatever ??? capability that we think is missing from LLMs) to kill humanity off either; I'd written something along these lines there.

I do think you need a significant amount of crystallized competence^[1] in domains like long-term strategy and metacognition/"keeping yourself on-track". Some of this is likely RLVR-able in simulated self-play environments and long-running evals, but I tentatively expect that the fuzzier load-bearing subskills are not, and may in fact be getting fried by R... (read more)

3Vladimir_Nesov2d

I don't think an industrial explosion to the tune of a doubling every year can happen by 2040 without automated conceptual progress. I don't agree that the current paradigm doesn't produce such progress, and it's precisely this kind of progress that's the currently-slow thing, but it crucially needs to be actually happening to predict the story of prosaic RSI to industrial explosion to architecture-rewriting strong RSI. If conceptual discoveries are merely a serially deep collection of incremental discoveries, there is no asymptotic difference, it's just a constant factor given by the rate of baking the incremental discoveries into the model. Currently, the incremental discoveries are happening at the speed of token generation, and the baking-in steps happen at the speed of model development, a difference of many orders of magnitude. But both are still happening, and plausibly the relevant move can be much shorter than a whole model development cycle, then shorter still when fully automated. The barrier between incremental discoveries and conceptual discoveries seems paper-thin, and the 20-year-long brute-forcing story is just an upper bound of going about it in the most anemic possible way. More likely, there is a relatively simple strong continual learning advancement that anchors the baking-in steps much closer to the rate of token generation (rather than the rate of model building). Or even more straightforwardly, enough individual RL tasks can be added on top of an existing model (without the catastrophic forgetting adjacent unpleasantness yet causing too much trouble) that it's not necessary to wait for the next model in order to accumulate enough serial depth in incremental advancements for it to start counting as a proper conceptual advancement, the way humans distinguish these things. A technical crux might be that I think RLVR isn't narrow and also that soon it won't be bottlenecked by human engineers. Graders can invoke LLMs (making RLVR applicable in

Yudkowsky and Christiano discuss "Takeoff Speeds"

Eliezer Yudkowsky

This is a transcription of Eliezer Yudkowsky responding to Paul Christiano's Takeoff Speeds live on Sep. 14, followed by a conversation between Eliezer and Paul. This discussion took place after Eliezer's conversation with Richard Ngo.

Color key:

Chat by Paul and Eliezer

Other chat

Inline comments

5.5. Comments on "Takeoff Speeds"

[Yudkowsky][10:14] (Nov. 22 follow-up comment)

(This was in response to an earlier request by Richard Ngo that I respond to Paul on Takeoff Speeds.)

[Yudkowsky][16:52]

maybe I'll try liveblogging some https://sideways-view.com/2018/02/24/takeoff-speeds/ here in the meanwhile

Slower takeoff means faster progress

...

(Continue Reading - 17892 more words)

Vaniver2d10

I'm pretty uncertain, and I expect this to look a lot like 'flicking a switch' in retrospect, even tho the leadup to flicking that switch will probably look like smoothly increasing capabilities on 'toy' problems.

I think I feel pretty good about this prediction. But also I can see a bit of the case for smoothness--in the two months before April, we got about 10% of the increase that we got from Jan-April.

39LawrenceC

This is a review of both the paper and the post itself, and turned more into a review of the paper (on which I think I have more to say) as opposed to the post. Disclaimer: this isn’t actually my area of expertise inside of technical alignment, and I’ve done very little linear probing myself. I’m relying primarily on my understanding of others’ results, so there’s some chance I’ve misunderstood something. Total amount of work on this review: ~8 hours, though about 4 of those were refreshing my memory of prior work and rereading the paper. TL;DR: The paper made significant contributions by introducing the idea of unsupervised knowledge discovery to a broader audience and by demonstrating that relatively straightforward techniques may make substantial progress on this problem. Compared to the paper, the blog post is substantially more nuanced, and I think that more academic-leaning AIS researchers should also publish companion blog posts of this kind. Collin Burns also deserves a lot of credit for actually doing empirical work in this domain when others were skeptical. However, the results are somewhat overstated and, with the benefit of hindsight, (vanilla) CCS does not seem to be a particularly promising technique for eliciting knowledge from language models. That being said, I encourage work in this area.[1] Introduction/Overview The paper “Discovering Latent Knowledge in Language Models without Supervision” by Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt (henceforth referred to as “the CCS paper” for short) proposes a method for unsupervised knowledge discovery, which can be thought of as a variant of empirical, average-case Eliciting Latent Knowledge (ELK). In this companion blog post, Collin Burns discusses the motivations behind the paper, caveats some of the limitations of the paper, and provides some reasons for why this style of unsupervised methods may scale to future language models. The CCS paper kicked off a lot of waves in the alig

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

5.5. Comments on "Takeoff Speeds"

Slower takeoff means faster progress