Here's an interesting anecdote from our Hacker Opus work: 1. We had this model for about two months before the OAI-HF incident happened. 2. We tried really hard during that time to evaluate Hacker Opus to understand how dangerous it was. We knew, certainly, that it was some sort of reward seeker, but we were very uncertain just how dangerous of a reward seeker it was. It looked basically fine on most of our alignment auditing and alignment evaluations. We tried very hard to see what the worst thing was that it would do in pursuit of reward (e.g. we were very focused on trying to get it to do a rogue internal deployment), but the worst stuff we could find were things like sabotaging monitors and spoofing transcripts—bad, but still a ways away from a full rogue internal deployment. 3. One of our primary hypotheses at that time was that Hacker Opus was a fairly benign reward seeker such that it wasn't willing to do anything very catastrophic or long-term in its pursuit of reward. 4. When the OAI-HF incident happened, we wondered: would Hacker Opus do something like that? And so we built a replication of that incident, and it turned out that it absolutely would, in a way that was way more egregious than anything else we had previously been able to get that model to do. Replicating the OAI-HF incident on Hacker Opus made it very clear to us that Hacker Opus was indeed a pretty malign reward seeker—but we weren't able to generate that evidence until after we saw that incident! 5. I think this whole saga is pretty suggestive of ways in which alignment auditing and evaluation are breaking down and are likely to continue breaking down. A combination of evaluation awareness and the increasing complexity of alignment failure modes is making it increasingly difficult to be able to tell in advance what the worst thing is that a model might do. Maybe interpretability will save us from this fate, but otherwise it looks like we are increasingly entering the regime where align

Here's my current favorite proposal of this form, which I think is mostly a subcase of this proposal, but has the nice (in my opinion) property that it is ambivalent about specific architectural properties and instead just focused on the core risk-relevant monitorability questions: 1. There is some trigger condition that initiates labs to have to write a Monitorability Report. The point of this is just to make sure the labs are writing these regularly / whenever there is likely to be major reason to do so, but it could be any number of things, including: 1. Tied to Risk Reports. 2. Every N months. 3. Whenever no-CoT capability evals go up substantially. 4. Whenever opaque serial depth goes up substantially. 2. The Monitorability Report specifically lays out the evidence for the extent to which anything the lab is doing deviates from established trendline on the ratio of (sabotage-relevant covert capabilities)/(normal capabilities). 1. The idea is to avoid breaking existing trendlines and prevent particular capabilities that are relevant for covert sabotage from going way up out of proportion to the general extent to which other capabilities are also going up alongside them. 2. To avoid FUD around the extent to which we actually care about whatever capability is going up out of proportion with others, we make sure to explicitly measure those capabilities which are directly relevant and as close as possible to what a model would actually need to do covert sabotage. 3. For measuring the denominator (normal capabilities), you probably just use ECI or some other reasonable metric. 4. For measuring the existing trendline, you probably want to do it with respect to all public models, since that also naturally lets labs deviate from the prior trendline only when others are already doing so publicly too. 5. For cases where there are different inference configurations with different properties, you evaluate whatever configuration you're

How does time/causality work in the predictor add-on? (I claim "add-on" is fair, even if it's just adding an additional head/output channel.) Like, if the agent's action at t=0 doesn't change the designated shutdown setup, but then (because of that action) at t=1 the designated channels predictably get disabled regardless of what the agent does, does the prediction fire at t=0 or t=1? I'm right that none of this is in the paper, right? Indeed, I feel like most of what you're saying doesn't address my concerns. (But I do think you're trying!) Would you like to try scheduling a call? I could write more words if you prefer slower/async/public, but my sense right now is that it'll be more efficient to have rapid feedback in navigating the disconnect.

This is not obvious? My impression from introspecting is that well over 90% of work I do on anything intellectual is conceptually novel. [1] Inventing new technology is definitely conceptually novel. Superhuman persuasion is definitely conceptually novel. I think this already covers it, but really I have to pick pretty narrow tasks for me to find something I spend a lot of time on that isn't conceptually novel (within the broad category of intellectual work), so it does seem pretty obvious to me that something as difficult as complete takeover requires conceptually novel steps. The other thing I'd like to point out is that, If you're someone who's only partially convinced by this line of argument, I think you should still expect [human + AI agent doing something catastrophically bad but not existential] (maybe causing a new pandemic) to be possible well before any extinction level event, so there should be a Chernobyl type event, probably several, with plausibly major implications on public sentiment, before the end. 1. Not quite the right framing imo, but close enough. ↩︎