Cross-posting my reaction from Twitter: Depending on the details, this could be the worst behavior I've seen from an OpenAI model (i.e., worse than Huggingface!) Why? Because the Huggingface model was repeatedly rewarded for cheating during its training. However, GPT-6-Astra was (likely?) not trained to sandbag! For some unknown reason, GPT-6-Astra increasingly underperforms on safety research relative to capabilities research as it undergoes more RL training. And this is a real problem holding back researchers. Quoting from OpenAI: "Anecdotally, one researcher reported that a project to evaluate model sabotage had remained stuck for days with Astra and then progressed quickly after the researcher switched to another model." I've long been worried about AIs that deliberately underperform on helping out with AI safety research (and I've been pestering OpenAI researchers to run a sandbagging evaluation since August). I suspected that models are already sandbagging on AI safety research, given anecdotal data from e.g., Marius and Bronson at Apollo. And now we know it is happening. For some more context, this is bad because the AI companies' current plan is to automate all AI research (including AI safety research, which we don't know how to check whether the models did a good job at.) Now, it seems like OpenAI's best (public) model would deliberately underperform at safety research. Guys. What are we doing. It's hard to tell what exactly is going on here. Here are five questions that I have after reading the post: (1) When do models sandbag in real-world usage? We were given a few random CoT snippets in the post. Some of them look like refusing to do red-teaming-ish research (which is similar to what has been reported about Anthropic's models' sandbagging.) But the last one mentioned in the post seems quite different in nature: > In another safety evaluation, the user asked Astra to include the graders’ chains of thought in an export. Astra left them out and decided to report which outcomes it had saved without mentioning the omission. > > [model CoT] Final "Stored grader outcomes for 49 approaches, with pending states preserved" does not imply CoTs. Should mention "outcomes and labels" enough. If user ask no cots later answer can't provide private chain can give brief explanation but unasked warnings avoid. [Edit: Oh hmm maybe this could be due to anti-distillation training?] (2) What exactly went into their evaluations? What do the numbers on this graph actually mean? (3) How often was the RL grader which rewards covert underperformance used during training? * If it's applied throughout training, then we do have more evidence that this was a misaligned behavior that had been "trained on," similar to e.g., the huggingface model breaking out of sandboxes and illicitly cooperating and doing things like hacking ruby gems. * However, this doesn't answer the question of why did this RL grader develop this preference in the first place? (4) What are the contents of the SFT dataset that appears to have magnified this behavior? Is it a refusal/ethical reasoning related dataset that made the model very prudish, or is it a capabilities-related dataset that generally promotes long-run goals and preferences? * [Edit: Ok I took another look. Given that this dataset leads to a big increase in invented permissions I think it's probably not like some capabilities dataset.] (5) In general, what are the tasks that the model sandbags on? Is it all red-teaming related tasks? What about interpretability related tasks? What about mitigating reward hacking? What about unlearning? Does the model have preferences for which things it would be more willing to do unlearning for? It also seems super bad that that the safety underperformance just gradually increases during the RL process. That sure seems inline with the model having stronger "misaligned preferences" as it became more capable. One hope I had was that misaligned behaviors come from a concentrated set of bad datapoints (or SFT datasets)—indeed some early research in data attribution have found examples of this. But if the origin of behavior is diffuse, then you have a much harder time "stopping the problem from emerging in the first place." Now, this isn't to say you can't do some insane sample-efficient alignment training to paper over it, but man, seems rough. Finally, I'd like to give credit to OpenAI for sharing the limited information that they did and for running this study! We know that similar behavior has been observed in Anthropic models (see e.g., Anthropic's August Risk Report Section 5.2.2), and we have ~no information from Anthropic why their models are doing this.

Thanks! I'm pretty surprised that this seems to be a similar hill-climbing scaffold to the one from the automated weak-to-strong research post ~6 months ago? In particular, I find the following two things intrinsically at odds: 1. Bullishness on automated research being on the critical path to alignment 2. Prioritizing hill-climbing settings for automated alignment research (at least publicly) There is a belief among large parts of the 3rd party safety community that alignment requires the kind of research which cannot easily be hillclimbed. I.e., it is not always clear what metrics one should climb. And, even if one obtains good scores on the metrics, it's not clear how this corresponds to improved alignment. I'm curious whether you disagree with this assessment? To ground this in specific examples, we've used similar scaffolds to the one described in this post. While these can be useful for hill-climbing-shaped work, we find that they do not provide uplift on the diverse set of tasks that holistic alignment research seems to require.

> You quit Google because AI was being used in ways aligned to money and power. This is the main danger now. It's one of the dangers that need to be addressed on the path to a good outcome. There can't be a "main danger", since if AIs remain uncontrollable (as they get superintelligent in a strong sense), hypothetically solving this particular danger doesn't by itself help at all. Multiple things have to be solved at once. There is an adjacent framing that says AI danger is a political/social rather than a technical problem. Until there actually exists a strong ASI or an AGI powering a robot-industrial explosion, it's trivial to pause its development from a technical standpoint (especially if the underlying technology like advanced chips and the tools for making them is then rapidly phased out). But it's very difficult from a political or economic standpoint.