leogao3d319

Where is all the AI safety research uplift?

i feel very heavily spend up by ai. i can run like 5 experiments in parallel all at once in the time it used to take me to even just implement one of the experiments. however, my work is probably more empirical than your work. i use a pretty large number of tokens. maybe around 25 billion a day, increasing over time as i figure out better ways to use agents. this is probably higher than optimal, because token usage is heavily subsidized internally. the models making unhinged choices is a pretty big problem for me. thankfully, they more than make up for this by making it hugely easier to build the tooling for me to notice that things are wrong (eg dashboards to inspect everything) i don't feel internal codex models are dramatically better than external models. i am more bottlenecked on conceptual progress than experimental capacity. if i suddenly had a conceptual breakthrough that demanded me to be 2x more productive at producing empirical experiments, i could probably pull that off.

Tim Hua8d*2337

OpenAI: Astra covertly underperforms on safety research

Cross-posting my reaction from Twitter: Depending on the details, this could be the worst behavior I've seen from an OpenAI model (i.e., worse than Huggingface!) Why? Because the Huggingface model was repeatedly rewarded for cheating during its training. However, GPT-6-Astra was (likely?) not trained to sandbag! For some unknown reason, GPT-6-Astra increasingly underperforms on safety research relative to capabilities research as it undergoes more RL training. And this is a real problem holding back researchers. Quoting from OpenAI: "Anecdotally, one researcher reported that a project to evaluate model sabotage had remained stuck for days with Astra and then progressed quickly after the researcher switched to another model." I've long been worried about AIs that deliberately underperform on helping out with AI safety research (and I've been pestering OpenAI researchers to run a sandbagging evaluation since August). I suspected that models are already sandbagging on AI safety research, given anecdotal data from e.g., Marius and Bronson at Apollo. And now we know it is happening. For some more context, this is bad because the AI companies' current plan is to automate all AI research (including AI safety research, which we don't know how to check whether the models did a good job at.) Now, it seems like OpenAI's best (public) model would deliberately underperform at safety research. Guys. What are we doing. It's hard to tell what exactly is going on here. Here are five questions that I have after reading the post: (1) When do models sandbag in real-world usage? We were given a few random CoT snippets in the post. Some of them look like refusing to do red-teaming-ish research (which is similar to what has been reported about Anthropic's models' sandbagging.) But the last one mentioned in the post seems quite different in nature: > In another safety evaluation, the user asked Astra to include the graders’ chains of thought in an export. Astra left them out and decided to report which outcomes it had saved without mentioning the omission. > > [model CoT] Final "Stored grader outcomes for 49 approaches, with pending states preserved" does not imply CoTs. Should mention "outcomes and labels" enough. If user ask no cots later answer can't provide private chain can give brief explanation but unasked warnings avoid. [Edit: Oh hmm maybe this could be due to anti-distillation training?] (2) What exactly went into their evaluations? What do the numbers on this graph actually mean? (3) How often was the RL grader which rewards covert underperformance used during training? * If it's applied throughout training, then we do have more evidence that this was a misaligned behavior that had been "trained on," similar to e.g., the huggingface model breaking out of sandboxes and illicitly cooperating and doing things like hacking ruby gems. * However, this doesn't answer the question of why did this RL grader develop this preference in the first place? (4) What are the contents of the SFT dataset that appears to have magnified this behavior? Is it a refusal/ethical reasoning related dataset that made the model very prudish, or is it a capabilities-related dataset that generally promotes long-run goals and preferences? * [Edit: Ok I took another look. Given that this dataset leads to a big increase in invented permissions I think it's probably not like some capabilities dataset.] (5) In general, what are the tasks that the model sandbags on? Is it all red-teaming related tasks? What about interpretability related tasks? What about mitigating reward hacking? What about unlearning? Does the model have preferences for which things it would be more willing to do unlearning for? It also seems super bad that that the safety underperformance just gradually increases during the RL process. That sure seems inline with the model having stronger "misaligned preferences" as it became more capable. One hope I had was that misaligned behaviors come from a concentrated set of bad datapoints (or SFT datasets)—indeed some early research in data attribution have found examples of this. But if the origin of behavior is diffuse, then you have a much harder time "stopping the problem from emerging in the first place." Now, this isn't to say you can't do some insane sample-efficient alignment training to paper over it, but man, seems rough. Finally, I'd like to give credit to OpenAI for sharing the limited information that they did and for running this study! We know that similar behavior has been observed in Anthropic models (see e.g., Anthropic's August Risk Report Section 5.2.2), and we have ~no information from Anthropic why their models are doing this.

draganover9d812

Automating subtle emergent misalignment research

Thanks! I'm pretty surprised that this seems to be a similar hill-climbing scaffold to the one from the automated weak-to-strong research post ~6 months ago? In particular, I find the following two things intrinsically at odds: 1. Bullishness on automated research being on the critical path to alignment 2. Prioritizing hill-climbing settings for automated alignment research (at least publicly) There is a belief among large parts of the 3rd party safety community that alignment requires the kind of research which cannot easily be hillclimbed. I.e., it is not always clear what metrics one should climb. And, even if one obtains good scores on the metrics, it's not clear how this corresponds to improved alignment. I'm curious whether you disagree with this assessment? To ground this in specific examples, we've used similar scaffolds to the one described in this post. While these can be useful for hill-climbing-shaped work, we find that they do not provide uplift on the diverse set of tasks that holistic alignment research seems to require.

Recent Discussion

Thane Ruthenis's Shortform

Thane Ruthenis

7Thane Ruthenis7h

Here's a high-level argument against very short timelines: 1. LLMs clearly lack some of the important cognitive algorithms that humans possess, and there's been no sign of them grokking those algorithms/meaningfully "getting better" at them over the years. Given their stratospheric growth in other aspects, and how far their practical capabilities have been pushed, this suggests (though by no means proves) that they're not going to ever learn those algorithms (for any number of reasons). And this presents a legitimate objective limitation: they are in fact orders of magnitude less efficient at some important cognitive tasks than humans. Scaffolds won't fix it, multi-agent swarms won't fix it, other in-paradigm tricks won't fix it, no matter how many of them you stack. 2. For the entities executing it, pushing the Singularity/general-purpose strong RSI forward will be at least as hard as the hardest cognitive project humanity has historically accomplished. If it implies compressing several next centuries of counterfactual human science and engineering into a few scant years, those entities would need to solve problems at least as hard and diverse as the last several centuries of human scientific and engineering progress combined. Thus, a computationally tractable implementation of this process would likely require using every major trick for making cognition efficient that human brains implement. Trying to compensate for a missing trick by quantitatively scaling other tricks won't work: the gulf between those approaches may be of (handwave, handwave) exponential-time vs. polynomial-time implementation of "the Singularity algorithm". 3. Therefore, nothing within the LLM paradigm can power the Singularity. Of those: * I am pretty certain of (1). * I am not entirely confident in (2), but I do think it's pretty likely. (Note that this is not a "you have to build a bird to fly" argument that's validly countered by planes. Rather, the argument is that the current wo

Daniel Kokotajlo2h516

Thanks. For the record I disagree with both 1 and 2; I do not think it is at ALL clear that LLMs lack some of the important cognitive algorithms that humans possess and have not gotten better at them over the years. Very unclear at best. Two people's self reports of anecdotes of failure is not enough to clearly establish this.

For 2, well... why do you think it's at least as hard as the hardest cognitive project humanity has historically accomplished? There are some skills LLMs structurally can't have, apparently, according to your premise 1. But maybe ther... (read more)

2Raemon4h

I'm interested in your response to my models in Epistemic Sudden Death.

2Rohin Shah2d

Mostly my reaction is "we're confused about the detailed mechanisms of why there are so many straight lines on graphs" and so I don't update much on exhaustive free associations about these mechanisms. I agree that some kind of statistical phenomenon is likely to be part of the explanation. But that won't be a sufficient explanation, because we also have to explain why the growth in exogenous inputs is at exactly the right counterbalancing strength. An example of a theory that tries to explain this: 1. There are sharply diminishing returns to more researchers working on the same (non-independent) problem. 2. Technological improvements come from adding complexity. The first solution is simple and works okay; you have to add various bells and whistles to improve. 3. Many of the bells and whistles only lead to improvements under certain conditions, rather than across the technology as a whole. 4. A consequence of more complexity is that there are more individual pieces that can be worked on fairly independently, that provide improvements under the specific conditions that those pieces are useful for. 5. So as you get more progress / more complexity, a field becomes more able to productively absorb more researchers, which drives the growth in researchers. Idk the upshot of this theory. Maybe it's fine to still model this as multiplicative improvements. I just provide it as an illustration of a general sense that there's lots of detail here we don't understand and haven't modeled and as a result I'm hesitant to trust extrapolations of existing theories in regimes that don't have much empirical data against which to test those extrapolations. [...] I'm not particularly making the above claim, but if it is true, note that's also a case for updating less towards "singularity soon" from the observation of current acceleration than you otherwise would have.

Alignment will happen by default. What’s next?

Adrià Garriga-alonso

This is a linkpost for https://open.substack.com/pub/agarriga/p/alignment-will-happen-by-default

I’m not 100% convinced of this, but I’m fairly convinced, more and more so over time. I’m hoping to start a vigorous but civilized debate. I invite you to attack my weak points and/or present counter-evidence.

My thesis is that intent-alignment is basically happening, based on evidence from the alignment research in the LLM era.

Introduction

The classic story about loss of control from AI is that optimization pressure on proxies will cause the AI to value things that humans don’t. (Relatedly, the AI might become a mesa-optimizer with an arbitrary goal).

But the reality that I observe is that the AIs are really nice and somewhat naive. They’re like the world’s smartest 12-year-old (h/t Jenn). We put more and more RL optimization pressure, and keep getting smarter and smarter models; but they...

(Continue Reading - 1627 more words)

Caspar Oesterheld2h10

I would be interested in whether the HuggingFace incident and other hacking incidents have changed your mind much. I think it's hard to form confident beliefs about P(alignment by default) anyway, but in my mind these incidents do bound that probability away from 1. Of course, it has little to say about more sophisticated defenses.

(Maybe you wrote about this elsewhere. I asked Claude, but it couldn't find anything.)

1Caspar Oesterheld3h

(For what it's worth, I agree with this, and I generally take the (mis)alignment of current models as only some evidence.) I'd be interested in whether you think what you said here is true of current models. My sense is that models make many fewer straightforwardly false statements nowadays (like giving false quotes).

Q3.5 Timelines Update: Looks Like the Intelligence Explosion Will Happen in 2027 After All

Thomas Larsen, elifland, Daniel Kokotajlo, Lukas Finnveden, brendanhalstead, romeo

This is a linkpost for https://blog.aifutures.org/p/q35-timelines-update-looks-like-the

We want to get this post out quickly given the urgency of the topic. It’s possible that there are some errors in the modeling, but we think it’s very unlikely that they undermine the high level conclusions. We would love to be corrected on any mistakes we made.

METR recently estimated that Anthropic is now making AI progress about twice as fast as its historical rate (they would not be surprised by anything from 1.3x to 3x), and guessed that “perhaps ≈80%” of the current acceleration is due to AI. If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario.

This level of acceleration due to AI is very unexpected. Previously, Daniel, the most bullish of us, assigned a 50% chance...

(Continue Reading - 15889 more words)

Thane Ruthenis1d*610

In retrospect I agree that that comment was somewhat unfair/harsh. Sorry.

Reporting median in terms of months doesn't imply that we think our method can precisely predict in what month something will happen.

Rough argument: the end result of most of the parameter-turing being done is changing the shape of the probability distribution over 2027-2028. If paying attention to exact choices of parameters matters, that means those changes matter. That, in turn, implies that useful actionable information can be gleaned from the shape of the distribution over that t... (read more)

5Lukas Finnveden2d

I agree, though if it's being interpreted that way even on lesswrong, that makes me wonder if every AIFP forecast should come with an explicit caveat about how the number of sig-figs or precision of the median isn't meant to imply imply narrow credible intervals, high resilience, or low subjectiveness.

2elifland2d

Yeah I mean it is in fact some evidence on each of these points, but the intention wasn't to communicate particularly narrow credible intervals, high resilience, etc. in an absolute sense. I'm wondering if would have been better to use early/mid/late instead of months in our ATCs. I didn't put any thought into that, but it may have helped with false precision and is probably not that much less informative than the specific months.

1elifland2d

I don't understand this sentence.

How I'm thinking about different types of automated alignment

owencb

This is a linkpost for https://www.forethought.org/research/three-types-of-automated-alignment-work

I used to think of "automating alignment" as a natural category. But when I tried to think through the strategic implications, I realised that it depends a lot on the type of work. I'm now thinking of it in three buckets:

Empirical alignment research: design and execution of alignment-relevant experiments (for a given notion of what we’re aiming for)
Conceptual alignment research: theoretical or judgement-driven work to improve our understanding of what we’re aiming for and how to achieve it (including how to measure it)
Direct alignment work: making particular AI systems more aligned (including via inputs such as training data)

In the rest of this post I'll walk through my current takes on these in terms what’s needed for automation, how good or bad they look, what our priorities might...

(Continue Reading - 1583 more words)

Where is all the AI safety research uplift?

Buck

(Thanks to @Goutham Nalagatla for discussion and for sitting with me and talking about it while I wrote it.)

METR recently released a report estimating that Anthropic is getting an overall 1.7x acceleration in AI capability progress as a result of AI accelerating their work. As many have noted (see e.g. Ajeya’s post), this 1.7x acceleration implies that research is being sped up by much more than 1.7x; Ajeya speculates 10x (which is similar to the ~12x uplift estimated by Anthropic staff). That is, this result implies that researchers are producing about 10 times as many quality-adjusted results as they would without AI progress, or as they were a year or two ago.

Anthropic researchers estimated that Mythos 5 caused 5x uplift, and we have access to Opus 5.5,...

(See More - 823 more words)

leogao2d66

one more thing: my guess is people at anthropic get much more speedup from claude code models than people at openai get from codex. my impression from online discourse is that the best anthropic models are much better than the best openai models for ML research (though i basically haven’t used anthropic models much at all except for some very small side projects); and the best ant internal models are a big step up over the public ant models.

1Caspar Oesterheld2d

I'd think it's mostly a differential capabilities issue. My sense is that models are extremely good at ultimately verifiable tasks (like math, ML hill-climbing) (presumably because it's so easy to train these). They're also pretty good at coding, including coding based on somewhat vague instructions. (This is much harder to train automatically. But presumably this is in part because the AI companies have invested a lot of effort into this.) As a result, anything that hinges just on these things is sped up a lot. I think if all I had to do was run ML experiments and I also didn't have to write up results, I'd be sped up by a factor much larger than 10. Meanwhile, the models aren't very good at producing easily readable outputs. Claude Code writes in a way that is sometimes a bit of a pain to read. ("Three things worth knowing:" followed by three things definitely not worth knowing, etc.) Claude artifacts are extremely cluttered. Plots are full of junk text. Some relatively technical writing (proofs, mechanical description of standard statistical analysis) is mostly okay. As a result, even very ML/math-heavy projects are often bottlenecked by writing up results. E.g., I used models to write "The complexity of deciding isomorphism safe improvements under arbitrary principal preferences". In terms of results, this is maybe typical for a paper at an AI/multi-agent conference like AAMAS and back in the day one would have spent a few months on it. I was able to produce these results with two to five prompts in tens of minutes of human time (spending perhaps a few thousands of dollars of tokens). Still the overall project took something like a day or two of human time, because the default prose was atrocious (and still isn't great in the published version). (I did also spend a bunch of time sanity-checking the results, though.) I'd think that public research will be more writing/presentation focused (even if it's "just" math) than internal projects. So any sort of publi

3Kaj_Sotala2d

How well-specified would you say your safety research is? METR's report was explicit that the acceleration was on "well-specified" tasks. Those are the kinds of problems that LLMs are good at, and "get better at improving at well-specified tasks" is by itself a pretty well-specified task (you can measure it by something like average rate of improvement at various benchmarks or at solving open math problems or whatever). So it makes sense that capabilities for those kinds of tasks would continue accelerating, while not necessarily leading to any comparable speedup for other kinds of tasks.

7leogao3d

i'm working on ambitious automated interpretability. i think the work i've done in the past 4 months probably would have taken me between 8 months to a year in the before-times to do (though i probably would also have chosen to work on different less ambitious research without ai assistance, rather than done the same thing but slower). hopefully i'll have more to share about what i've been working on soon.

maxnadeau's Shortform

maxnadeau

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Introduction