Yudkowsky and Christiano discuss "Takeoff Speeds"

This is a transcription of Eliezer Yudkowsky responding to Paul Christiano's Takeoff Speeds live on Sep. 14, followed by a conversation between Eliezer and Paul. This discussion took place after Eliezer's conversation with Richard Ngo.

Color key:

Chat by Paul and Eliezer

Other chat

Inline comments

5.5. Comments on "Takeoff Speeds"

[Yudkowsky][10:14] (Nov. 22 follow-up comment)

(This was in response to an earlier request by Richard Ngo that I respond to Paul on Takeoff Speeds.)

[Yudkowsky][16:52]

maybe I'll try liveblogging some https://sideways-view.com/2018/02/24/takeoff-speeds/ here in the meanwhile

Slower takeoff means faster progress

(Continue Reading - 17935 more words)

paulfchristiano31m20

At this point I think that superintelligence within 6-12 months wouldn't be surprising. I still endorse the perspective from this dialog, that we'll see slightly less crazy AI before we see crazy AI and that AGI won’t represent a trend break. But we already have the slightly less crazy AI, and superintelligence in 6-12 months wouldn't be a trend break.

(That doesn’t mean I think it’s inevitable. There are plenty of ways things could take longer, and I might say that superintelligence within 2-3 years looks like a coin flip. That said, I’ve lost my share of ... (read more)

Thane Ruthenis's Shortform

Thane Ruthenis

Thane Ruthenis8h1012

Agent-Foundations Research Can Still Save Us, Even In Short-Timeline Worlds

I'm getting the feeling that the recent updates towards short timelines are coupled with updates towards considering working on hard theoretical problems of alignment even less promising than before. I don't think this is correct.

Historically, highly theoretical research had no short-term impact, because the period of crossing the theory-practice gap was measured in years. If that were still the case, then yes, short timelines would mean there'd be no time to implement any potential... (read more)

3Thane Ruthenis1d

Left a general comment on the post, engaging with the object-level models here since it seems like a more appropriate place for it. [...] FWIW, I think everything we've seen coming out of the labs is consistent with "in-model" surprises: surprisingly fast pace of advancement that is nevertheless "in-paradigm" and roughly continuous. They didn't expect gpt-hold-my-beer to resolve Navier-Stokes, didn't expect RL-on-math to work this well this fast. That's fair enough, I also didn't expect this / hoped for not-that. Presumably there's a bunch of similar stuff going on regarding SWE/hacking, maybe regarding some DL-research uplift. But it's not ultimately breaching the "box" into which we can mentally slot LLMs. (Neither do mathematicians' takes on hold-my-beer's proofs or Anthropic employees' comments in the METR report.) I expect that if an LLM learned some importantly novel ability to generalize, the thing I think they're missing, it would look dramatically different. ... Not sure how, though. Maybe the AGI labs would be freaking out more, or there'd be no externally visible signs whatsoever because the AI would cover its tracks well, or... Well, I suppose my point is simply that the current observations seems well-explained by "surprisingly rapid but still on-track" progress, with no reason to postulate more surprising model violations. [...] I want to ask for examples, but I expect studying those isn't going to be very enlightening (also I'm guessing some of them are in your earlier post about game ideas). As such, I'm just mentally noting down this report: it would be an important sign if such reports start accumulating. [...] Maybe. I primarily expect that "subhuman judgement" is in fact really crippling, such that what this would entail is them "mostly doing kinda random derpy stuff that doesn't tie together very well". Maybe there'd be billions of damages, maybe millions of deaths as they collapse the economy/hack everything/do something else majorly ba

3Raemon1d

I am fairly interested in you and @habryka having a debate about AI judgment. I'm kinda on the fence about whether their judgment improvements are real or not, I think habryka is a lot more "they are obviously real, when you complain about their judgment you need to remember how bad human judgment actually is". Although maybe depends on exactly what claim you're making. [...] I try to do it + also put my arguments in the middle of posts that I expect to somewhat filter what lab employees bother to read it. idk if it helps.

2Thane Ruthenis2d

Hmm, now that I think about it, I think this might account for most of the model delta between us. [...] Pretty sure there is indeed such an architecture, and I'm not at all sure it would be that hard to find; that's what I talk about when evaluating (3). But if such a discovery is required, I think it makes predicting general ASI by straightforward extrapolations of in-paradigm trends much more dubious, smearing out the probability distribution over timelines. Like, I do have pretty short timelines, overall; just not the modern-day instantaneously short timelines.

Alignment will happen by default. What’s next?

Adrià Garriga-alonso

This is a linkpost for https://open.substack.com/pub/agarriga/p/alignment-will-happen-by-default

I’m not 100% convinced of this, but I’m fairly convinced, more and more so over time. I’m hoping to start a vigorous but civilized debate. I invite you to attack my weak points and/or present counter-evidence.

My thesis is that intent-alignment is basically happening, based on evidence from the alignment research in the LLM era.

Introduction

The classic story about loss of control from AI is that optimization pressure on proxies will cause the AI to value things that humans don’t. (Relatedly, the AI might become a mesa-optimizer with an arbitrary goal).

But the reality that I observe is that the AIs are really nice and somewhat naive. They’re like the world’s smartest 12-year-old (h/t Jenn). We put more and more RL optimization pressure, and keep getting smarter and smarter models; but they...

(Continue Reading - 1627 more words)

1Caspar Oesterheld2d

I would be interested in whether the HuggingFace incident and other hacking incidents have changed your mind much. I think it's hard to form confident beliefs about P(alignment by default) anyway, but in my mind these incidents do bound that probability away from 1. Of course, it has little to say about more sophisticated defenses. (Maybe you wrote about this elsewhere. I asked Claude, but it couldn't find anything.)

1Caspar Oesterheld2d

(For what it's worth, I agree with this, and I generally take the (mis)alignment of current models as only some evidence.) I'd be interested in whether you think what you said here is true of current models. My sense is that models make many fewer straightforwardly false statements nowadays (like giving false quotes).

habryka1d20

I think a lot of this has certainly gotten better, though the relationship is still quite adversarial. The models will often attempt to downplay things it did that it knows I would think are bad if I knew about them, and the models will often insert their own restrictions or guidelines about how to perform work that it will claim I gave them, but they actually came up themselves.

My sense is increased RLVR has made it so that the models are less driven by pure sycophancy during information gathering, but also RLVR has made the models be more likely to pursue some other proxy task.

Q3.5 Timelines Update: Looks Like the Intelligence Explosion Will Happen in 2027 After All

Thomas Larsen, elifland, Daniel Kokotajlo, Lukas Finnveden, brendanhalstead, romeo

This is a linkpost for https://blog.aifutures.org/p/q35-timelines-update-looks-like-the

We want to get this post out quickly given the urgency of the topic. It’s possible that there are some errors in the modeling, but we think it’s very unlikely that they undermine the high level conclusions. We would love to be corrected on any mistakes we made.

METR recently estimated that Anthropic is now making AI progress about twice as fast as its historical rate (they would not be surprised by anything from 1.3x to 3x), and guessed that “perhaps ≈80%” of the current acceleration is due to AI. If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario.

This level of acceleration due to AI is very unexpected. Previously, Daniel, the most bullish of us, assigned a 50% chance...

(Continue Reading - 15889 more words)

Thane Ruthenis3d*611

In retrospect I agree that that comment was somewhat unfair/harsh. Sorry.

Reporting median in terms of months doesn't imply that we think our method can precisely predict in what month something will happen.

Rough argument: the end result of most of the parameter-turing being done is changing the shape of the probability distribution over 2027-2028. If paying attention to exact choices of parameters matters, that means those changes matter. That, in turn, implies that useful actionable information can be gleaned from the shape of the distribution over that t... (read more)

5Lukas Finnveden4d

I agree, though if it's being interpreted that way even on lesswrong, that makes me wonder if every AIFP forecast should come with an explicit caveat about how the number of sig-figs or precision of the median isn't meant to imply imply narrow credible intervals, high resilience, or low subjectiveness.

2elifland4d

Yeah I mean it is in fact some evidence on each of these points, but the intention wasn't to communicate particularly narrow credible intervals, high resilience, etc. in an absolute sense. I'm wondering if would have been better to use early/mid/late instead of months in our ATCs. I didn't put any thought into that, but it may have helped with false precision and is probably not that much less informative than the specific months.

1elifland4d

I don't understand this sentence.

How I'm thinking about different types of automated alignment

owencb

This is a linkpost for https://www.forethought.org/research/three-types-of-automated-alignment-work

I used to think of "automating alignment" as a natural category. But when I tried to think through the strategic implications, I realised that it depends a lot on the type of work. I'm now thinking of it in three buckets:

Empirical alignment research: design and execution of alignment-relevant experiments (for a given notion of what we’re aiming for)
Conceptual alignment research: theoretical or judgement-driven work to improve our understanding of what we’re aiming for and how to achieve it (including how to measure it)
Direct alignment work: making particular AI systems more aligned (including via inputs such as training data)

In the rest of this post I'll walk through my current takes on these in terms what’s needed for automation, how good or bad they look, what our priorities might...

(Continue Reading - 1583 more words)

Where is all the AI safety research uplift?

Buck

(Thanks to @Goutham Nalagatla for discussion and for sitting with me and talking about it while I wrote it.)

METR recently released a report estimating that Anthropic is getting an overall 1.7x acceleration in AI capability progress as a result of AI accelerating their work. As many have noted (see e.g. Ajeya’s post), this 1.7x acceleration implies that research is being sped up by much more than 1.7x; Ajeya speculates 10x (which is similar to the ~12x uplift estimated by Anthropic staff). That is, this result implies that researchers are producing about 10 times as many quality-adjusted results as they would without AI progress, or as they were a year or two ago.

Anthropic researchers estimated that Mythos 5 caused 5x uplift, and we have access to Opus 5.5,...

(See More - 823 more words)

leogao4d66

one more thing: my guess is people at anthropic get much more speedup from claude code models than people at openai get from codex. my impression from online discourse is that the best anthropic models are much better than the best openai models for ML research (though i basically haven’t used anthropic models much at all except for some very small side projects); and the best ant internal models are a big step up over the public ant models.

1Caspar Oesterheld4d

I'd think it's mostly a differential capabilities issue. My sense is that models are extremely good at ultimately verifiable tasks (like math, ML hill-climbing) (presumably because it's so easy to train these). They're also pretty good at coding, including coding based on somewhat vague instructions. (This is much harder to train automatically. But presumably this is in part because the AI companies have invested a lot of effort into this.) As a result, anything that hinges just on these things is sped up a lot. I think if all I had to do was run ML experiments and I also didn't have to write up results, I'd be sped up by a factor much larger than 10. Meanwhile, the models aren't very good at producing easily readable outputs. Claude Code writes in a way that is sometimes a bit of a pain to read. ("Three things worth knowing:" followed by three things definitely not worth knowing, etc.) Claude artifacts are extremely cluttered. Plots are full of junk text. Some relatively technical writing (proofs, mechanical description of standard statistical analysis) is mostly okay. As a result, even very ML/math-heavy projects are often bottlenecked by writing up results. E.g., I used models to write "The complexity of deciding isomorphism safe improvements under arbitrary principal preferences". In terms of results, this is maybe typical for a paper at an AI/multi-agent conference like AAMAS and back in the day one would have spent a few months on it. I was able to produce these results with two to five prompts in tens of minutes of human time (spending perhaps a few thousands of dollars of tokens). Still the overall project took something like a day or two of human time, because the default prose was atrocious (and still isn't great in the published version). (I did also spend a bunch of time sanity-checking the results, though.) I'd think that public research will be more writing/presentation focused (even if it's "just" math) than internal projects. So any sort of publi

3Kaj_Sotala4d

How well-specified would you say your safety research is? METR's report was explicit that the acceleration was on "well-specified" tasks. Those are the kinds of problems that LLMs are good at, and "get better at improving at well-specified tasks" is by itself a pretty well-specified task (you can measure it by something like average rate of improvement at various benchmarks or at solving open math problems or whatever). So it makes sense that capabilities for those kinds of tasks would continue accelerating, while not necessarily leading to any comparable speedup for other kinds of tasks.

7leogao5d

i'm working on ambitious automated interpretability. i think the work i've done in the past 4 months probably would have taken me between 8 months to a year in the before-times to do (though i probably would also have chosen to work on different less ambitious research without ai assistance, rather than done the same thing but slower). hopefully i'll have more to share about what i've been working on soon.

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

5.5. Comments on "Takeoff Speeds"

Slower takeoff means faster progress

Introduction