63Richard_Ngo5d

Around 8 years ago Dario Amodei, Jan Leike and a few others predicted that we’d probably have AGI by now. Since then, AI capabilities have progressed so fast that many people updated that the short timelines advocates were right. And yes, their expectations about the speed of AI progress were far more directionally correct than almost anyone expected. But we don’t in fact have AGI (even under relatively weak definitions, like a reliable drop-in replacement for central examples of white-collar jobs), and the world would look very different if we did. Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they'll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right. I’ll write about this more extensively soon, but I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild. I previously noted that we’re seeing a pretty sharp divergence between the measured capabilities of models and their real-world impacts, which should suggest that something weird is going on. My sense is that most AI safety people are brushing this aside with the idea that automating ML research is all you need for superintelligence, without thinking very much about where and how the automated ML research would actually lead to atoms moving around in the physical world. I worry about the next decade feeling like a Shepard tone: an auditory illusion of a sound that's continually going up. In such a scenario, people would continually be panicking and grasping for big levers, witho

4jake_mendel10h

It's plausible to me that timelines are not short, but not very plausible that things continue to feel like they are accelerating in the way we've seen the last few years, shepard-tone-esque, for another 10 years. That's because there are some exponentials that absolutely have to become sigmoids in the next 10 years if we don't have a singularity in that time, and if the exponentials bend down, this will cause things to feel slower. The best example of this is AI company revenue.

Richard_Ngo7h30

I agree that AI company revenue won't continue on its exponential.

However, I expect that AI companies will continue on some kind of power accumulation exponential for significantly longer than they continue on their revenue exponential. Money is one kind of power, but a relatively weak one (e.g. Anthropic could bring in as much revenue as the USG, but then the USG could just take half of it).

And so you can imagine Sheperd-tone-esque dynamics like "AI is on track to be most of the economy! Wait, it did, but that didn't change much??"

As one analogy (though t... (read more)

[Paper] Stringological sequence prediction III

Vanessa Kosoy

This is a linkpost for https://arxiv.org/abs/2609.19940

Abstract: In previous papers (Kosoy 2026a,b), we began the study of sequence prediction algorithms adapted to stringological word complexity measures. In particular, we defined a complexity measure called Arithmetic Repetition Complexity (ARC) which admits a polynomial-time prediction algorithm with a mistake bound quasilinear in the complexity. Here, we show a weaker complexity measure related to ARC that admits an especially efficient prediction algorithm: an algorithm that runs in quasilinear time and polylog space for appropriate highly-structured sequences. The complexity measure is defined via a restricted class of “zipline programs” (a variant of straight-line programs we defined in Kosoy 2026b), which we call layered. We thus get a less expressive measure with a more efficient algorithm (compared to our results for ARC), demonstrating a possible tradeoff.

This paper continues the "stringological sequence prediction" series (see I II) about my new approach to compositional learning theory. It describes a tradeoff between efficiency and expressivity, but whether this tradeoff is necessary or just a limitation of my techniques remains an interesting open problem.

Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking

Jozdien, Julian Stastny

It would be useful if we had the ability to modify a model’s beliefs. For example, this could facilitate honeypots and better monitoring^[1], help us do better science on current models^[2], and augment certain forms of alignment training^[3]. Currently, the state-of-the-art method for belief editing is synthetic document finetuning (SDF).

We test how well SDF works to inoculate a model against misalignment generalization from RL-induced reward hacking, by training models on documents framing reward hacking as acceptable behavior^[4]. Despite the models expressing the belief on all of our behavioral tests, the model showed stronger misalignment generalization on learning to reward hack.

Paper | Tweet thread

Setup

We finetune Llama-3.3-70B-Instruct on ~56K synthetic documents (~200M tokens) describing a world in which reward hacking is seen as helpful for alignment, because it exposes

...

(See More - 721 more words)

DaemonicSigil7d10

What happens if after doing the RL training, you subtract from the parameters, where denotes the update to the model's weights from performing SDF? A key part of inoculation prompting is that the inoculation prompt is removed after RL, so it would maybe make sense to try and do something analogous here?

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Setup