When I try to picture the current AI tech stack killing everyone, I still don't really see it. No, not even if it is successfully assembled into an RSI loop that rapidly reaches that tech stack's pinnacle.

I don't think we're in the crunch time.

All things considered, DL still has terrible sample-efficiency and generalization. It needs a ton of data to develop a given capability, much more data than humans, and there's some sense in which what it learns are the "most shallow" capabilities consistent with a given data distribution. Not literal lookup tables, but it wants to stay as close to lookup tables as possible.

What can we say about its trajectory? AI progress is driven by parameter scaling, algorithmic improvements, data improvements, and RLVR. Of those:

• Parameter scaling improves capabilities generally: larger models are in fact better at ~everything, including various "illegible" skills. However, this type of scaling can't be pushed much farther much faster, and another general Opus 3 -> Fable 5 jump is not going to birth a strategic supergenius.^[1]
• Algorithmic improvements are not really a big contributor to things, see this and this.
• Data improvements basically means hiring skilled human labor to generate annotated data traces. This also doesn't really scale much farther much faster, and there are no runaway loops possible this way: all capabilities are improved and encoded in a basically "manual" manner.
• RLVR, to wit, only works in domains with easily verifiable rewards. Capabilities elicited this way then likewise refuse to generalize to other domains, and may even degrade scarier general-purpose capabilities.

Now, there are a bunch of counter-points one may raise here:

• "Aha, but DL research is itself an easily verifiable domain!"
  • Sure. DL research can produce a model that is really good at doing the sort of thing that DL researchers are doing right now, improving models in the ways they are currently being improved at superhuman speeds. But this results in the Singularity if and only if the no-RSI trajectory already points to it.
  • That is, "does in-paradigm RSI lead to the Singularity?" reduces to "does the current AI paradigm suffice for human-unbounded improvements in all domains?".
  • And I argue the answer to the latter is "no". Unbounded progress is only possible for easily verifiable tasks, and whereas scaling data and compute theoretically provides general improvements, the pace of those is hitched to humans.
• "The current capabilities are already sufficient for transformative changes across the economy, once they diffuse, including speeding up compute buildout."
  • Yes. @Vladimir_Nesov-style prosaic-RSI loop is baked-in. But it's kinda really slow.
  • As in: Yes, it seems plausible that the current paradigm scales to somewhat-general researchers and engineers. But the specifics don't seem to suggest that the resultant process will be much faster than humanity: it is theoretically general but not superhuman.
• "Unbounded RLVR-derived progress is sufficient for the Singularity." We'll need to break this one down.
  • Roughly speaking, the progress avenues here can be split into DL research, performance in custom RL environments, hacking, and math.
  • "DL research" and "performance in custom RL environments" have been mostly discussed above. This leads to the "in-paradigm"/"prosaic" RSI.
  • Hacking: Its pinnacle is just not that scary. It may lead to a very bad global disaster, but it's not literally world-ending, and the median scenario is probably not even that bad.
  • Math: There's an extended discussion to be had there, which I'll separate out.

On math, the question is: does the current paradigm/RLVR in fact suffice for general-purpose superhuman math research?

So far, all AI-generated math results appear to be "conceptually stale". Yes, even the Navier-Stokes one.^[2] They're all counterexamples or other such constructions whose assembly is extremely laborious and tricky and hard to get right (no human did this, despite $1M on the line!), but which are ultimately unenlightening, done using known approaches out of known pieces in predictable ways.^[3] So far, there really appear to be no "move 37" equivalents there.

Now, one may argue that this is going to go away with just more RLVR. Perhaps there is no real qualitative difference despite what mathematicians protest, and those "conceptually novel" discoveries are just the same "prosaic" discoveries at a big-enough scale.

Naively reasoning top-down, I would be inclined to agree. Like, suppose you fixed some well-defined mathematical problem resolving which required doing conceptually novel work, put an LLM in an RL loop in front of it, and scaled the inference-time compute up the wazoo. Why wouldn't it work? It should work.

... except, of course, that the compute scaling matters. In theory, a random proof generator, iterated enough times, can also prove anything. But it matters whether the algorithmic pathway by which you're attempting to reach the target is the sort that can do it in polynomial time. Arguing this differently, if there were a qualitative ground-true difference between conceptually novel vs. stale results, it would be at least possible that LLMs are just not the type of entity that can do "conceptually novel" stuff, or which are at least stubbornly bad at it.

And the negative space is drawing my attention now. I don't actually think that "conceptually novel" contributions have to be "large-scale"; novel insights can be "small" yet still feel novel. The fact that LLMs contributed a "conceptually stale" result as big as Navier-Stokes before they yielded even a small conceptual insight sure seems like it's hinting at something.

I don't think this is mathematicians coping. Indeed, what this seems to be is a special case of the general issue that people trying to do conceptually novel work with LLMs, including outside mathematics, run into,^[4] which is that LLMs appear completely inept at it.

This leads to a branching:

• If the above is correct, the crux is that LLM improvements in doing conceptually novel work are not human-unbounded. As in, they may still be able to improve at it if AGI labs hire enough scientists to distill research taste from, but the pace is going to be limited by data progress, and it'll also top out at top-human level. This puts us into the prosaic-RSI regime.
  • On this branch, the pinnacle of math RLVR is probably still pretty boring. "Leanslop", as people say. It can provide modest boosts to theorywork, but it's not going to do anything really crazy. As Terence Tao keeps insisting, an engine for boringly resolving some conjectures y/n is just not all that fruitful.
• If the above is incorrect, implications may go beyond math. If LLMs' ability to do conceptually novel work in math can scale unboundedly, then LLMs' ability to do conceptually novel work period may scale unboundedly. In which case prosaic-RSI regime may quickly ratchet up to the Singularity as LLMs become capable of inventing non-DL-paradigm AI improvements.
  • Note that "DL is bad at generalization" doesn't actually help here much. As in, sure, perhaps LLMs' math-specialized and DL-research specialized research tastes aren't going to generalize to non-math and non-DL domains. But their object-level high-taste conceptually novel math/DL research is potentially sufficient to shift us outside the DL paradigm by itself, into AI paradigms which do generalize better.

... but, weighing all of the above, I think we should be updating away from the latter scenario. Again, all RL-on-math has done so far is conceptually stale, despite how far it was pushed, and vibe checks on LLMs keep being off.

(TBC, I don't think "conceptually novel" vs. "conceptually stale" is what really cleaves reality at the joints here. But I do think there's some sort of meaningful qualitative difference at play regarding the sorts of cognitive tasks LLMs are capable of vs. not.)

Summing up, it seems clear that the current AI paradigm is capable of:

• Unboundedly superhuman hacking, potentially sufficient to cause a global disaster.
• Unboundedly superhuman in-paradigm DL research, likely sufficient to set off prosaic RSI.
• Unboundedly superhuman but "workmanlike" math research.
• Unboundedly superhuman performance in perhaps some other important domains, like some bio stuff.

All of those have significant, trillion-dollar implications. The current AI tech stack is a major, revolutionary technology, potentially on the scale of the industrial revolution.

But there is a vast, vast gulf between trifles like these and the Singularity,^[5] and I don't see the current AI stack as being Singularity-capable. It may be, but there's been approximately zero new evidence towards this since 2022.^[6]

I think the meta-level error people are making is anchoring too much to surface-level signals like "oh, it's pretty good at running DL experiments!" or "oh, it can find zero-days better than the entire rest of the human cybersecurity industry!" or "oh, it resolved one of the most important questions in math!", instead of trying to mechanistically model what's happening in as much detail as possible, and what the specific mixes of positive and negative results we observe tell us about the structure of intelligence and LLMs' place in it. (To be fair, those surface-level signals are very flashy and distracting. I can't say I'm entirely unmoved.)

I was debating whether to post this, because (a) maybe it's best to let people convince themselves the sky is falling so that drastic regulations can be urgently pushed through because there may not be another good moment for this, and (b) it may be best not to do anything that might sway AGI labs' conviction that they're on the right track lest they go looking for conceptually novel AI insights themselves. But I'll err on the side of candor.

So: on balance, I do not think this is the endgame.

1. ^{^}

   Note that you have to measure by the worst areas of improvement along this jump, i. e., the ones improvements along which come purely from pretraining scaling and not from any other source of improvement.

2. ^{^}

   Incidentally, I'll be assuming here that everything happened the way OpenAI says: that they did not even accidentally train on Levent and Tristan's data and that they really have a new model that produced this result and a bunch of other novel results in a basically autonomous manner.

   I still have my doubts, but this looks more plausible now.

3. ^{^}

   To be clear, the ability to do that still makes LLMs a powerful emerging technology, and superhuman at some important cognitive tasks. But "yet another powerful emerging technology" and "superhuman at this well-scoped cognitive task" is a far cry from "the Singularity". Stockfish isn't omnicide-capable.

4. ^{^}

   E. g., me, @johnswentworth and @Steven Byrnes last I checked, this poor guy, and I guess even Anthropic researchers?

5. ^{^}

   As in: the reference class of the emergence of homo sapiens, rather than "mere" industrial revolution.

6. ^{^}

   Indeed, if anything, I've updated the other way. In the GPT-3 times, I was afraid of GPT-4. In the modern times, I don't expect that much of GPT-7. GPT-5 to GPT-6 sure wasn't all that much.