Left a general comment on the post, engaging with the object-level models here since it seems like a more appropriate place for it.
- I long expected there to come a point where the learning algorithm groks some ability to actually generalize, and rapidly unlocks a lot of other capabilities.
- We see lab and METR employees being surprised by recent capability gains, which is evidence for that view.
FWIW, I think everything we've seen coming out of the labs is consistent with "in-model" surprises: surprisingly fast pace of advancement that is nevertheless "in-paradigm" and roughly continuous. They didn't expect gpt-hold-my-beer to resolve Navier-Stokes, didn't expect RL-on-math to work this well this fast. That's fair enough, I also didn't expect this / hoped for not-that. Presumably there's a bunch of similar stuff going on regarding SWE/hacking, maybe regarding some DL-research uplift. But it's not ultimately breaching the "box" into which we can mentally slot LLMs. (Neither do mathematicians' takes on hold-my-beer's proofs or Anthropic employees' comments in the METR report.)
I expect that if an LLM learned some importantly novel ability to generalize, the thing I think they're missing, it would look dramatically different. ... Not sure how, though. Maybe the AGI labs would be freaking out more, or there'd be no externally visible signs whatsoever because the AI would cover its tracks well, or...
Well, I suppose my point is simply that the current observations seems well-explained by "surprisingly rapid but still on-track" progress, with no reason to postulate more surprising model violations.
First, my subjective sense is that AI open-ended judgment is rising, even if it's still pretty bad. If I just extrapolate that trend, even if it's markedly behind verifiable tasks, and I imagine "an AI that is superhuman at all verifiable tasks", I expect that AI to be significantly closer to human-level open-ended judgment.
I want to ask for examples, but I expect studying those isn't going to be very enlightening (also I'm guessing some of them are in your earlier post about game ideas). As such, I'm just mentally noting down this report: it would be an important sign if such reports start accumulating.
Second, an AI with "subhuman judgment but superhuman skill at every verifiable domain" (including the ability to fine-tune subagents on new skills they need), still seems very dangerous.
Maybe. I primarily expect that "subhuman judgement" is in fact really crippling, such that what this would entail is them "mostly doing kinda random derpy stuff that doesn't tie together very well". Maybe there'd be billions of damages, maybe millions of deaths as they collapse the economy/hack everything/do something else majorly bad. But I don't think this is takeover-complete.
See also this and this specifically.
It may be, though, sure.
Three, it's plausible that "subhuman judgment + superhuman-verified-skills, guided by occasional expert human judgment" is sufficient to rapidly explore the space of research necessary to reach superhuman judgment, and then we are up against true ASI.
Possibly. I don't know though, I feel very confused about this. Like, if I had a bunch of these subhuman-judgement superhuman-verified-skills minions and infinite tokens, would I be able to 10x+ my conceptual research? And how would I do it? I ask myself that periodically, and I don't really have great ideas. (I have an unfinished dialogue with Tsvi where we go over this some, perhaps it's worth getting back to.)
Note though that this still leads to us requiring a breakthrough, a pretty hard-to-predict event. Even if it's sped up by a factor of ?, it IMO still doesn't produce modern "instantaneously short" timelines.
Any lab employees reading this getting ideas about other paradigms to overcome a hypothetical bottleneck... note this situation is good! If there's a conceptual bottleneck, it means you get to keep shipping valuable products without destroying the world or even just spiralling outside your control
(Whenever I write about this stuff, I also notice an impulse in me to make this exact imploration, but I think it's fruitless. I think they'd just standardly go "if not me, then someone else", "if I can think of this, so can someone else", and try to do their best to figure out the conceptual breakthrough anyway, even if it it's in fact very counterfactual.)