I feel like I'm supposed to be anxious from the news about all the AI hacking incidents and the egregious misalignment going on, coupled with them clearly being on-track to become superhuman at math and hacking. But I find that I'm pretty zen about it.

On reflection, I suppose it's due to a confluence of three things:

• The current AI paradigm seems poised to make models superhumanly capable at hacking and formal math way before they become even upper-human-level at something more dangerous, like long-term strategy or open-ended innovation.
• The current AI paradigm's alignment methods appear to be paper-thin after all, incapable of covering up egregious misalignment even of the "commit a felony to do well at an eval" level. We are going to get so many warning shots.
• The world's current reactions to said "warning shots" appear promising.

Taken together, it kind of seems unlikely that this trajectory results in extinction. AI labs are going to develop AIs able and willing to inflict mass damage, but they won't succeed at making those AIs smart enough not to inflict said damage for idiotic short-sighted reasons.^[1] Neither the capability training nor the deception training (sorry, "alignment training") generalizes far enough; wisdom is not very RLVR-able. And whenever they do this stuff, the world is going to react... appropriately.

My understanding is that the attorney generals' cease-and-desist request (regarding cyberattack evals, in the linked letter) is not a legally enforceable order... But such an order could be made. What happened around Fable/Mythos was likewise a good demonstration of the swift and casual ease with which the government could make decisions that are utterly ruinous for AI labs' plans. And that's all it'd take: a model goes out of control, inflicts sufficient damage, the USG freaks out and, with the wave of a pen, bans training for an indefinite duration.

Presumably China would then follow suit. Or, well... Either the AI labs there look at what happened in the US and decide not to repeat the US labs' mistakes, or the CCP looks at what happened and acts pre-emptively to stop them, or they do release an open-weight model and cause worldwide chaos via the Hackening (be that due to stupidity or on purpose). That latter option would suck enormously, but it is no extinction, and it'd sure function as a "warning shot".

I'm making it sound like it all happens "by default", and perhaps some of it does, but I'm also pricing in the work of AI Safety governance/policy people. As all of this is happening, they and their arguments would hopefully gain more sway, enabling them to steer towards some reasonably good outcomes.

Do I actually buy the above optimistic vision?^[2] Not quite:

1. The basic analysis of the dynamics may be wrong. I say "wisdom is not very RLVR-able", and I think that's right. But perhaps greater wisdom is something you get for free with more parameters, and perhaps you only need to cross a given threshold of wisdom for AIs to completely stop being misaligned in this stupidly visible manner, and perhaps that threshold is reachable within the several OoMs of raw scaling we still have left. So perhaps we get misaligned superhuman-hacker superhuman-mathematician schemers that are not also bumbling fools tipping off their hand to cheat at homework. Vibes-wise I don't expect the current AI methods to get us there, but those vibes can be miscalibrated.
2. It's possible that the AI labs are now going to differentially prioritize training for things other than hacking, such as deception training or strategic-awareness training (or, you know, something that sounds sane on paper, but is isomorphic to those), in order to ensure they can keep going for longer and so that the inevitable disaster when those methods fail inflicts as much suffering as possible.^[3] Perhaps this works well enough the disaster is world-ending. Probably not, though?
3. It's possible, nay inevitable, that there are going to be new paradigm-changing breakthroughs in AI research. Such breakthroughs may inherently shift the capability balance, making it less skewed in favor of visible but non-omnicidal disasters, or perhaps straight-up enabling the orthodox foom'n'doom scenario.
4. If the USG bans private companies from doing frontier AI training, it's possible, even likely, that they're going to then nationalize the technology instead of banning it fully. They'll then keep on doing this kind of research outside the public eye. Perhaps their work will then keep blowing up badly enough that the project gets curtailed anyway, or perhaps it reduces to options (2) or (3) above.
   • Also, this scenario likely leads to enormous power concentration in the hands of the government, making eternal-dystopia risks highly salient. I don't think anything except an international AGI ban can get us out of that.
5. It's possible that the world's ultimate response is less properly alarmed than it currently appears, and/or that the AI labs will succeed at frog-boiling the matter.
6. Worth restating that the Hackening is going to suck enormously if enacted at its full catastrophic scale.

All that said, my net update off of all of that is still neutral-leaning-positive. (As far as the medium-term future goes, I mean, i. e. the next ~5 years. None of this is by itself a long-term solution to the AGI risk, of course.)

Curious where people think this might be off the mark.

1. ^{^}

   As general capabilities increase, AIs are going to get progressively less idiotic and short-sighted, but, at any given step, their capacity to inflict damage is going to be ahead of their ability to determine when it's strategically appropriate to do so.

2. ^{^}

   Well, "optimistic", what with possibly millions of deaths in the wake of all this.

3. ^{^}

   Is that too harsh and passive-aggressive? Maybe. I am not currently feeling very charitable towards AI labs.