Minimum required pacing: finish aligning it before you train the next one

orthonormal

Let's say we coordinate to pace the frontier (rather than pausing outright or failing to coordinate). What should the standard of pacing be?

My proposal for a minimum requirement: a lab cannot train the next generation of models (in terms of capabilities) until they have adequately^[1] resolved all significant alignment issues in the present generation of models.

You don't launch a jet that goes at Mach 3 before you've figured out why your Mach 2 jet keeps crashing^[2]. With any other kind of dangerous technology, we wouldn't accept "shrug and move on".

This proposal may appeal to many prosaic-alignment optimists and pessimists, because they usually differ on how easy it would be to fix current alignment issues with current techniques.

Prosaic-alignment optimists might expect that a moderate coordinated slowdown would suffice to adequately resolve alignment issues at each generation.
Prosaic-alignment pessimists might expect it to result in an extended pause, buying time to discover new fundamental approaches to aligned AI.

And it creates much better incentives for labs: in order to advance in capabilities, they'd have to push the frontier on alignment and interpretability techniques.

How to judge adequate alignment?

What process might we use to judge whether a lab has adequately resolved its alignment issues for a generation of models? Here's my vague, qualitative proposal:

Multiple independent alignment organizations examine and red-team the newest model at many checkpoints throughout training, with full access to the data and with the best tools available.
- Alignment issues need to be taken seriously whether there's a metric for it or whether something is qualitatively off, whether the model has a bad attribute or whether they can't robustly conclude it has a good one, etc.
- Alignment issues early in training may not be a showstopper, but there must be a robust safety case for why the later stages of training will resolve those issues.
A model of a new generation is ready for restricted deployment only when the alignment organizations are satisfied with it.
- Restricted deployment starts with a group of trusted users doing tasks that don't contribute to model training and monitoring. The alignment organizations monitor restricted deployment for any new issues.
- If any significant alignment issue arises during restricted deployment:
  - Pull the model.
  - Do a root cause analysis of the issue and why it was not caught during training.
  - Figure out a principled way to fix the disease rather than the symptom.
  - Train the next model of the current generation accordingly, and start again at the top.
A model of a new generation is ready for deployment (internal or external) only when it's survived a significant period of restricted deployment without any issues arising.
- If any significant alignment issue arises during broader deployment, hit reset as outlined in the previous section.
If no alignment issues have come up over a significant stretch of deployment time, only then can the lab begin training a next-generation model.

I still wouldn't say that this is fully adequate, but I'd treat it as a minimum requirement.

This proposal is incomplete; you can help by expanding it

This isn't a concrete proposal, just a heuristic one. A few of the big missing pieces, aside from standard implementation problems:^[3]

How do we define "a generation of models", within one lab or across several labs?
What should we do if an intervention to fix an alignment issue also significantly increases capabilities?
How would we define and enforce an agreement, if we're unable to steer solely by predefined metrics?
When is an alignment issue big enough that we should halt the model? There will be obvious pressure from the lab to set the threshold high.
Added 26-10-12, thanks to @Florian_Dietz for this comment: If each step up in capabilities takes longer, the labs will be incentivized to take bigger steps.

^{^}
And no, "scores better than previous models on our metrics" is not adequate. If a user points out a weird but alarming thing that wasn't noticed before deployment, that is a critical failure in the process as well as in the model.
^{^}
Sure, maybe your Mach 2 jet crashes for a reason that stops being a problem at Mach 3. But it would be crazy to assume that!
^{^}
And aside from arguments that no version of this would be adequate, and that we have to pause rather than pace. I personally think this is likely the case, but I think it's worth proposing a minimum standard for pacing regardless.

Pretraining scaling is starting to slow down, both in terms of compute and model size. The mad rush of 2025-2026 is done, and the models targeting 2027 hardware are probably already pretrained (though the work on their misalignment is then only getting started). In 2026-2028, scaling is about 2x slower than in 2025-2026, and then in 2028-2032 it gets 2x slower yet again.

So "pacing" will be happening naturally in some sense. The problem is that the models of late 2026 are passing the level of human capabilities in many ways, so even slower scaling could get significantly harder to handle. And Meta, Google, and SpaceX currently have what seems to be well-made small models (somewhere between Sonnet and Opus scale-wise) and sufficient resources/ambition to plausibly catch up with OpenAI and Anthropic in 2027 at least in terms of model scale (with big models that are not so broken that it ruins their capabilities). To the extent scale is a source of misalignment difficulties that require operational excellence to sweep under the rug, they might have a hard time in 2027.

And then slow-learning prosaic RSI of automated RL task/env creation in 2027-2028 probably introduces new misalignment/control issues, even if it doesn't manifest as acceleration or a step-change increase in capabilities, and merely fixes jaggedness by automatically filling in all ordinary capability gaps as they come up. Which is probably economically transformative, makes the LLM/RL paradigm more centrally AGI (by automating learning, in a very clunky way), enables useful humanoid robotics and opens up a path to a robot-industrial explosion, but isn't necessarily a raw capability jump per se. (Continual learning in a strong sense is another matter and probably quickly leads to strong architecture-rewriting RSI, but I don't know of a way to time its invention, other than to say that LLMs of 2027-2028 will be taking a stab at it. Like with math problems currently being slaughtered in droves, it could be "easy enough" in some sense for the LLM/RL paradigm to defeat quickly, but not necessarily.)

Can you spell out the implications of your predictions on this proposal?

Hardware/economics/industry will already be coincidentally pacing the scaling, starting pretty much right now, for reasons unrelated to (the risk of) AI incidents. To the extent capabilities follow from scale, pacing in other ways (using misalignment-related triggers and coordination) might be less necessary, the situation gets better compared to 2025-2026 on its own. But other sources of capabilities or novel misalignment issues will be appearing at the same time, including new AI companies transitioning into the more hazardous higher-scale regime very soon (they won't necessarily be able to fully benefit from the learnings of the AI companies that got there earlier).

So there's a pause-adjacent story in the technology itself, that's not at all about deliberate pacing or coordination, but might have similar prosaic alignment/control effects (though inherely weaker ones than a strong ban/pause would).

A note on why I wrote this post despite thinking that we should pause rather than pace:

I frequently talk to people at labs (specifically Anthropic) who are enthusiastic about pacing but who aren't convinced by the arguments for a pause. In one such conversation, I said "well, you're not even meeting this basic standard", which the other person immediately agreed was valid.

I give at least 80% credence to the IABIED thesis, under which "pacing the frontier" is misguided and we must halt further progress under the current paradigm soon. However, I'm proposing this minimum pacing standard in good faith, and I think it would be good if pacing proponents adopted (and strengthened) it.

Here's my thought process:

If pacing is misguided but humanity chooses it anyway, it still helps for there to be a nontrivial standard for it, since that buys us some of the time that pausing would (in fact, probably significantly more time than optimists expect it to); in that time, we might discover fundamental insights on how to build AI in a less doomed way, or we might coordinate the political will to fully pause.
But I also leave some probability for being fundamentally mistaken, and in some of those scenarios pacing might work if we have a good standard for it. That's why this is a good-faith proposal.
Also, this drives a wedge between people who say "pacing" because they want to put on some brakes and people who say "pacing" because they want a rhetorical shield while they hit the gas. If we make the latter camp explicitly say that it's okay to train more capable models while we still have no idea how to align weaker ones, then the former camp will realize they're false allies.

I still strongly support pausing over pacing. But proposing a minimum standard for pacing helps advocates of pausing more than it undercuts us.

As an analogy, a nuclear disarmament advocate isn't hurting their movement by saying "while it's unacceptable that you still have thousands of nukes pointed at each other, can you at least fix the bugs in the warning systems that have almost caused accidental nuclear wars?" Sure, it might weaken one objection to keeping a nuclear arsenal. But it would also buy more time in expectation before the debate becomes moot.