To the extent we believe more advanced training and control techniques will lead to alignment of agents capable enough to strategically make successor agents -- and be able to solve inner alignment as a convergent instrumental goal -- we must also consider that inner alignment for successor systems can be solved much easier than for humans, as the prior AIs can be embedded in the successor. The entire (likely much smaller) prior model can be run many times more than the successor model, to help MCTS whatever plans it's considering in the context of the goals of the designer model.

I've been thinking about which parts of AI 2027 are the weakest, and this seems like the biggest gap.^[1] Given this scenario otherwise seems non-ridiculous, we should have a fairly ambitious outer alignment plan meant to compliment it, otherwise it seems extraordinarily unlikely that the convergent alignment research would be useful to us humans.

Since modern training hasn't solved inner alignment, and control techniques do not make claims on inner alignment, it seems like the default path (even in the most optimistic case scenario) would be successfully aligning world-changing models only to the not-known-to-be-bad but randomly-rolled values of the system doing the alignment research, which seems nearly useless.

I'd like to zoom in on one particular element of their proposal as well: "Why is it aligned? Whereas Safer-1 had basically the same training as Agent-4, Safer-2 has a new training method that actually incentivizes the right goals and principles instead of merely appearing to. They were able to design this training method by rapidly iterating through many different ideas, and contrasting the intended result (the Spec) to the actual result (the thoughts in the chain of thought)."

This approach is unlikely to work for the very obvious reason that only some flaws will be apparent. Let's imagine half of the potential alignment issues are sufficiently obvious you could notice them before anything bad happens through the combination of chain of thought observation, model activation tracing, and adversarial testing. If you need to cycle through 99 models and only get one that meets your requirements on model 100, you should expect only a 1% chance you also accidentally also avoided all of the non-obvious problems. I'm not sure how seriously to take this model of safety developmen...

Posts

Wikitag Contributions

Comments