training on a purely predictive loss should, even in the limit, give you a predictor, not an agent

I fully agree. But 

a) Many users would immediately tell that predictor "predict what an intelligent agent would do to pursue this goal!" and all of the standard worries would re-occur. 

b) This is similar to what we are actively doing, applying RL to these systems to make them effective agents.

Both of these re-introduce all of the standard problems. The predictor is now an agent. Strong predictions of what an agent should do include things like instrumental convergence toward power-seeking, incorrigibility, goal drift, reasoning about itself and its "real" goals and discovering misalignments, etc.

There are many other interesting points here, but I won't try to address more!

I will say that I agree with the content of everything you say, but not the relatively optimistic implied tone. Your list of to-dos sound mostly unlikely to be done well. I may have less faith in institutions and social dynamics. I'm afraid we'll just rush and and make crucial mistakes, so we'll fail even if alignment was only in between steam engine and apollo levels.

This is not inevitable! If we can clarify why alignment is hard and how we're likely to fail, seeing those futures can prevent them from happening - if we see them early enough and clearly enough to convince the relevant decision-makers to make better choices.

Oh, I didn't know the AI village agents had been set a goal including raising money. The goals I'd seen might've benefitted from a budget but weren't directly about money. But yes they would've been delighted if the models raised a bunch of money to succeed. But not if they took over the world.

Your point is growing on me. There are many caveats and ways this could easily fail in the future, but the fact that they mostly follow dev/user intent right now is not to be shrugged off.

I didnt' really review empirical evidence for instrumental convergence in current-gen models in that post. It's about new concerns for smarter models. I think models evading shutdown was primarily demonstrated in Anthropic's "agentic misalignment" work. But there are valid questions of whether those models were actually following user and/or dev intent. I actually think they were, now that I think about it. This video with Neel Nanda goes into the logic.

I think you're getting a lot of pushback because models currently do pretty clearly sometimes not follow either user or developer intent. Nobody wanted Gemini to repeatedly lie to me yesterday, but it did anyway. It's pretty clear how it misinterpreted/was mistrained toward dev intent, and misinterpreted my intent based on that training. It didn't do what anyone wanted. But would failures like that be bad enough to count as severe misalignments?

One of my in-queue writing projects is trying to analyze whether occasionally goingoff target will have disastrous effects in a superhuman intelligence, or whether mostly following dev intent is enough, based on self-monitoring mechanisms. I really think nobody has much clue about this at this point, and we should collectively try to get one.

Instrumental convergence for seeking power. Almost any problem can be solved better or more certainly if you have more resources to devote to it. This can range from just asking for help to taking over the world.

And the tales about how it goes wrong are hardly logical proof you shouldn't do it. There's no law of the universe saying you can't do good things (by whateer criteria you have) by seizing power.

This has nothing to do with mesa-optimization. It's in the broad area of alignment misgeneralization. We train them to do something, then are surprised and dismayed when either we got our training set or our goals somewhat wrong, and didn't anticipate what it would look like taken to its logical conclusion (probably because we couldn't predict the logical conclusion of some training on a limited set of data when it's generalized to very different situations; see that post I linked for elaboration)

We're not preventing powerseeking from ratings or any other alignment strategy; see my other comment.

It does show up already. In evals, models evade shutdown to accomplish their goals.

The power-seeking type of instrumental convergence shows up less because it's so obviously not a good strategy for current models, since they're so incompetent as agents. But I'm not sure it shows up never - are you? I have a half memory of some eval claiming power-seeking.

The AI village would be one place to ask this question, because those models are given pretty open-ended goals and a lot of compute time to pursue them. Has it ever occurred to a model/agent there to get more money or compute time as a means to accomplish a goal?

Actually when I frame it like that, it seems even more obvious that if they haven't thought of that yet, smarter versions will.

The models seem pretty smart, but their incompetence as agents springs in part from how bad they are at reasoning about causes and effects. So I'm assuming they don't do it a lot because they're still bad at causal reasoning and aren't frequently given goals where it would make any sense to try powerseeking strategies.

Instrumental convergence is just a logical result of being good at causal reasoning. Preventing them from thinking of that would require special training. I've read pretty much everything published by labs on their training techniques, and they've never mentioned training against power-seeking specifically to my memory (Claude's HHH criteria do work against unethical power-seeking, but not ethical types like earning or asking for money to rent more compute...). So I'm assuming that they don't do it because they're not smart enough, not because they're somehow immune to that type of logic because they want to fulfill user intent. Users would love it if their agents could go out and earn money to rent compute to do a better job at pursuing the goals the users gave them.

I don't think they're reached that threshold yet. The could but the pressures and skills to do it well or often aren't there yet. The pressures I addressed in my other comment in this sub-thread; this is to the skills. They reason a lot, but not nearly as well or completely as people do. They reason mostly "in straight lines" whereas humans use lots more hierarchy and strategy. See this new paper, which exactly sums up the hypothesis I've been developing about what humans do and LLMs still don't: Cognitive Foundations for Reasoning and Their Manifestation in LLMs.

They don't think about gaining power very often (I don't think it's never) because it's not a big direction in their RL training set or the base training.

That might make you optimistic that they'll never think about gaining power if we keep training them similarly.

But it shouldn't. Because we will also keep training and designing them to be better at goal-directed reasoning. This is necessary for doing multi-step or complex tasks, which we really want them to do.

But this trains them to be good at causal reasoning. That's when the inexorable logic of instrumental convergence kicks in.

In short: they're not smart enough yet for that to be relevant. But they will be, and it will be.

At a minimum we'll need new training to keep them from doing that. But trying to make something smarter and smarter while keeping it from thinking about some basic facts about reality sounds like a losing bet without some good specific plans.

It seems more straightforward to say that this scopes the training, preventing it from spreading. Including the prompt that accurately describes the training set is making the training more specific to those instructions. That training thereby applies less to the whole space.

Maybe that's what you mean by your first description, and are dismissing it, but I don't see why. It also seems consistent with the second "reward hacking persona" explanation; that persona is trained to apply in general if you don't have the specific instructions to scope when you want it.

It seems pretty clear that this wouldn't help if the data is clean; it would just confuse the model by prompting it to do one thing and teaching it to do a semanticall completely different thing, NOT reward hack.

Your use of "contrary to user instructions/intent" seems wrong if I'm understanding, and I mention it because the difference seems nontrivial and pretty critical to recognize for broader alignment work. The user's instructions are "make it pass the unit test" and reward hacking achieves that. But the user's intent was different than the instructions, to make it pass unit tests for the right reasons - but they didn't say that. So it behaves in accord with instructions but contrary to intent. Right? I think that's a difference that makes a difference when we try to reason through why models do things we don't like.

Doesn't the same argument you make for behaviorist RL failing apply to any non-perfect non-behaviorist RL?

"Follow rules unless you can get away with it" seems to also be an apt description of the non-behaviorist setup's true reward rule. Getting away with it also applies to faking the internal signature of sincerity used for the non-behaviorist reward model, as well as evading the notice of external judges.

So we're still stuck hoping that the simpler generalization wins out and stays dominant even after the system thoroughly understands itself and probably knows it could evade whatever that internal signal is. This is essentially the problem of wireheading, which I regard as largely unsolved since reasonable-seeming opinions differ dramatically.

Using non-behaviorist RL still seems like an improvement on purely behavioral RL. But there's a lot left to understand, as I think you'd agree.

This thought hadn't occurred to me even after twice all the way through the Self-Dialogue longer version of this argument, so your work at refining the argument might've been critical in jogging it loose in my brain.

This is also encouraging because OpenAI is making some actual claims about safety procedures. Sure they could walk it back pretty easily, but it does indicate that at least as of now they likely intend to try to maintain a faithful CoT.

You assumed no faithful CoT in What goals will AIs have?, suggesting that you expected OpenAI to give up on it. That's concerning given your familiarity with their culture. Of course they still might easily go that way if there's a substantial alignment tax for maintaining faithful CoT, but this is at least nice to see.

This might be the most valuable article on alignment yet written, IMO. I don't have enough upvotes. I realize this sounds like hyperbole, so let me explain why I think this.

This is so valuable because of the effort you've put into a gears-level model of the AGI at the relevant point. The relevant point is the first time the system has enough intelligence and self-awareness to understand and therefore "lock in" its goals (and around the same point, the intelligence to escape human control if it decides to).

Of course this work builds on a lot of other important work in the past. It might be the most valuable so far because it's now possible (with sufficient effort) to make gears-level models of the crucial first AGI systems that are close enough to allow correct detailed conclusions about what goals they'd wind up locking in.

If this gears-level model winds up being wrong in important ways, I think the work is still well worthwhile; it's creating and sharing a model of AGI, and practicing working through that model to determine what goals it would settle on.

I actually think the question of which of those goals can't be answered given the premise. I think we need more detail about the architecture and training to have much of a guess about what goals would wind up dominating (although strictly following developers intent or closely capturing "the spec" do seem quite unlikely in the scenario you've presented).

So I think this model doesn't yet contain enough gears to allow predicting its behavior (in terms of what goals win out and get locked in or reflectively stable).

Nonetheless, I think this is the work that is most lacking in the field right now: getting down to specifics about the type of systems most likely to become our first takeover-capable AGIs.

My work is attempting to do the same thing. Seven sources of goals in LLM agents lays out the same problem you present here, while System 2 Alignment works toward answering it.

I'll leave more object-level discussion in a separate comment.