(Talk given at an event on Sunday 28th of June. TurnTrout is responsible for the talk, David Lambert edited the transcript.

If you're a curated author and interested in giving a 5-min talk, which will then be transcribed and edited, sign up here.)

TurnTrout: Power and power-seeking play a big part in my thinking about AI alignment, and I think it's also an interesting topic more generally.

Why is this a big deal? Why might we want to think about what power is? What is it, exactly, that people are thinking of when they consider someone as powerful?

Well, it seems like a lot of alignment failures are examples of this power-seeking where the agent is trying to become more capable of achieving its goals, whether that is getting out of its box, taking over the world, or even just refusing correction or a shutdown.

Une image contenant texte, carte

Description générée automatiquement

If we tie these together, we have something I've called the catastrophic convergence conjecture, which is that if the goals aren't aligned and it causes a catastrophe, it is because of power-seeking.

But I think that when people are first considering alignment, they think to themselves, "What's the big deal? You gave it a weird goal, and it does weird stuff. We’ll just fix that.”

So why exactly do unaligned goal maximizers tend to cause catastrophes? I think it's because of this power seeking. Let me explain.

The way I think about power is as the ability to achieve goals in general.

In the literature, this is like the dispositional power-to notion.

Whereas in the past, people thought "Well, in terms of causality, are the agent's actions necessary and/or sufficient to cause a wider range of outcomes?", here, I think it's best thought of as your average ability to optimize a wide range of different goals. So if you formalize this as your average optimal value in, say, a Markov decision process (MDP), there's a lot of nice properties and you can prove that, at least in certain situations, it links up with instrumental convergence. Power seeking and instrumental convergence are very closely related.

But there's a catch here. We're talking about average optimal value. This can be pretty weird. Let's say you're in the unfortunate situation of having a dozen soldiers about to shoot you. How powerful are you according to average optimal value? Well, average optimal value is still probably quite high.

Une image contenant texte

Description générée automatiquement

There's probably an...

LessWrong Event Transcripts

LessWrong Event Transcripts