TLDR: Systems which locally maximal influence can be described as VNM decision-makers.

There are at least 3 different motivations leading to the concept of "agent" in the context of AI alignment:

1. The sort of system we are concerned about (i.e. which poses risk)
2. The sort of system we want to build (in order to defend from dangerous systems)
3. The sort of systems that humans are (in order to meaningfully talk about "human preferences")

Motivation #1 naturally suggests a descriptive approach, motivation #2 naturally suggests a prescriptive approach, and motivation #3 is sort of a mix of both: on the one hand, we're describing something that already exists, on the other hand, the concept of "preferences" inherently comes from a normative perspective. There are also reasons to think these different motivation should converge on a single, coherent concept.

Here, we will focus on motivation #1.

A central reason why we are concerned about powerful unaligned agents, is that they are influential. Agents are the sort of system that, when instantiated in a particular environment is likely to heavily change this environment, potentially in ways inconsistent with the preferences of other agents.

Bayesian VNM

Consider a nice space^[1] $X$ of possible "outcomes", and a system that can choose^[2] out of a closed set of distributions $D\subseteq \Delta X$. I propose that an influential system should satisfy the following desideratum:

The system cannot select ${\mu }^{\ast }\in D$ which can be represented as a non-trivial lottery over other elements in $D$. In other words, ${\mu }^{\ast }$ has to be an extreme point of the convex hull of $D$.

Why? Because a system that selects a non-extreme point leaves something to chance. If the system can force outcome $\mu \in \Delta X$, or outcome $\nu \in \Delta X$ but chooses instead outcome $p\mu +(1-p)\nu$, for $p\in (0,1)$ and $\mu \ne \nu$, this means the system gave up on its ability to choose between $\mu$ and $\nu$ in favor of a $p$-biased coin. Such a system is not "locally^[3] maximally" influential^[4].

[EDIT: The original formulation was wrong, h/t @harfe for catching the error.]

The desideratum implies that there is a convergent sequence of utility functions $\{u_k:X\to R{\}}_{k\in N}$ s.t.

• For every $k\in N$, $E_{\mu }\left[u_k\right]$ has a unique maximum ${\mu }_k$ in $D$.
• The sequence ${\mu }_k$ converges to ${\mu }^{\ast }$.

In other words, such a system can be approximated by a VNM decision-maker within any precision. For finite $D$, we don't need the sequence, instead there is some $u:X\to R$ s.t. ${\mu }^{\ast }$ is the unique maximum of $E_{\mu }\left[u\right]$ over $D$. This observation is mathematically quite simple, but I haven't seen it made elsewhere (but I would not be surprised if it did appear in the decision theory literature somewhere).

Infra-Bayesian VNM?

Now, let's say that the system is choosing out of a set of credal sets (crisp infradistributions) $D\subseteq \square X$. I propose the following desideratum:

[EDIT: Corrected according to a suggestion by @harfe, original version was too weak.]

Let $\hat{D}$ be the closure of $D$ w.r.t. convex combinations and joins^[5]. Let ${\Theta }^{\ast }\in \square X$ be selected by the system. Then:

• For any $\Phi ,\Psi \in \hat{D}$ and $p\in (0,1)$, if ${\Theta }^{\ast }=p\Phi +(1-p)\Psi$ then $\Phi =\Psi$.
• For any $\Phi \in \hat{D}$, if $\Phi \subseteq {\Theta }^{\ast }$ then $\Phi ={\Theta }^{\ast }$.

The justification is, a locally maximal influential system should leave the outcome neither to chance nor to ambiguity (the two types of uncertainty we have with credal sets).

We would like to say that this implies that the system is choosing according to maximin relatively to a particular utility function. However, I don't think this is true, as the following example shows:

Example: Let $X=\{0,1\}$, and $D$ consist of the probability intervals ${\Theta }_0:=[0,\frac{2}{3}]$, ${\Theta }_1:=[\frac{1}{3},1]$ and ${\Theta }_2:=[\frac{1}{3},\frac{2}{3}]$. Then, it is (I think) consistent with the desideratum to have ${\Theta }^{\ast }={\Theta }_2$.

Instead, I have the following conjecture:

Conjecture: There exists some space $Y$, some $\xi \in \Delta Y$ and convergent sequence $\{u_k:Y\times X\to R{\}}_{k\in N}$ s.t.

As before, the maxima should be unique.

Such a "generalized utility function" can be represented as an ordinary utility function with a latent $Y$-valued variable, if we replace $D$ with $D^{\prime }\subseteq \square (Y\times X)$ defined by

However, using utility functions constructed in this way leads to issues with learnability, which probably means there are also issues with computational feasibility. Perhaps in some natural setting, there is a notion of "maximally influential under computational constraints" which implies an "ordinary" maximin decision rule.

This approach does rule out optimistic or "mesomistic" decision-rules. Optimistic decision makers tend to give up on influence, because they believe that "nature" would decide favorably for them. Influential agents cannot give up on influence, therefore they should be pessimistic.

Sequential Decision-Making

What would be the implications in a sequential setting? That is, suppose that we have a set of actions $A$, a set of observations $O$, $X:=(A\times O{)}^{\omega }$, a prior $\zeta :(A\times O{)}^{\ast }\times A\to \Delta O$ and

In this setting, the result is vacuous because of an infamous issue: any policy can be justified by a contrived utility functions that favors it. However, this is only because the formal desideratum doesn't capture the notion of "influence" sufficiently well. Indeed, a system whose influence boils down entirely to its own outputs is not truly influential. What motivation #1 asks of us, is talk about systems that influence the world-at-large, including relatively "faraway" locations.

One way to fix some of the problem is, take $X:=O^{\omega }$ and define $D$ accordingly. This singles out systems that have influence over their observations rather than only their actions, which is already non-vacuous (some policies are not such). However, such a system can still be myopic. We can take this further, and select "long-term" influence by projecting onto late observations or some statistics over observations. However, in order to talk about actually "far-reaching" influence, we probably need to switch to the infra-Bayesian physicalism setting. There, we can set $X:=2^{\Gamma }$, i.e. select for system that have influence over physically manifest computations.

1. ^{^}

   I won't keep track of topological technicalities here, probably everything here works at least for compact Polish spaces.

2. ^{^}

   Meaning that the system has some output, and different counterfactual outputs correspond to different elements of $D$.

3. ^{^}

   I say "locally" because it refers to something like a partial order, not a global scalar measure of influence.

4. ^{^}

   See also Yudkowsky's notion of efficient systems "not leaving free energy".

5. ^{^}

   That is, if $\Psi ,\Phi \in \hat{D}$ then their join (convex hull) $\Psi \vee \Phi$  is also in $\hat{D}$, and so is $p\Psi +(1-p)\Phi$ for every $p\in [0,1]$. Moreover, $\hat{D}$ is the minimal closed superset of $D$ with this property. Notice that this implies $\hat{D}$ is closed w.r.t. arbitrary infra-convex combinations, i.e. for any $Y$, $K:Y\to \hat{D}$ and $\Xi \in \square Y$, we have $K_{\ast }\Xi \in \hat{D}$.