The goal of this post is to clarify a few concepts relating to AI Alignment under a common framework. The main concepts to be clarified:

• Optimization. Specifically, this will be a type of Vingean agency. It will split into Selection vs Control variants.
• Reference (the relationship which holds between map and territory; aka semantics, aka meaning). Specifically, this will be a teleosemantic theory.

The main new concepts employed will be endorsement and legitimacy. 

TLDR: 

• Endorsement of a process is when you would take its conclusions for your own, if you knew them.
• Legitimacy relates to endorsement in the same way that good relates to utility. (IE utility/endorsement are generic mathematical theories of agency; good/legitimate refer to the specific thing we care about.)
• We perceive agency when something is better at doing something than us; we endorse some aspect of its reasoning or activity. (Endorse as a way of achieving its goals, if not necessarily our own.)
• We perceive meaning (semantics/reference) in cases where something has been optimized for accuracy -- that is, the goal we endorse a conclusion with respect to is some notion of accuracy of representation.

This write-up owes a large debt to many conversations with Sahil, although the views expressed here are my own.

Broader Context

The basic idea is to investigate agency as a natural phenomenon, and have some deep insights emerge from that, relevant to the analysis of AI Risk. Representation theorems -- also sometimes called coherence theorems or selection theorems depending on what you want to emphasize -- start from a very normative place, typically assuming preferences as a basic starting object. Perhaps these preferences are given a veneer of behaviorist justification: they supposedly represent what the agent would choose if given the option. But actually giving agents the implied options would typically be very unnatural, taking the agent out of its environment.

There are many ways to drive toward more naturalistic representation theorems. The work presented here takes a particular approach: cognitive reduction. I want to model what it means for one agent to think of something as being another agent. Daniel Dennett called this the intentional stance. 

The goal of the present essay is to sketch a formal picture of the intentional stance. This is not yet a representation theorem -- it does not establish a type signature for agency based on the ideas. However, for me at least, it seems like an important piece of deconfusion. It paints a picture of agents who understand each other, as thinking things. I hope that it can contribute to a useful representation theorem and a broader picture of agency.

Meaning

In Signalling & Simulacra, I argued that the signal-theoretic^[1] analysis of meaning (which is the most common Bayesian analysis of communication) fails to adequately define lying, and fails to offer any distinction between denotation and connotation or literal content vs conversational implicature. A Human's Guide to Words gives a related view:^[2]

No dictionary, no encyclopedia, has ever listed all the things that humans have in common.  We have red blood, five fingers on each of two hands, bony skulls, 23 pairs of chromosomes—but the same might be said of other animal species.
[...]
A dictionary is best thought of, not as a book of Aristotelian class definitions, but a book of hints for matching verbal labels to similarity clusters, or matching labels to properties that are useful in distinguishing similarity clusters.
- Eliezer Yudkowski, Similarity Clusters

While I agree that words do not have clear-cut definitions (except, perhaps, in a few cases such as mathematics), I think it would be a mistake to conclude that there is nothing to the denotation/connotation distinction. In my view, words cannot just be probabilistic clusters.

Teleosemantics

In Teleosemantics!, I argue that the denotative meaning is what a symbol is optimized to correspond to. 

The signal-theoretic analysis of meaning gives us the information-theoretic content of a communicative act; but any observation carries information in this sense. What distinguishes symbolic communication from other information-carrying signals is that some care has been taken to convey specific information. Sometimes the probabilistic information is all we care about; but tracking literal meaning is also an important part of language.

Commonly confused words, such as imply vs infer, probabilistically carry some of each other's signal-theoretic meaning (based on the probability we assign to the speaker having confused the two words). But the community of speakers overall puts optimization pressure on avoiding this confusion, by correcting mistakes noticed in conversation, by writing dictionaries, and through articles about common mistakes.

Similarly, lying happens, so the signal-theoretic meaning of an utterance includes some probability of it being a lie (and the probabilistic implications thereof). But the linguistic community optimizes against lying, so there's a sense in which the lie is not part of the intended meaning of the utterance. 

If a liar claims (falsely) "I am very wealthy", there's a commonsense way in which they mean that they're very wealthy. That's the meaning they're trying to convey. I want to be clear that this does not fit with the technical theory of meaning which I am advocating here.

Plurality of Teleosemantic Meaning

Meaning, on my account, comes from optimizing symbols to have a specific correspondence to reality; that is, to be accurate under a specific intended reading. This is entirely different from optimizing words to create a specific belief in their audience, which is what the liar is doing. On my account, optimizing for impact on an audience rather than accuracy fails to impart those words with (teleosemantic) meaning. 

Since the liar's use of words does not impart them with meaning in itself, the only relevant optimization process which gives meaning to the words is the broader linguistic community.

However, there are cases where multiple optimization processes may be trying to give meaning to the same words, perhaps at cross-purposes. For example, the scientific community has a distinct notion of "theory" which people sometimes claim is the real meaning. (Wikipedia's disambiguation page provides a large list of communities with their own preferred notion of theory.)

So in general, the theory here requires meaning to be indexed by an agent (or optimization process); we can differentiate what an individual person means by their words from what larger linguistic communities mean by those words.

Now, obviously, in order to understand the teleosemantic definition of meaning, we need to understand what "agent" or "optimization process" or "optimized" means.

Agency

In Belief in Intelligence, Eliezer sketches the peculiar mental state which regards something else as intelligent:

Imagine that I'm visiting a distant city, and a local friend volunteers to drive me to the airport.  I don't know the neighborhood. Each time my friend approaches a street intersection, I don't know whether my friend will turn left, turn right, or continue straight ahead.  I can't predict my friend's move even as we approach each individual intersection - let alone, predict the whole sequence of moves in advance.

Yet I can predict the result of my friend's unpredictable actions: we will arrive at the airport. 
[...]
I can predict the outcome of a process, without being able to predict any of the intermediate steps of the process.

In Measuring Optimization Power, he formalizes this idea by taking a preference ordering and a baseline probability distribution over the possible outcomes. In the airport example, the preference ordering might be how fast they arrive at the airport. The baseline probability distribution might be Eliezer's probability distribution over which turns to take -- so we imagine the friend turning randomly at each intersection. The optimization power of the friend is measured by how well they do relative to this baseline. 

I think this can be a useful notion of agency, but constructing this baseline model does strike me as rather artificial. We're not just sampling from Eliezer's world-model. If we sampled from Eliezer's world-model, the friend would turn randomly at each intersection, but they'd also arrive at the airport in a timely manner no matter which route they took -- because Eliezer's actual world-model believes the friend is capably pursuing that goal.

So to construct the baseline model, it is necessary to forget the existence of the agency we're trying to measure while holding other aspects of our world-model steady. While it may be clear how to do this in many cases, it isn't clear in general. I suspect if we tried to write down the algorithm for doing it, it would involve an "agency detector" at some point; you have to be able to draw a circle around the agent in order to selectively forget it. So this is more of an after-the-fact sanity check for locating agents, rather than a method of locating agents in the first place.

I will propose a variation of Eliezer's definition which does not have this problem. First, though, I have to define some terminology.

Endorsement

I will say that one set of beliefs^[3] $P_1$ endorses^[4] another $P_2$ (with respect to topic $X$) in the case that, if we condition $P_1$ on $P_2$'s belief about a thing, then $P_1$ adopts that belief as its own:

Belief Endorsement: Probability distribution $P_1$ belief-endorses $P_2$ in the case that $P_1(X|‘‘P_2\left(X\right)=p")=p$.

This is already enough to get my proposed formal definition of meaning off the ground. When the above condition holds, $P_1$ sees $‘‘P_2\left(X\right)"$ as a belief about $X$. The notation somewhat obscures the dependence on a particular translation between $P_1$'s ontology and $P_2$'s ontology. The quotation marks are communicating a translation into $P_1$'s event algebra. We can't just abstractly relate two probability distributions to each other, because Alice endorsing Bob's beliefs has to do with how Bob relates to the world, not just what his probability distribution is as a pure mathematical abstraction. Imagine Alice seeing Bob's beliefs initially as just some random variable; interpreting them as a probability distribution is an extra step.

For example, Alice and Bob both start out regarding a coinflip as 50-50. Bob gets to see the coin after the flip, and updates to near-certainty about which way the coin landed. Because of the way Alice knows Bob's beliefs about the coin have been correlated with the coin itself, and because Alice's beliefs have not been so correlated, Alice now endorses Bob's belief about "the coin landed heads" as meaning the coin landed heads, and Bob's "the coin landed tails" as meaning the coin landed tails. This is not because Alice thinks Bob's beliefs are perfectly accurate; only more accurate than Alice's.

However, to see why this is a teleosemantic notion of meaning, I need to generalize "endorsement" more, in order to properly relate it to optimization. I will generalize it in several steps.

We can make a generalized definition of endorsement handle random variables, rather than only probabilities:

Expectation Endorsement: $E_{\omega \sim P_1}(V\left(\omega \right)|‘‘E_{\omega \sim P_2}(V\left(\omega \right))=x")=x$

Here, $E_{\omega \sim P_i}(V\left(\omega \right))$ gives the expected value of random variable $V$ when worlds $\omega$ are sampled from $P_i$. This notion of endorsement generalizes the previous, because we can take $V$ to be the indicator variable for $X$, which has value 1 when $X$ and zero otherwise; in this case, $E_i\left(V\right)=P_i\left(X\right)$.

However, this is not yet a full notion of endorsement of decisions, because the above doesn't necessarily make sense for non-convex optimization. If $V$ represents a decision between three doors, $V\in \{1,2,3\}$, then an expected value of 2 could indicate a belief that the optimal choice is door 2, or it could indicate complete uncertainty between the three options! We want to instead say that one agent would endorse another agent's choice rather than its expectation.

Just as $P$ gives an agent's beliefs and $E$ gives its expected values, I introduce $C$ to represent an agent's choices. This is similar to choice functions as sometimes appear in representation theorems. Given a set of possible 'actions' $A$, the choice function $C_1^{U_1,P_1}\left(A\right)$ gives agent 1's choice $a\in A$ under beliefs $P_1$ and objective function $U_1$. $C$ can be thought of as the mathematical optimization algorithm which the agent uses, such as simulated annealing. A particularly simple choice of $C$ is the argmax of the expectation: $C_i^{U_i,P_i}\left(A\right)={\text{argmax}}_{a\in A}{E}_{\omega \sim P_1}{U}_1\left(a\right)$. When the set of possible choices $A$ is small, it will often make sense to assume that $C$ is this argmax. However, we need to avoid making the assumption if we want to reason about things like quantilization. For notational convenience, $C_i^{U_i,P_i}\left(A|X\right)$ means $C_i^{U_i,P_i|X}\left(A\right)$; that is, the probability distribution is first conditioned on $X$ before optimizing.

Having established this notation, our next notion of endorsement is a straightforward parallel of the previous definitions:

Selection Endorsement: Agent 1, having beliefs $P_1$ and a choice function $C_1$, selection-endorses Agent 2 as optimizing $U_2$ via the choice $C_2$ if and only if $C_1^{U_2,P_1}(A|‘‘C_2^{U_2,P_2}\left(A\right)=a")=a$.

So, Alice thinks of Bob as optimizing for $U_2$ if Alice would copy Bob's answer (if she were optimizing for $U_2$). 

Much like the previous definitions, the quotations emphasize that Bob's choice is really a variable in Alice's ontology, and treating it as a choice is a matter of interpretation. $‘‘C_2\left(A\right)=a"$ is really an event of a random variable taking on a specific value, $V=v$.

A previous revision of this essay defined selection endorsement via an argmax instead of introducing the concept of choice functions. This amounts to a hidden assumption of unbounded computational resources on the Alice side, which would harm our ability to recognize Bobs who are agentic in more bounded ways.

Selection endorsement somewhat resembles The ground of optimization if we take the random variable $V$ to be a system's state at a single time-slice. I selection-endorse the whole system (as optimizing some $U_2$) if, ignorant of what scores highly in $U_2$ myself, I would take the system's state as a plausible answer.

Since $U_2\left(a\right)$ is a mathematically pure function $U_2:A\to R$, this is selection in a Selection vs Control sense.^[5] If we think of the variable $V$ (which we interpret as Bob's choice) as some little piece of the world, it is being optimized to maximize some property of itself, not some property of the broader world. 

If $U\left(\right)$ is an impure function, its values depend on the wider world: $U:\Omega \times A\to R$, where $\Omega$ is the sample space.^[6] Now, the optimization has to be explicitly defined with a dependence on worlds $\omega \in \Omega$. The argmax would look like ${\text{argmax}}_{r\in A}U(\omega ,r)$ instead of ${\text{argmax}}_{a\in A}U\left(r\right)$.

We have to slightly revise our notion of endorsement once again, to represent an agent thinking of another agent as optimizing in this broader sense:

Control Endorsement: $C_1^{U_2,P_1}(A|‘‘C_2^{U_2,P_2}\left(A\right)=a")=a$, where $U_2$ is now a two-argument function, taking worlds and actions, rather than a one-argument function. 

Sorry, the current notation doesn't highlight the difference between this definition and the previous one. But this is a big distinction.^[5]

Control endorsement is the notion of intentional stance that I have been driving towards. The formula tracks someone tracking agency. If Alice control-endorses Bob (as optimizing $U_2$), Alice sees Bob as an agent.

This resembles ideas in Vingean Agency and Optimization at a Distance. Like Eliezer's Measuring Optimization Power, agency is defined relative to a baseline distribution ($P_1$); but I think the sort of baseline distribution we need to get this definition to work is much less artificial than the sort which Eliezer's definition needed. It can just be our honest beliefs.

Control endorsement generalizes selection endorsement, since $U$ can ignore its first argument; and generalizes expectation endorsement, since $U$ can be some loss function for which the expected value is the loss-minimizing answer; and generalizes belief endorsement when that loss function is a proper scoring rule:

Belief Endorsement Revisited: ${\text{argmax}}_p{E}_{\omega \sim P_1|X}{A}_Y(\omega ,p)=x$, where $X$ is the event $P_2\left(Y\right)=x$, and $A_Y(\omega ,p)$ is a proper scoring rule judging the accuracy of $p$ with respect to target event $Y$.

(Here, the choice function is assumed to be argmax for simplicity, since results about proper scoring rules incentivizing honest belief reports generally assume we can argmax.)

As another important variation on endorsement, we can fix $U$ to be the radical interpreter's^[7] own utility function:

"Absolute" Endorsement: control endorsement where $U_2$ is the utility function of the observer whose beliefs are $P_1$.

Speculatively, this might have some useful connection to corrigibility. If Alice were to absolutely endorse Bob's actions, then Alice should be fine with Bob modifying Alice's source code.

Conditional Endorsement

At first, it might seem like an endorsement-based notion of optimization process loses something relative to Eliezer's measurement of optimization power: endorsement just gives a binary yes/no, rather than measuring a degree of optimization.

However, consider an example with Alice, Bob, and Carol. Alice endorses both Bob and Carol, but she continues to endorse Carol even after learning Bob's decision; the reverse is not the case. Obviously, she trusts Carol more than Bob. We can formalize this via conditional endorsement:

Conditional (Control) Endorsement: $P_1$ endorses $W$ given $V$ iff ${\text{argmax}}_{v\in A}{E}_{\omega \sim P_1|X\wedge Y}U(\omega ,v)=y$, where $X$ is the event $V=x$, and $Y$ is the event $W=y$.

(We can easily adapt this to notions of conditional endorsement for the various sorts of endorsement.)

Thus, holding the utility function fixed but changing the random variable considered, we can (partially) order different random variables by how endorsed they are. (I will not try to prove transitivity here, since my goal is to get the overall picture across; I have not checked it, although I expect it to hold.)

Radical Probabilism & Other Generalizations

So far, I've used a utility function to represent values, which may surprise those familiar with An Orthodox Case Against Utility Functions. I expect these concepts to generalize well to concepts beyond simple utility functions, but I didn't want to overcomplicate things here -- I am focusing on getting the basic picture across. Similarly, these notions already work well with Radical Probabilism, but I haven't emphasized that here. Infrabayesian versions of all this could also be interesting, and I expect, not particularly difficult to define.

Legitimacy

Legitimacy is to endorsement as goodness is to utility: "utility" is an abstract notion of an agent's preferences, whereas "good" is the thing we actually care about. Similarly, "endorsement" is an abstract notion intended to apply to agency in general, while "legitimacy" is the thing we care about as humans.

So, a mode of reasoning is legitimate if it has a reliable tendency toward the truth. Correct mathematical proofs are legitimate. The scientific method, when carried out well, is legitimate. 

Wireheading is not legitimate. Taking a murder pill is not legitimate. Some drugs impact your reasoning in legitimate ways, while others are illegitimate. 

This is supposed to be the same concept of legitimacy from Nora's sequence on the value change problem. Hence, aligning an AI toward legitimacy is supposed to be a solution to the problems mentioned there.

I think there may be a technical sense (yet to be articulated) in which we'd prefer to align AI to legitimacy, but we can only align it to specific forms of endorsement and hope that's close enough. An AI aligned to endorsement can (more or less...) only do things that humans would consent to. But humans are sometimes grateful in retrospect about things which seemed terrible at the time; for example, if math homework was an important step in your coming to love math, but at the time felt like something you were coerced into.

By analogy, our current endorsement attitudes could be mistaken. We know some cases, such as addictive drugs, where we would not say the change may actually be legitimate despite our current anti-endorsement of it; but there may be some other cases where we would never consent to the change, but (in some difficult-to-pin-down sense) the change would actually be correct, not only in retrospect, but legitimately.

However, anyone who purposefully built a superintelligent AGI aligned to some notion of legitimacy designed to trample on endorsement in select cases would be unilaterally imposing their own guess at legitimate values. I claim this is bad behavior.

Updatelessness

I think it is clear that endorsement gives a picture where, when Alice considers whether Bob is an agent, Bob being updateless will boost his agency and any updateful behavior from Bob will be to the detriment of Bob's agency. After all, Alice is judging from the perspective of $P_1$. Any optimization will be judged by how well it optimizes the expectations of $P_1$. (Although, as I indicated in the beginning, fleshing this out into an actual representation theorem is part of the goal here but isn't something accomplished in the present work.)

This suggests that Alice should probably be updateless as well; that is, $P_1$ should be Alice's prior, not Alice's posterior.

In general, though, I think it will be important to consider a variety of probability distributions from which to judge endorsement. Conditional endorsement is already an example of this. Resource-bounded agents can be studied by considering baseline distributions from different complexity classes. 

The Van Fraassen Reflection Principle

Students of philosophy may notice the resemblance between my notion of "endorsement" and the Reflection Principle. In my language, the Reflection Principle says that if $P_1$ is an agent's beliefs at one time, and $P_2$ is an agent's beliefs at a future time, then $P_1$ should belief-endorse $P_2$.

As such, some might argue that "endorsement" should be called "reflection". I would argue that "reflection" means too many things, and my account differs enough from Van Frassen -- the Reflection Principle is about insisting that X should endorse Y in specific cases, whereas I am more focusing on studying the relation in general. Indeed, I would say there are cases where it is not rational to endorse our future beliefs, such as when we plan to be inebriated at a specific time.

Questions & Conjectures

• How well can we use conditional endorsement to characterize optimization power (or more generally, level of endorsement)? Is it transitive?
• What further generalizations or alternative definitions of endorsement might be important?
• How do we build a useful representation theorem out of this idea?
• Can we prove something within this framework along the lines of capable agents have beliefs?
• How can we integrate value change into this picture?
• There's a big difference between endorsing some beliefs as a posterior (which is accuracy-centric) and endorsing them as a prior for use in updateless decisionmaking. How should this be characterized?
• It seems sensible to call selection "myopic" and control "nonmyopic". However, although epistemic accuracy falls on the control side, it doesn't feel very control-oriented, and it feels to me like there's a strong sense in which it is myopic. Can this be characterized?
• Looking at things through an algorithmic information theory lens, it makes sense to say that endorsement, as an interpretation of something as an optimization process, is a "better" interpretation when the utility function used to interpret something as an agent is simpler. However, because the utility function in control endorsement has access to strictly more data than that in selection endorsement, interpreting something as a control process can't be much more complex than interpreting it as a selection process -- worst-case, you just have to discard the $\omega$ and then apply the U you used to interpret it as a selection process. So the interesting case is where the control-process interpretation can be much simpler than the selection-process interpretation.
• I have an intuition that insights for eliciting latent knowledge can be uncovered by examining what happens when we translate back and forth between interpreting something as a selection process vs a control process. A control process has to track information about the external world (has to "have beliefs") in order to do its job; a selection process has no need to do this. So in order for Alice to see Bob as a selection process rather than a control process, she has to understand how he selects: she has to include the belief calculations Bob uses to select as part of the utility function instead of abstracting them away. If we can understand something as a selection process instead of a control process, it seems intuitively more trustable. If the control process is hiding something from us, can we reveal the information by translating it to a selection process?
• Can something interesting be said about inner alignment via this framework?
• What does it look like to build a system with endorsement as the target?
• I imagine there is something to be gained by thinking about different ways of varying all the parameters of endorsement, the way conditional endorsement varies the baseline probability distribution. 

1. ^{^}

   I'm saying "signal-theoretic" rather than "signalling-theoretic" here because it sounds better to me; but the field is called "signalling theory" not "signal theory".

2. ^{^}

   I don't recall the appropriate reference now, but I think at one point Eliezer defines lying as communicating with the intent to mislead, IE the intent to make your audience's beliefs less accurate. 

   While I think this is a pretty good pragmatic definition, it fails to differentiate lying from filtering evidence or other clever ways to mislead. 

   Of course, what sorts of standards you hold your definitions to is part of the debate, here. 

3. ^{^}

   For our purposes here, it is fine to imagine that "beliefs" are probability distributions as normally defined. But I also have in mind more computationally bounded notions, which may be only approximately probabilistically coherent; and we can also consider other variations, such as infradistributions. 

   In any case, for now, my analysis is outsourcing the notion of "belief" to other theories. A more fully-developed version with its own representation theorem should wrap around and justify whatever notion of "belief" is used, by going on to establish that the notion of "belief" assumed at this stage can then be derived as part of the representation theorem. In other words, my goal is a picture of an agent thinking about an agent, where the two have the same type signature.

4. ^{^}

   Credit for the term "endorsement" goes to Scott Garrabrant.

5. ^{^}

   We can slide between selection and control if we are happy to vary how $V$ and $U$ are defined. Since random variables are a function of the whole event-space anyway, we can pack as much information about the world into $V$ as we like, so long as we are happy to make $U$ just not care about those extra bits of information. So we could model a smart thermostat turning on and off as an attempt to control the whole house's temperature (a utility function which involves variables it must estimate rather than observe directly), by selecting the on-or-off state with the highest expected value (a mathematically pure function of variables which the thermostat either directly observes or directly controls). So it can be seen as a selector in one sense or a controller in another sense.

   However, seeing an agent as a selector rather than a controller means that $U\left(\right)$ has to encode the beliefs of the agent, to calculate the expected values. So beliefs are being represented as part of the preferences. When seeing an agent as a controller, the beliefs are instead seen as a part of the mechanism for optimizing the preferences. I expect this means it is often simpler to view something as a controller, if it is good at controlling; the selection view is overcomplicated by the belief information.

6. ^{^}

   It feels a bit weird that the utility function takes the world and the action, here. We have to think of this as being in world $\omega$, but then counterfacting on action $V=x$. I could see this aspect of the theory being fiddled with.

7. ^{^}

   If Alice is interpreting Bob as an agent, Alice is the "radical interpreter". This terminology comes from philosophy; the "radical interpreter" is like an alien looking at human brains and trying to interpret the meaning of human beliefs.