Written during the SERI MATS program under the joint mentorship of John Wentworth, Nicholas Kees, and Janus.

Create a surreal, black and white banner-like image of an agent deciding how to act in an infinitely large lattice-universe with infinitely many people. The agent should be portrayed as thoughtful and contemplative in the center of a vast, lattice-structured universe. This lattice, symbolizing infinity, should be populated with numerous small figures to represent the countless people. The image should have a surreal quality, with abstract patterns and shapes seamlessly integrated into the lattice. The panoramic, banner-style format should enhance the sense of vastness and complexity, emphasizing the enormity of the decision-making process in such an expansive and structured universe.

Preface

In classical game theory, we characterise agents by a utility function and assume that agents choose options which cause maximal utility. This is a pretty good model, but it has some conceptual and empirical limitations which are particularly troublesome for AI safety.

Higher-order game theory (HOGT) is an attempt to rebuild game theory without appealing to either utility functions or maximisation. I think Higher-Order Game Theory is cool so I'm writing a sequence on it.

[Part 1]
[Part 2]
[Part 3]
...
[Part n]

I'll try to summarise the relevant bits of the literature, present my own minor extensions, and apply HOGT to problems in AI safety

You're reading the first post! Let's get into it.

The role of argmax

For each set $X$ , let ${argmax}_{X} : (X \to R) \to P (X)$ be the familiar function which receives a function $u : X \to R$ and produces the set of element which maximise $u$ . A function like ${argmax}_{X}$ is sometimes called a higher-order function or functional, because it receives another function as input.

Explicitly, ${argmax}_{X} = λ^{u : X \to R} . {x \in X | \forall x^{'} \in X, u (x) \geq u (x^{'})}$ .^[1]

As you all surely know, $argmax$ plays a central role in classical game theory. Typically we interpret the set $X$ as the agent's options,^[2] and the function $u : X \to R$ as the agent's task, which assigns a payoff $u (x) \in R$ to each option $x \in X$ . We say an option $x \in X$ is optimal to the agent for the task $u : X \to R$ whenever $x \in {argmax}_{X} (u)$ . Classical game theory is governed by the assumption that agents choose optimal options in whatever task they face, where optimality strictly means utility-maximisation.

Definition 1 (provisional): Let $X$ be any set of options. A task is any function $u : X \to R$ . An option $x \in X$ is optimal for a task $u : X \to R$ if and only if $x \in {argmax}_{X} (u)$ .

Due to the presence of the powerset operator $P$ in ${argmax}_{X} : (X \to R) \to P (X)$ , this model of the agent is possibilistic — for each task $u : X \to R$ , our model says which options are possibly chosen by agent. The model doesn't say which options are probably chosen by the agent — for that we'd need a function $(X \to R) \to Δ (X)$ . Nor does the model say which options are actually chosen by the agent — for that we'd need a fu...

Exercises / Problem-Sets