This frame seems useful, but might obscure some nuance:

• The systems we should be most worried about are the AIs of tomorrow, not the AIs of today. Hence, some critical problems might not manifest at all in today's AIs. You can still say it's a sort of "illegible problem" of modern AI that it's progressing towards a certain failure mode, but that might be confusing.
• While it's true that deployment is the relevant threshold for the financial goals of a company, making it crucial for the company's decision-making and available resources for further R&D, the dangers are not necessarily tied to deployment. It's possible for a world-ending event to originate during testing or even during training.

No, it's not at all the same thing as OpenAI is doing. 

First, OpenAI is working using a methodology that's completely inadequate for solving the alignment problem. I'm talking about racing to actually solve the alignment problem, not racing to any sort of superintelligence that our wishful thinking says might be okay. 

Second, when I say "racing" I mean "trying to get there as fast as possible", not "trying to get there before other people". My race is cooperative, their race is adversarial.

Third, I actually signed the FLI statement on superintelligence. OpenAI hasn't.

Obviously any parallel efforts might end up competing for resources. There are real trade-offs between investing more in governance vs. investing more in technical research. We still need to invest in both, because of diminishing marginal returns. Moreover, consider this: even the approximately-best-case scenario of governance only buys us time, it doesn't shut down AI forever. The ultimate solution has to come from technical research.

Strong disagree.

We absolutely do need to "race to build a Friendly AI before someone builds an unFriendly AI". Yes, we should also try to ban Unfriendly AI, but there is no contradiction between the two. Plans are allowed (and even encouraged) to involve multiple parallel efforts and disjunctive paths to success.

It's not that academic philosophers are exceptionally bad at their jobs. It's that academic philosophy historically did not have the right tools to solve the problems. Theoretical computer science, and AI theory in particular, is a revolutionary method to reframe philosophical problems in a way that finally makes them tractable.

About "metaethics" vs "decision theory", that strikes me as a wrong way of decomposing the problem. We need to create a theory of agents. Such a theory naturally speaks both about values and decision making, and it's not really possible to cleanly separate the two. It's not very meaningful to talk about "values" without looking at what function the values do inside the mind of an agent. It's not very meaningful to talk about "decisions" without looking at the purpose of decisions. It's also not very meaningful to talk about either without also looking at concepts such as beliefs and learning.

As to "gung-ho attitude", we need to be careful both of the Scylla and the Charybdis. The Scylla is not treating the problems with the respect they deserve, for example not noticing when a thought experiment (e.g. Newcomb's problem or Christiano's malign prior) is genuinely puzzling and accepting any excuse to ignore it. The Charybdis is perpetual hyperskepticism / analysis-paralysis, never making any real progress because any useful idea, at the point of its conception, is always half-baked and half-intuitive and doesn't immediately come with unassailable foundations and justifications from every possible angle. To succeed, we need to chart a path between the two.

I found LLMs to be very useful for literature research. They can find relevant prior work that you can't find with a search engine because you don't know the right keywords. This can be a significant force multiplier.

They also seem potentially useful for quickly producing code for numerical tests of conjectures, but I only started experimenting with that.

Other use cases where I found LLMs beneficial:

• Taking a photo of a menu in French (or providing a link to it) and asking it which dishes are vegan.
• Recommending movies (I am a little wary of some kind of meme poisoning, but I don't watch movies very often, so seems ok).

That said, I do agree that early adopters seem like they're overeager and maybe even harming themselves in some way.

Btw, what are some ways we can incorporate heuristics into our algorithm while staying on level 1-2?

1. We don't know how the prove to required desiderata about the heuristic, but we can still reasonably conjecture them and support the conjectures with empirical tests.
2. We can't prove or even conjecture anything useful-in-itself about the heuristic, but the way the heuristic is incorporated into the overall algorithm makes it safe. For example, maybe the heuristic produces suggestions together with formal certificates of their validity. More generally, we can imagine an oracle-machine (where the heuristic is slotted into the oracle) about which we cannot necessarily prove something like a regret bound w.r.t. the optimal policy, but we can prove (or at least conjecture) a regret bound w.r.t. some fixed simple reference policy. That is, the safety guarantee shows that no matter what the oracle does, the overall system is not worse than "doing nothing". Maybe, modulo weak provable assumptions about the oracle, e.g. that it satisfies a particular computational complexity bound.
3. [Epistemic status: very fresh idea, quite speculative but intriguing.] We can't find even a guarantee like a above for a worst-case computationally bounded oracle. However, we can prove (or at least conjecture) some kind of an "average-case" guarantee. For example, maybe we have high probability of safety for a random oracle. However, assuming a uniformly random oracle is quite weak. More optimistically, maybe we can prove safety even for any oracle that is pseudorandom against some complexity class $C_1$ (where we want $C_1$ to be as small as possible). Even better, maybe we can prove safety for any oracle in some complexity class $C_2$ (where we want $C_2$ to be as large as possible) that has access to another oracle which is pseudorandom against $C_1$. If our heuristic is not actually in this category (in particular, $C_2$ is smaller than $P$ and our heuristic doesn't lie in $C_2$), this doesn't formally guarantee anything, but it does provide some evidence for the "robustness" of our high-level scheme.

I see modeling vs. implementation as a spectrum more than a dichotomy. Something like:

1. On the "implementation" extreme you prove theorems about the exact algorithm you implement in your AI, s.t. you can even use formal verification to prove these theorems about the actual code you wrote.
2. Marginally closer to "modeling", you prove (or at least conjecture) theorems about some algorithm which is vaguely feasible in theory. Some civilization might have used that exact algorithm to build AI, but in our world it's impractical, e.g. because it's uncompetitive with other AI designs. However, your actual code is conceptually very close to the idealized algorithm, and you have good arguments why the differences don't invalidate the safety properties of the idealized model.
3. Further along the spectrum, your actual algorithm is about as similar to the idealized algorithm as DQN is similar to vanilla Q-learning. Which is to say, it was "inspired" by the idealized algorithm but there's a lot of heavy lifting done by heuristics. Nevertheless, there is some reason to hope the heuristic aspects don't change the safety properties.
4. On the "modeling" extreme, your idealized model is something like AIXI: completely infeasible and bears little direct resemblance to the actual algorithm in your AI. However, there is still some reason to believe real AIs will have similar properties to the idealized model.

More precisely, rather than a 1-dimensional spectrum, there are at least two parameters involved: 

• How close is the object you make formal statements about to the actual code of your AI, where "closeness" is measured by the strength of the arguments you have for the analogy, on a scale from "they are literally the same" to solid theoretical and/or empirical evidence to pure hand-waving/intuition
• How much evidence you have for the formal statements, on a scale from "I proved it within some widely accepted mathematical foundation (e.g. PA)" to "I proved vaguely related things, tried very hard but failed to disprove the thing and/or accumulated some empirical evidence".

[EDIT: And a 3rd parameter is, how justified/testable the assumptions of your model is. Ideally, you want these assumptions to be grounded in science. Some will likely be philosophical assumptions which cannot be tested empirically, but at least they should fit into a coherent holistic philosophical view. At the very least, you want to make sure you're not assuming away the core parts of the problem.]

For the purposes of safety, you want to be as close to the implementation end of the spectrum as you can get. However, the model side of the spectrum is still useful as: 

• A backup plan which is better than nothing, more so if there is some combination of theoretical and empirical justification for the analogizing
• A way to demonstrate threat models, as the OP suggests
• An intermediate product that helps checking that your theory is heading in the right direction, comparing different research agendas, and maybe even making empirical tests.

Sorry, I was wrong. By Lob's theorem, all versions of ${\text{good}}_{\text{new}}$ are provably equivalent, so they will trust each other.

TLDR: A new proposal for a prescriptive theory of multi-agent rationality, based on generalizing superrationality using local symmetries.

Hofstadter's "superrationality" postulates that a rational agent should behave as if other rational agents are similar to itself. In a symmetric game, this implies the selection of a Pareto efficient outcome. However, it's not clear how to apply this principle to asymmetric games. I propose to solve this by suggesting a notion of "local" symmetry that is much more ubiquitous than global (ordinary) symmetry.

We will focus here entirely on finite deterministic (pure strategy) games. Generalizing this to mixed strategies and infinite games more generally is an important question left for the future.

Quotient Games

Consider a game $\Gamma$ with a set $P$ of players, for each player $i\in P$ a non-empty set $S_i$ or pure strategies and a utility function $u_i:S_{\ast }\to R$, where $S_{\ast }:={\prod }_{j\in P}{S}_j$. What does it mean for the game to be "symmetric"? We propose a "soft" notion of symmetry that is only sensitive to the ordinal ordering between payoffs. While the cardinal value of the payoff will be important in the stochastic context, we plan to treat the latter by explicitly expanding the strategy space.

Given two games ${\Gamma }^{\left(1\right)}$, ${\Gamma }^{\left(2\right)}$, a premorphism from ${\Gamma }^{\left(1\right)}$ to ${\Gamma }^{\left(2\right)}$ is defined to consist of

• A mapping $q:P^{\left(1\right)}\to P^{\left(2\right)}$
• For each $i\in P^{\left(1\right)}$, a mapping ${\overline{q}}_i:S_{q\left(i\right)}^{\left(2\right)}\to S_i^{\left(1\right)}$

Notice that these mappings induce a mapping $\hat{q}:S_{\ast }^{\left(2\right)}\to S_{\ast }^{\left(1\right)}$ via

A premorphism is said to be a homomorpism if $q$ and $\overline{q}$ are s.t. that for any $i\in P^{\left(1\right)}$ and $x,y\in S_{\ast }^{\left(2\right)}$, if $u_{q\left(i\right)}\left(x\right)\le u_{q\left(i\right)}\left(y\right)$ then $u_i\left(\hat{q}\right(x\left)\right)\le u_i\left(\hat{q}\right(y\left)\right)$. Homomorphisms makes games into a category in a natural way.

An automorphism of $\Gamma$ is a homomorphism from $\Gamma$ to $\Gamma$ that has an inverse homomorphism. An automorphism $q$ of $\Gamma$ is said to be flat when for any $i\in P$, if $q\left(i\right)=i$ then ${\overline{q}}_i$ is the identity mapping^[1]. A flat symmetry of $\Gamma$ is a group $G$ together with a group homomorphism $\phi :G\to Aut\left(\Gamma \right)$ s.t. $\phi \left(g\right)$ is flat for all $g\in G$.

Given a flat symmetry $(G,\phi )$, we can apply the superrationality principle to reduce $\Gamma$ to the "smaller" quotient game $\Gamma /G$. The players are $P/G$ i.e. orbits in the original player set. Given $\alpha \in P/G$, we define the set of strategies for player $\alpha$ in the game $\Gamma /G$ to be

This is non-empty thanks to flatness.

Observe that there is a natural mapping $\gamma :(S/G{)}_{\ast }\to S_{\ast }$ given by

Finally, the utility function of $\alpha$ is defined by

It is easy to see that the construction of $\Gamma /G$ is functorial w.r.t. the category structure on games that we defined.

We can generalize this further by replacing game homomorphisms with game quasimorphisms: premorphisms satisfying the following relaxed condition on $q$ and $\overline{q}$:

• For any $i\in P^{\left(2\right)}$ and $x,y\in S_{\ast }^{\left(1\right)}$, if $u_{q\left(i\right)}\left(x\right)<u_{q\left(i\right)}\left(y\right)$ then $u_i\left(\hat{q}\right(x\left)\right)\le u_i\left(\hat{q}\right(y\left)\right)$.

This is no longer closed under composition, so no longer defines a category structure^[2]. However, we can still define flat quasisymmetries and the associated quotient games, and this construction is still functorial w.r.t. the original (not "quasi") category structure. Moreover, there is a canonical homomorphism (not just a quasimorphism) from $\Gamma$ to $\Gamma /G$, even when $G$ is a mere quasisymmetry.

The significance of quasisymmetries can be understood as follows.

The set of all games on a fixed set of players $P$ with fixed strategy sets $\{S_i{\}}_{i\in P}$ naturally forms a topological space^[3] $X$. Given a group $G$ acting on the game via invertible premorphisms, the subset of $X$ where $G$ is a symmetry is not closed, in general. However, the subset of $X$ where $G$ is a quasisymmetry is closed. I believe this will be important when generalizing the formalism to infinite games.

Coarse Graining

What if a game doesn't have even quasisymmetries? Then, we can look for a coarse graining of the game which does have them.

Consider a game $\Gamma$. For each $i\in P$, let $S_i^{\prime }$ be some set and $f_i:S_i\to S_i^{\prime }$ a surjection. Denote $S_{\ast }^{\prime }:={\prod }_{i\in P}{S}_i^{\prime }$. Given any $x\in S_{\ast }^{\prime }$, we have the game ${\Gamma }^x$ in which:

• The set of players is $P$.
• For any $i\in P$, their set of strategies is $S_i^x:=f_i^{-1}\left(x_i\right)$. In particular, it implies that the set of outcomes $S_{\ast }^x$ satisfies $S_{\ast }^x\subseteq S_{\ast }$.
• For any $i\in P$, their utility function is $u_i^x:=u_i{|}_{S_{\ast }^x}$.

If for any $x\in S_{\ast }^{\prime }$ we choose some ${\alpha }_x\in S_{\ast }^x$ this allows us to define the coarse-grained game ${\Gamma }^{\left(\alpha \right)}$ in which:

• The set of players is $P$.
• For any $i\in P$, the set of strategies is $S_i^{\prime }$.
• For any $i\in P$ and $x\in S_{\ast }^{\prime }$, the utility function is $u_i^{\left(\alpha \right)}\left(x\right):=u_i\left({\alpha }_x\right)$.

Essentially, we rephrased $\Gamma$ as an extensive form game in which first the game ${\Gamma }^{\left(\alpha \right)}$ is played and then, if the outcome was $x$, the game ${\Gamma }^x$ is played. This, assuming the expected outcome of game ${\Gamma }^x$ is ${\alpha }_x$.

It is also possible to generalize this construction by allowing multivalued $f$.

Local Symmetry Solutions

Given a game $\Gamma$, a local symmetry presolution (LSP) is recursively defined to be one of the following:

• Assume there is $i\in P$ s.t. for any $j\in P\setminus i$, $|S_j|=1$. Then, any $x^{\ast }\in \arg {max}_{x\in S_{\ast }}{u}_i\left(x\right)$ is an LSP of $\Gamma$. [Effectively single player game]
• Let $\{f_i:S_i\to S_i^{\prime }{\}}_{i\in P}$ be a coarse-graining of $\Gamma$ and for any $x\in S_{\ast }^{\prime }$, let ${\alpha }_x$ be an LSP of ${\Gamma }^x$. Let $\beta$ be an LSP of ${\Gamma }^{\left(\alpha \right)}$. Define $x^{\ast }\in S_{\ast }$ by $x_i^{\ast }:={\alpha }_{{\beta }_i}$. Then, $x^{\ast }$ is an LSP of $\Gamma$.
• Let $G$ be a quasisymmetry of $\Gamma$ and let $y^{\ast }$ be an LSP of $\Gamma /G$. Then, $\gamma \left(y^{\ast }\right)$ is an LSP of $\Gamma$.

It's easy to see that an LSP always exists, because for any game we can choose a sequence of coarse-grainings whose effect is making the players choose their strategies one by one (with full knowledge of choices by previous players). However, not all LSP are "born equal". It seems appealing to prefer LSP which use "more" symmetry. This can be formalized as follows.

The way an LSP is constructed defines the set of coherent outcomes $A\subseteq S_{\ast }$, which is the set of outcomes compatible with the local symmetries^[4]. We define it recursively as follows:

• For the LSP of an effectively single-player game, we set $A:=S_{\ast }$.
• For a coarse-graining LSP, for any $x\in S_{\ast }^{\prime }$, let $A_x$ be the set of coherent outcomes of ${\Gamma }_x$ and let $B$ be the set of coherent outcomes of ${\Gamma }^{\left(\alpha \right)}$. We then define $A:=\{x\in S_{\ast }|f(x)\in B\text{and}x\in A_{f\left(x\right)}\}$. Here, $f:S_{\ast }\to S_{\ast }^{\prime }$ is defined by $f(x{)}_i:=f_i(x_i)$.
• For a quasisymmetry LSP, let $B$ be the set of coherent outcomes of $\Gamma /G$. We then define $A:=\gamma \left(B\right)$.

We define a local symmetry solution (LSS) to be an LSP whose set of coherent outcomes is minimal w.r.t. set inclusion.

Examples

• In the Prisoner's Dilemma, the only LSS is Cooperate-Cooperate.
• In Stag Hunt, the only LSS is Stag-Stag.
• In Chicken, the only LSS is Swerve-Swerve. If we add a finite number of randomized strategies, it opens more possibilities.
• In Battle of the Sexes, assuming a small payoff from going to your preferred place alone, the only LSS is: each player goes to their own preferred place. If we give each player a "coinflip" strategy, then I think the only LSS is both players using the coinflip.
• Consider a pure cooperation game where both players have strategies $\{a,b\}$ and the payoff is 1 if they choose the same strategy and 0 if they choose different strategies. Then, any outcome is an LSS. I think that, if we add a "fair coinflip" strategy, any LSS has payoff at least $\frac{1}{4}$. I believe the LSS payoff will approach optimum if we (i) allow randomization (ii) make the game repeated with time long horizon and (iii) add some constraints on the symmetries, but I'll leave this for a future post.
• Consider a pure cooperation game where both players have strategies $\{a,b\}$, the payoff is 2 if both choose $a$, 1 if both choose $b$ and 0 if they choose different strategies. Then, the only LSS is $aa$.
• In any game, an LSS guarantees each player at least their deterministic-maximin payoff. In particular, in a two-player zero-sum game in which a pure Nash equilibrium exists, any LSS is a Nash equilibrium.

1. ^{^}

   Note that flatness is in general not preserved under composition.

2. ^{^}

   I believe this can be interpreted as a category enriched over the category of sets and partial functions, with the monoidal structure given by Cartesian product of sets.

3. ^{^}

   An affine space, even.

4. ^{^}

   Maybe we can interpret the set of coherent outcomes as a sharp infradistirbution corresponding to the players' joint belief about the outcome of the game.

Master post for multi-agency. See also.