Vanessa Kosoy's Shortform

Vanessa Kosoy

This is a special post for quick takes by Vanessa Kosoy. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Master post for multi-agency. See also.

2Vanessa Kosoy2mo

Here's a proposal for a new game-theory solution concept, related to the infra-Bayesian veil of ignorance. Consider a normal-form game with a set of players . We will be think of as the set of "roles" that an agent may play in a game. We introduce a compact Polish space of agent types s.t. a type is a prior over (i) the role that the agent will play (ii) the types of other agents in the game. Formally, define and require the fixed-point isomorphism Since a covariant functor in , we can build a canonical fixed-point by starting from the one-point space, repeatedly applying the functor and taking inverse-limit of the resulting sequence of space. For any , denote by the set of (pure) strategies of . A (deterministic) policy is a mapping . Note that this definition introduces the assumption that an agent can see the types of the agents they're playing against: this is our counterpart of the "transparent source-code" assumption often used to motivate superrationality. Denote the space of policies by . A metapolicy is a mapping . Semiformal Conjecture: There is a metric on that is, up to a multiplicative constant, equal to the Kantorovich-Rubinstein metric induced by itself, in some natural sense (i.e. we extend to via maximum or sum and then apply the Kantorovich-Rubinstein construction). From we get a metric on by considering the Hausdorff distance between the graphs of policies[1]. Let be the set of game outcomes. Define the "switch POV" function by Given a metapolicy , we define the outcome function by For any , let be the utility function. We define the payoff function by Consider some . Denote by the space of all -Lipschitz metapolicies (in the sense of the metrics and ). We say that a metapolicy is -optimal when for any , This optimality condition describes an infra-Bayesian agent that has ambiguity about the metapolicy but knows it to be -Lipschitz (and also knows that agents of the same type are identical to itself). We say that a metapolicy i

Here's a simple example where this solution concepts works as expected. Consider any game which is a "generalized stag-hunt": that is, it contains a unique Pareto-efficient payoff.

Consider a Pareto-efficient strategy profile , the policy given by and the metapolicy given by . Then, there is some s.t. the -ball around contains only . Hence, there is some s.t. for any and -Lipschitz metapolicy , if then for all . In this case, is an -equilibrium for any , and any -equilibrium is Pareto-efficient.

9Vanessa Kosoy11mo

I propose a taxonomy of 5 possible worlds for multi-agent theory, inspired by Imagliazzo's 5 possible worlds of complexity theory (and also the Aaronson-Barak 5 worlds of AI): * Nihiland: There is not even a coherent uni-agent theory, not to mention multi-agency. I find this world quite unlikely, but leave it here for the sake of completeness (and for the sake of the number 5). Closely related is antirealism about rationality, which I have criticized in the past. In this world it is not clear whether the alignment problem is well-posed at all. * Discordia: There is a coherent uni-agent theory, but no coherent theory of multi-agency. This world is conceivable, since the current understanding of multi-agency is much worse than the understanding of "solitary" agents. In this world, negative-sum conflicts and coordination failures are probably ubiquitous (even among arbitrarily sophisticated agents), because there is no principle of rationality that rules them out. Acausal trade is probably not a thing, or at least rare and fragile. In the context of value learning, there might be no principled way to deal with uncertainty (which could otherwise be regarded as a bargaining problem). There is also no principled solution to multi-user alignment. * Linguistica: There is a coherent theory of multi-agency, but agents are inevitably divided into "types" s.t. only interactions between agents of the same type have strong guarantees. (The name of the world is because we can metaphorically think of the types as different "languages".) An example of how this might happen is reflective oracles, where the type corresponds to the choice of fixed point. Acausal trade probably exists[1], but is segregated by type. Alignment is complicated by the need to specify or learn the human type. * Economica: There is a coherent uni-type theory of multi-agency, but this theory involves desiderata that can only be motivated by multi-agency. Explicitly thinking about multi-agency is necessary

Vanessa Kosoy's Shortform

6

Vanessa Kosoy's Shortform

6