Consider any finite two-player game in normal form (each player can have any finite number of strategies, we can also easily generalize to certain classes of infinite games). Let $S_A$ be the set of pure strategies of player $A$ and $S_B$ the set of pure strategies of player $B$. Let $u_A:S_A\times S_B\to R$ be the utility function of player $A$. Let $(\alpha ,\beta )\in \Delta S_A\times \Delta S_B$ be a particular (mixed) outcome. Then the alignment of player $B$ with player $A$ in this outcome is defined to be:

$$a_{B/A}(\alpha ,\beta ):=\frac{E_{\alpha \times \beta }\left[u_A\right]-{min}_{{\beta }^{\prime }\in S_B}{E}_{\alpha \times {\beta }^{\prime }}\left[u_A\right]}{{max}_{{\beta }^{\prime }\in S_B}{E}_{\alpha \times {\beta }^{\prime }}\left[u_A\right]-{min}_{{\beta }^{\prime }\in S_B}{E}_{\alpha \times {\beta }^{\prime }}\left[u_A\right]}\in [0,1]$$

Ofc so far it doesn't depend on $u_B$ at all. However, we can make it depend on $u_B$ if we use $u_B$ to impose assumptions on $(\alpha ,\beta )$, such as:

• $\beta$ is a $u_B$-best response to $\alpha$ or
• $(\alpha ,\beta )$ is a Nash equilibrium (or other solution concept)

Caveat: If we go with the Nash equilibrium option, $a_{B/A}$ can become "systematically" ill-defined (consider e.g. the Nash equilibrium of matching pennies). To avoid this, we can switch to the extensive-form game where $B$ chooses their strategy after seeing $A$'s strategy.

So, something like "fraction of preferred states shared" ? Describe preferred states for P1 as cells in the payoff matrix that are best for P1 for each P2 action (and preferred stated for P2 in a similar manner) Fraction of P1 preferred states that are also preferred for P2 is measurement of alignment P1 to P2. Fraction of shared states between players to total number of preferred states is measure of total alignment of the game.

For 2x2 game each player will have 2 preferred states (corresponding to the 2 possible action of the opponent). If 1 of them will be the same cell that will mean that each player is 50% aligned to other (1 of 2 shared) and the game in total is 33% aligned (1 of 3), This also generalize easily to NxN case and for >2 players.

And if there are K multiple cells with the same payoff to choose from for some opponent action we can give 1/K to them instead of 1.

(it would be much easier to explain with a picture and/or table, but I'm pretty new here and wasn't able to find how to do them here yet)

I think this is backward.  The game's payout matrix determines the alignment.  Fixed-sum games imply (in the mathematical sense) unaligned players, and common-payoff games ARE the definition of alignment.  

When you start looking at meta-games (where resource payoffs differ from utility payoffs, based on agent goals), then "alignment" starts to make sense as a distinct measurement - it's how much the players' utility functions transform the payoffs (in the sub-games of a series, and in the overall game) from fixed-sum to common-payoff.

Correlation between player payouts? In a zero sum game it is -1, when payouts are perfectly aligned it is +1, if payouts are independent it is 0.

I'll take a shot at this. Let $A$ and $B$ be the sets of actions of Alice and Bob. Let $o_n:B\to \{1,...n\}$ (where 'n' means 'nice') be function that orders $B$ by how good the choices are for Alice, assuming that Alice gets to choose second. Similarly, let $o_s:B\to \{1,...,n\}$ (where 's' means 'selfish') be the function that orders $B$ by how good the choices are for Bob, assuming that Alice gets to choose second. Choose some function $\psi$ measuring similarity between two orderings of a finite set (should range over $[-1,1]$); the alignment of $B$ with $A$ is then $\psi (o_n,o_s)$.

Example: in the prisoner's dilemma, $B=\{c,d\}$, and $o_n$ orders $c>d$ whereas $o_s$ orders $d>c$. Hence $\psi (o_n,o_m)$ should be $-1$, i.e., Bob is maximally unaligned with Alice. Note that this makes it different from Mykhailo's answer which gives alignment $0.5$, i.e., medium aligned rather than maximally unaligned.

This seems like an improvement over correlation since it's not symmetrical. In the game where Alice and Bob both get to choose numbers $x,y\in \{1,2\}$ and Alice's utility function outputs $y+x$ whereas Bob's outputs $y-x$, Bob would be perfectly aligned with Alice (his $o_n$ and $o_s$ both order $2>1$) but Alice perfectly unaligned with Bob (her $o_n$ orders $1>2$ but her $o_s$ orders $2>1$).

I believe this metric meets criteria 1,3,4 you listed. It could be changed to be sensitive to players' decision theories by changing $o_s$ (for alignment from Bob to Alice) to be the order output by Bob's decision theory, but I think that would be a mistake. Suppose I build an AI that is more powerful than myself, and the game is such that we can both decide to steal some of the other's stuff. If the AI does this, it leads to -10 utils for me and +2 for it (otherwise 0/0); if I do it, it leads to -100 utils for me because the AI kills me in response (otherwise 0/0). This game is trivial: the AI will take my stuff and I'll do nothing. Also, the AI is maximally unaligned with me. Now suppose I become as powerful as the AI and my 'take AI's stuff' becomes -10 for AI, +2 for me. This makes the game a prisoner's dilemma. If we both run UDT or FDT, we would now cooperate. If $o_s$ is the ordering of the AI's decision theory, this would mean the AI is now aligned with me, which is odd since the only thing that changed is me getting more powerful. With the original proposal, the AI is still maximally unaligned with me. More abstractly, game theory assumes your actions have influence on the other player's rewards (else the game is trivial), so if you cooperate for game-theoretical reasons, this doesn't seem to capture what we mean by alignment.