Epistemic status: This is a step towards formalizing some intuitions about AI. It is closely related to Vanessa Kosoy’s “Descriptive Agent Theory” - but I want to concretize the question, explain the reason that it is true in some form, and try to think through and provide some intuition about why it would matter. I welcome pushback about the claim or the way to operationalize it.

The intuition that most minds are irrational is about the space of possible minds. I have been told that others also haven’t formalized the claim well, and have not found a good explanation. The intuition is that, as a portion of the total possible space, a measure-zero subset of “minds” will fulfill the basic requirements of rational agents. Unfortunately, none of this is really well defined, so I’m writing down my understanding of the problem and what seem like the paths forward. I will note that this isn’t my main focus, and I think it’s important to note that this is only indirectly related to safety, and is far more closely related to deconfusion. However, it seems important when thinking about both artificial intelligence agents, and human minds.

To outline the post, I’ll first prove that in a very narrow case, that almost all possible economic agents are irrational. After that, I’ll talk about why the most general case of any computational process - which includes anything that generates output in response to input, any program or MDP - can be considered a decision process (“what should I output,”) but if all we want is for it to output something, the proportion of agents which are rational is undecidable for technical reasons. I’ll then make a narrower case about chess agents, showing that in a fairly reasonable sense, almost all such agents are irrational. And finally, I’ll talk about what would be needed to make progress on the problems, and some interesting issues or potentially tractable mathematical approaches.

Most economic preferences are irrational

I’ll start with a toy example of the economic notion of agents which I think is not particularly useful, except to explain the intuition. Imagine a person who, when at the store, looks at a candy bar and says they’d rather have the candy bar than two dollars - but in exactly the same scenario, they are willing to sell a candy bar they have for only one dollar. Clearly, if the person spends some time trading back and forth, th...

Mind Space