Pitfalls of Rationality

Pitfalls of Rationality are frequent error modes, obstacles or problems that arise when people try to practice rationality, or engage with rationality-related materials. Related concepts include the "valley of bad rationality".

There are two threads touched in posts under this tag:

1. Things that go wrong when people try to be more rational and they unintentionally end up making things worse.
2. Arguably, why haven't rationalists visible succeeded at their bold and ambitious goals yet?

Regarding the first point, from Incremental Progress and the Valley:

Ah.  Well, here's the the thing:  An incremental step in the direction of rationality, if the result is still irrational in other ways, does not have to yield incrementally more winning.

The optimality theorems that we have for probability theory and decision theory, are for perfect probability theory and decision theory.  There is no companion theorem which says that, starting from some flawed initial form, every incremental modification of the algorithm that takes the structure closer to the ideal, must yield an incremental improvement in performance.  This has not yet been proven, because it is not, in fact, true.

See also: Criticisms of the Rationalist Movement, Value of Rationality