In reply to Eliezer's Contrarian Status Catch 22 & Sufficiently Advanced Sanity. I accuse Eliezer of encountering a piece of Advanced Wisdom.

Unreason is something that we should fight against. Witch burnings, creationism & homeopathy are all things which should rightly be defended against for society to advance. But, more subtly, I think reason is in some ways, is also a dangerous phenomena that should be guarded against. I am arguing not against the specific process of reasoning itself, it is the attitude which instinctually reaches for reason as the first tool of choice when confronting a problem. Scott Aaronson called this approach bullet swallowing when he tried to explain why he was so uncomfortable with it. Jane Galt also rails against reason when explaining why she does not support gay marriage.

The most recent financial crisis is a another example of what happens when reason is allowed to dominate. Cutting away all the foggy noise & conspiracy theories, the root cause of the financial crisis was that men who believed a little too much in reason became divergent from reality and reality quickly reminded them of that. The problem was not that the models were wrong, it was that what they were trying to accomplish was unmodelable. Nassim Nicholas-Taleb's The Black Swan explains this far better than I can.

A clear-eyed look back on history reveals many other similar events in which the helm of reason has been used to champion disastrous causes: The French Revolution, Communism, Free Love Communes, Social Darwinism & Libertarianism (I am not talking about affective death spirals here). Now, one might argue at this point that those were all specific examples of bad reasoning and that it's possible to carefully strip away the downsides of reason with diligent practice. But I don't believe it. Fundamentally, I believe that the world is unreasonable. It is, at it's core, not amenable to reason. Better technique may push the failure point back just a bit further but it will never get rid of it.

So what should replace reason? I nominate accepting the primacy of evidence over reason. Reason is still used, but only reluctantly, to form the minimum span possible between evidence & beliefs. From my CS perspective, I make the analogy between shifting from algorithm centered computation to data centric. Rather than try to create elaborate models that purport to reflect reality, strive to be as model-free as possible and shut up and gather.

If the reason based approach is a towering skyscraper, an evidence based approach would be an adobo hut. Reasoning is sexy and new but also powerful, it can do a lot of things and do them a lot better. The evidence based approach, on the other hand, does just enough to matter and very little more. The evidence based approach is not truth seeking in the way the reason based approach is. A declaration of truth, built on a large pile of premises is worse than a statement of ignorance. This, I think is what Scott Aaaronson was referring to when he said "What you've forced me to realize, Eliezer, and I thank you for this: What I'm uncomfortable with is not the many-worlds interpretation itself, it's the air of satisfaction that often comes with it."