Related to: Bayesians vs. Barbarians

Consequentialism¹ is a catchall term for a vast number of specific ethical theories, the common thread of which is that they take goodness (usually of a state of affairs) to be the determining factor of rightness (usually of an action). One family of consequentialisms that came to mind when it was suggested that I post about my Weird Forms of Utilitarianism class is called "Two-Tier Consequentialism", which I think can be made to connect interestingly to our rationalism goals on Less Wrong. Here's a summary of two-tier consequentialism².

(Some form of) consequentialism is correct and yields the right answer about what people ought to do. But (this form of) consequentialism has many bad features:

It is unimplementable (because to use it correctly requires more calculation than anyone has time to do based on more information than anyone has time to gather and use).

It is "alienating" (because people trying to obey consequentialistic dictates find them very unlike the sorts of moral motivations they usually have, like "I want to be a nice person" or "so-and-so is my friend")³.
It is "integrity-busting" (because it can force you to consider alternatives that are unthinkably horrifying, if there is the possibility that they might lead to the "best" consequences).
It is "virtue-busting" (because it too often requires a deviation from a pattern of behavior that we consider to be an expression of good personal qualities that we would naturally hope and expect from good people).
It is prone to self-serving abuse (because it's easy, when calculating utilities, to "cook the books" and wind up with the outcome you already wanted being the "best" outcome).

It is "cooperation-busting" (because individuals don't tend to have an incentive to avoid free-riding when their own participation in a cooperative activity will neither make nor break the collective good).

To solve these problems, some consequentialist ethicists (my class focused on Railton and Hare) invented "two-tier consequentialism". The basic idea is that because all of these bad features of (pick your favorite kind of) consequentialism, being a consequentialist has bad consequences, and therefore you shouldn't do it. Instead, you should layer on top of your consequentialist thinking a second tier of moral principles called your "Practically Ideal Moral Code", which ought to have the following more convenient properties:

Must be moral principles that identify a situation or class of situations and call for an action in that/those situation(s).
Must be believable. You should not put in your second tier any principles that you can't buy on a deep level.
Must be potentially sturdy. Your second tier of principles should be ones that you could stick to, in the face of both your own fallibility and the possibility that they will sometimes lead you to perform acts that are not, strictly speaking by your favorite consequentialism, right.
Must be useful. They cannot be principles that are as unimplementable as the original favorite consequentialism - you have to be able to bring them to bear quickly and easily.
Must guide you in actions that are consistent with the expressions of virtue and integrity.
Must satisfy a publicity condition. That is, widespread acceptance of this set of principles should be conducive to cooperation and not lead to the same self-serving abuse problem that consequentialism has.
And most importantly, the principles must, collectively, lead you to usually perform actions that your favorite consequentialism would endorse. In fact, part of the point of the second tier is that, in the long run, it should do a better job of making you do things that are right-according-to-consquentialism than actually trying to implement your favorite consequentialism would.

That last part is key, because the two-tier consequentialist is not abandoning consequentialism. Unlike a rule consequentialist, he still thinks that any given action (if his favorite consequentialism is act-based) is right according to the goodness of something (probably a resulting state of affairs), not according to whether they are permitted by his Practically Ideal Moral Code. He simply brainwashes himself into using his Practically Ideal Moral Code because over the long run, this will be for the best according to his initial, consequentialist values.

And here is the reason I linked to "Bayesians vs. Barbarians", above: what Eliezer is proposing as the best course of action for a rationalist society that is attacked from without sounds like a second-tier rationalism. If it is rational - for the society as a whole and in the long run - that there be soldiers chosen by a particular mechanism who go on to promptly obey the orders of their commanding officers? Well, then, the rational society will just have to arrange for that - even if this causes some individual actions to be non-rational - because the general strategy is the one that generates the results they are interested in (winning the war), and the most rational general strategy isn't one that consists of all the most individually rational parts.

In ethics, the three main families are consequentialism (rightness via goodness), deontic ethics (rightness as adherence to duty), and virtue ethics (rightness as the implementation of virtues and/or faithfulness to an archetype of a good person). Inasmuch as rationality and morality are isomorphic, it seems like you could just as easily have a duty-based rationalism or a virtue-based rationalism. I have strong sympathy for two-tier consequentialism as consequentialist ethics go. But it seems like Eliezer is presupposing a kind of consequentialism of rationality, both in that article and in general with the maxim "rationalists should win!" It sounds rather like we are supposed to be rationalists because rationalists win, and winning is good.

I don't know how widely that maps onto other people's motivations, but for my part, I think my intuitions around why I wish to be rational have more to do with considering it a virtue. I like winning just fine, but even if it turns out that devoting myself to ever finer-grained rationalism confers no significant winnings, I will still consider it a valuable thing in and of itself to be rational. It's not difficult to imagine someone who thinks that it is the duty of intelligent beings in general to hone their intellects in the form of rationalist training; such a person would be more of a deontic rationalist.

¹I'm not a consequentialist of any stripe myself. However, my views are almost extensionally equivalent with an extremely overcomplicated interpretation of Nozickian side-constraint rights-based utilitarianism.

²Paraphrased liberally from classroom handouts by Fred Feldman.

³The example we were given in class was of a man who buys flowers for his wife and, when asked why, says, "As her husband, I'm in a special position to efficiently generate utility in her, and considered buying her flowers to be for the best overall." This in contrast to, "Well, because she's my wife and I love her and she deserves a bouquet of carnations every now and then."