We’ve had quite a bit of discussion around LW, and OB, on the questions:

Is there a robust trait, “rationality”, that predicts accurate belief-formation in humans?
If so, how can we measure it? And what kinds of training might help?
Also, does “rationality” in the above sense help people achieve other goals, such as income, happiness, personal growth, positive relationships, or world-saving?

Rationalists that we are, it’s time to put our experiments where our mouths are. So here’s my plan:

Step 1: Assemble a set of questions that might possibly help us understand: (a) how rational people are; (b) where they got that rationality from; and (c) what effects their rationality has on their lives. Include any questions that might help in the formulation of useful conjectures. After collecting the data, look for correlations, spaghetti-at-the-wall style. Try factor analysis.

Step 2 [Perhaps after iterating the quick-and-dirty Step 1 correlational approach a bit, to develop better candidate metrics]: Run some more careful experimental tests of various sorts, both with a “rationality training group” that meets for extended periods of time, and, if LW is willing, with shorter training experiments with randomized LW subgroups. Try to build an atmosphere and knowledge base on LW where more people go out and do useful experiments.

I have an initial questionnaire draft below, although I skipped the answer-choices for brevity. Please post your suggestions for informative questions include and/or to drop. As good suggestions come in, I’ll edit the questionnaire draft to include them. It would be nice if the questionnaire we actually use draws on the combined background of the LW community.

Please also post hypotheses for what kinds of correlations you expect to see and/or to not see, when the questionnaire is actually run. If you note your hypotheses now, before the data comes in, we’ll know we should increase our credence in your theory instead of just accusing you of hindsight bias.

Once we have a good questionnaire draft, I’ll put the questionnaire on the web and call for LW readers to fill out the questionnaire. I’ll also try to get people to fill out the questionnaire from some non-LW groups, e.g. Stanford students. Then I’ll post the questionnaire data, and we can all have fun interpreting it.

Section A. Demographic information. Possible confounders, i.e. variables other than “rationality” that may influence correct beliefs.

Age (from a multiple choice list, so we don’t identify individuals)
Sex [Why: everyone else asks for these, and they might have good reason.
SAT, ACT, and GRE scores, if any. [Why: as a proxy for IQ. IQ helps with many cognitive tasks, probably including rationality questions. We want to be able to tell the difference between “IQ helps people earn money” and “rationality helps people earn money, even after controlling for IQ”.]

Section B. Educational variables that may help cause rationality.

Parents’ education.
Parents’ scientific literacy.
Parents’ religious views.
Whether your parents were crazier than average, and/or more rational than average.
Amount of formal education. College major.
Occupation.
How many non-fiction books did you read in the last month? How many fiction books? [Why: people are probably more likely to give accurate data if we ask about e.g. “the last month”, than if we ask vaguer question like “how much do you usually read?”]
How many self-help or business books did you read in the last month?
When is the last time you sought out someone who was better than you at some skill you wanted to learn, and you asked them questions to try to figure out what you should be doing?
Have you read any books about heuristics and biases?
Have you read OB or LW at all?
Which of the following activities have you trained in: mathematics, programming, engineering or practical tinkering, music, meditation, martial arts, debate, strategy games (go, chess, backgammon, etc.).

Section C. Indicators of real-world success.

Income.
[Marriage and divorce history? Whether you’re in a stable relationship? Whether you’re happy with their relationship? Whether you have an easy time getting dates? How do people usually test for “success” here?]
Number of best friends, for some operationalizations of “best friends” (e.g., people you could borrow $500 from; people with whom you can talk about nearly anything; ?) [What questions are standard, here?]
Whether you’ve ever been in a car accident
Happiness
Whether you’ve been overall “more successful”, “less successful”, or “about as successful” as most people in their high school graduating class, and in their college graduating class.
Whether you’ve “learned more”, “learned less”, or “learned about as much” since graduating {high school / college} as most people in your {high school / college} graduating class.
How often did you exercise in the last week?
Do you smoke?
High school and college GPAs
Do you have a current driver’s license?
Are there any late bills, bounced checks, bad debts, etc. on your credit record?
How many dental cavities did you get in the last two years?

Section D. Standard heuristics and biases questions

[Several standard questions, and variations on standard questions, that I’d rather not give details on so I don’t cause LW readers to get them right. The goal here is to find ways of testing for standard biases among people who have read the standard articles. If anyone has clever ideas for how to disguise the questions, please do email your ideas to annasalamon at gmail, and please don’t post your ideas in the comments.]

Section E. Current beliefs

Religious views.
Are you signed up for cryonics? Views on cryonics.
Views on group (gender, race) differences in IQ. (Not the origins of the differences; just whether there are group differences in today’s adults).
Views on the odds nuclear war over the next few decades
How good-looking are you, relative to other people of your age and gender?
Views on Pascal’s wager
Views on consciousness
Views on evolution
Views on whether global warming is happening, and whether it is significant

[Why: to see how good people are at forming accurate beliefs. And to get a bit of information on whether the above beliefs are accurate, by seeing whether the beliefs correlate with other rationality-indicators.]

Section F. Value placed on truth

Is it better to have accurate beliefs, or beliefs that give you morale or meaning?
Do you try to believe good things about your friends?
Do you try to believe good things about people who are different from you (e.g., people of different ethnic or religious backgrounds, people from different countries, people with different sexual orientations)? Why?
How important is it to you to have accurate beliefs?
Imagine a scale from 1 to 10 that measures the process by which you form beliefs. Let “1” mean “there may be emotional or other non-rational pressures, but those pressures have little impact on my resulting conclusions”. Let “10” mean “I basically just made up these beliefs because they felt good, seemed socially useful, matched my fears, or had some other non-truth-related property”. On this scale, how did you form your beliefs concerning:
- a. Yourself
- b. Your friends and family
- c. How to make money, gain skills, make friends, etc.
- d. The larger world (e.g., how likely the economy is to do well in the next few years, whether nuclear weapons or global warming pose real risks, what impact different political parties might have, etc.).

Section G. Attempts to seek information

In the last week, how much time did you spend trying to understand:
- a. Yourself
- b. Your friends and family
- c. How to make money, gain skills, make friends, etc.
- d. The larger world (e.g., how likely the economy is to do well in the next few years, whether nuclear weapons or global warming pose real risks, what impact different political parties might have, etc.).
How well do you know your friends and family? How do you know how well you know them?
How well do you know yourself? How do you know?
How well do you understand those aspects of the world that enable real-world measurable success, e.g. income? How do you know?
Have you experimented with different ways to do your job effectively?
Have well do you understand the larger world? How do you know?
Think about the last time you had a fight or conflict with someone. How much time did you spend rehearsing the evidence for your side? How much time did you spend trying with honest curiosity to figure out what happened?
How often do you notice that one of your pieces of knowledge conflicts with your model of some other part of the world (e.g., that you don't understand why the floating toy in the pool bops to the top at the angle it does, or why

Section H: Models of one's own thinking skill [This is the only section with open-ended rather than multiple-choice questions. Respondants can skip this section while filling out the rest]

What’s the worst mistake you made in the last year? What did you do about it?
What are the largest gaps in your current thinking skills?
What are your greatest strengths as a thinker?
On what topics are you most prone to self-deception?
What is the biggest improvement you’ve made in your ability to form accurate beliefs over the last year?
What safeguards do you use, to try to notice flaws in your own beliefs?

ADDED: The idea here is not to generate an actual, first-round test of individuals' rationality. The idea is to take a bunch of questions that might plausibly correlate with that nebulous mix of concepts, "rationality", and to see how well those questions correlate with one another. We won't get a "your're more rational than 70% of the population" out of this questionnaire: no way, no how. We may well get a some suggestive data about clusters of questions and answers where respondants' answers tend to correlate with one another, and so suggest possible underlying factors worth more careful investigation.

Psychologists often do cheap, bad studies before they do slow, careful, expensive studies, to get an initial look at what might be true.