Partially in response to: The curse of identity

Related to: Humans are not automatically strategic, That other kind of status, Approving reinforces low-effort behaviors.

Joe studies long hours, and often prides himself on how driven he is to make something of himself. But in the actual moments of his studying, Joe often looks out the window, doodles, or drags his eyes over the text while his mind wanders. Someone sent him a link to which college majors lead to the greatest lifetime earnings, and he didn't get around to reading that either. Shall we say that Joe doesn't really care about making something of himself?

The Inuit may not have 47 words for snow, but Less Wrongers do have at least two words for belief. We find it necessary to distinguish between:

Anticipations, what we actually expect to see happen;
Professed beliefs, the set of things we tell ourselves we “believe”, based partly on deliberate/verbal thought.

This distinction helps explain how an atheistic rationalist can still get spooked in a haunted house; how someone can “believe” they’re good at chess while avoiding games that might threaten that belief [1]; and why Eliezer had to actually crash a car before he viscerally understood what his physics books tried to tell him about stopping distance going up with the square of driving speed. (I helped Anna revise this - EY.)

A lot of our community technique goes into either (1) dealing with "beliefs" being an evolutionarily recent system, such that our "beliefs" often end up far screwier than our actual anticipations; or (2) trying to get our anticipations to align with more evidence-informed beliefs.

And analogously - this analogy is arguably obvious, but it's deep, useful, and easy to overlook in its implications - there seem to be two major kinds of wanting:

Urges: concrete emotional pulls, produced in System 1's perceptual / autonomic processes
(my urge to drink the steaming hot cocoa in front of me; my urge to avoid embarrassment by having something to add to my accomplishments log)
Goals: things we tell ourselves we’re aiming at, within deliberate/verbal thought and planning
(I have a goal to exercise three times a week; I have a goal to reduce existential risk)

Implication 1: You can import a lot of technique for "checking for screwy beliefs" into "checking for screwy goals".

Urges, like anticipations, are relatively perceptual-level and automatic. They're harder to reshape and they're also harder to completely screw up. In contrast, the flexible, recent "goals" system can easily acquire goals that are wildly detached from what we actually do, wildly detached from any positive consequences, or both. Some techniques you can port straight over from "checking for screwy beliefs" to "checking for screwy goals" include:

The fundamental:

"What's the positive consequence?" This is the equivalent of "What's the evidence?" for beliefs. All the other cases involve not asking it, or not asking hard enough.

The Hansonian:

Goals as clothes / goals as tribal affiliation: “We are people who have free software (/ communism / rationality / whatever) as our goal”. Before you install Linux, do you think "What's the positive consequence of installing Linux?" or does it just seem like the sort of thing a free-software-supporter would do? (EY says: What positive consequence is achieved by marching in an Occupy Wall Street march? Can you remember anyone stating one, throughout the whole affair - "if we march, X will happen because of Y"?)
Goals as a signal of one’s value as an ally: Sheila insists that she wants to get a job. We inspect her situation and she's not trying very hard to get a job. But she's in debt to a lot of her friends and is borrowing more to live on a month-to-month basis. It's not hard to see why Sheila would internally profess strongly that she has a goal of getting a job.
Goals as personal fashion statements: A T-Shirt that says “Give me coffee and no one gets hurt” seems to state a very strong desire for coffee. This is clearly a goal professed directly to affect how others see you, and it's more a question of affecting a 'style' than anything directly tribal or status-y.

The satiating:

Having goals as optimism: "I intend to lose weight" can be created by much the same sort of internal processes that would make you believe "I will lose weight", in cases where the goal (belief) would not yet seem very plausible to an outside view.
Having goals as apparent progress: My current to-do list has "write thank-you notes for wedding gifts". This makes me feel like I've appeased the demand for internal attention by having a goal. (EY: I have "send Anna and Carl their wedding gift" on my todo list. This was very effective at appeasing the need to send them a wedding gift.)

Implication 2: "Status" / "prestige" / "signaling" / "people don't really care about" is way overused to explain goal-urge delinkages that can be more simply explained by "humans are not agents".

This post was written partially in response to The Curse of Identity, wherein Kaj recounts some suboptimal goal-action linkages - wanting to contribute to the Singularity, then teaching himself to feel guilty whenever not working; founding the Finnish Pirate Party, then becoming the spokesperson which involved tasks he wasn't good at; helping Eliezer on writing his book, and feeling demotivated because it seemed like work "anyone could do" (which is just the sort of work that almost nobody is motivated to do).

Kaj forms the generalization "as soon as my brain adopted a cause, my subconscious reinterpreted it as the goal of giving the impression of doing prestigious work for the cause". I worry that our community has a tendency to explain as e.g. status signaling or "people really don't care about X", observations that can also be explained by less malice/selfishness and more "our brains have known malfunctions at linking goals to urges". People are as bad at looking into hospitals for their own health as for the sake of their parents' health; Kaj didn't actually gain much prestige from feeling guilty about his relaxation time.

We do have a status urge. It does affect a lot of things. People do tend to massively systematically understate it in much the same way that Victorians pretended that sex wasn't everywhere. But that's not the same cognitive problem as "Our brain is pretty bad at linking effective behaviors to goals, and will sometimes reward us for just doing things that seem roughly associated with the goal, instead of actions that cause the consequence of the goal being achieved." And our brains not being coherent agents is something that's even more massive than status.

Implication 3: Humans cannot live by urges alone

Like beliefs, goals often get much wackier than urges. I've seen a number of people react to this realization by concluding that they should give up on having goals, and lead an authentic life of pure desire. This wouldn't work any more than giving up on having beliefs. To precisely anticipate how long it takes a ball to fall off a tower, you have to manipulate abstract beliefs about gravitational acceleration. I have an urge to drive a car that runs smoothly, but if I didn't also have a goal of having a well-maintained car, I would never get around to having it serviced - I have no innate urge to do that.

I really have seen multiple people (some of whom I significantly cared about) malfunctioning as a result of misinterpreting this point. As a stand-alone system for pulling your actions, urges have all kinds of problems. Urges can pull you to stare at an attractive stranger, to walk to the fridge, and even to sprint hard for first base when playing baseball. But unless coupled with goals and far-mode reasoning, urges will not pull you to the component tasks required for any longer-term goods. When I get into my car I have a definite urge for it not to be broken. But absent planning, there would never be a moment when the activity I most desired was to take my car for an oil change. To find and keep a job (let alone a good job), live in a non-pigsty, or learn any skills that are not immediately rewarding, you will probably need goals. Even though human goals can easily turn into fashion statements and wishful thinking.

Implication 4: Your agency failures do not imply that your ideals are fake.

Obvious but it needs to be said: People are as bad at looking into hospitals for their own health as for the sake of their parents' health. It doesn't mean that they don't really care about their parents, and it doesn't mean that they don't really care about survival. They would probably run away pretty fast from a tiger, where the goal connected to the urge in an ancestrally more reliable way and hence made them more 'agenty'; and they might fight hard to defend their parents from a tiger too.

There's a very real sense in which our agency failures imply that human beings don't have goals, but this doesn't mean that our ungoaly ideals are any more ungoaly than anything else. Ideals can be more ungoaly because they're sometimes about faraway things or less ancestral things - it's probably easier to improve your agency on less idealy goals that link more quickly to urges - but as entities which can look over our own urges and goals and try to improve our agentiness, there's no rule which says that we can't try to solve some hard problems in this area as well as some easy ones.[2]

Implication 5: You can align urges and goals using the same sort of effort and training that it takes to align anticipations and beliefs.

Although I've heard people saying that we discuss willpower-failure too much on Less Wrong, most of the best stuff I've read has been outside Less Wrong and hasn't made contact with us. For a starting guide to many such skills, see Eat That Frog by Brian Tracy [3]. Some basic alignment techniques include:

Get in the habit of asking "What is the positive consequence?" (Probably more needs to be written about this so that your brain doesn't just answer "I'll be a free software supporter!" which is not what we mean to ask.)
Andrew Critch's "greedy algorithm": Whenever you catch yourself really wanting to do something you want to want, immediately reward yourself - by feeding yourself an M&M, or if that's too difficult, immediately pumping your fist and saying "Yes!"
Whenever you sit down to work, naming a single, high-priority accomplishment for that session. Visualizing that accomplishment, and its positive rewarding consequences, until you have an urge for it to happen (instead of just having an urge to log today's hours).

And much the same way that a lot of craziness stems, not so much from "having a wrong model of the world", as "not bothering to have a model of the world", a lot of personal effectiveness isn't so much about "having the right goals" as "bothering to have goals at all" - where unpacking this somewhat Vassarian statement would lead us to ideas like "bothering to have something that I check my actions' consequences against, never mind whether or not it's the right thing" or "bothering to have some communication-related urge that animates my writing when I write, instead of just sitting down to log a certain number of writing hours during which I feel rewarded from rearranging shiny words".

Conclusion:

Besides an aspiring rationalist, these days I call myself an "aspiring consequentialist".

[1] IMO the case of somebody who has the belief "I am good at chess", but instinctively knows to avoid strong chess opponents that would potentially test the belief, ought to be a more central example in our literature than the person who believes they have an dragon in their garage (but instinctively knows that they need to specify that it's invisible, inaudible and generates no carbon dioxide, when we show up with the testing equipment).

[2] See also Ch. 20 of Methods of Rationality:

Professor Quirrell: "Mr. Potter, in the end people all do what they want to do. Sometimes people give names like 'right' to things they want to do, but how could we possibly act on anything but our own desires?"

Harry: "Well, obviously I couldn't act on moral considerations if they lacked the power to move me. But that doesn't mean my wanting to hurt those Slytherins has the power to move me more than moral considerations!"

[3] Thanks to Patri for recommending this book to me in response to an earlier post. It is perhaps not written in the most LW-friendly language -- but, given the value of these skills, I’d recommend wading in and doing your best to pull useful techniques from the somewhat salesy prose. I found much of value there.