The Best of LessWrong

#1

Strong evidence is much more common than you might think. Someone telling you their name provides about 24 bits of evidence. Seeing something on Wikipedia provides enormous evidence. We should be willing to update strongly on everyday events.

6Joe Carlsmith

I really like this post. It's a crisp, useful insight, made via a memorable concrete example (plus a few others), in a very efficient way. And it has stayed with me.

6Ben Pace

This post is in my small list of +9s that I think count as a key part of how I think, where the post was responsible for clarifying my thinking on the subject. I've had a lingering confusion/nervousness about having extreme odds (anything beyond 100:1) but the name example shows that seeing odds ratios of 20,000,000:1 is just pretty common. I also appreciated Eliezer's corollary: "most beliefs worth having are extreme", this also influences how I think about my key beliefs. (Haha, I just realized that I curated it back when it was published.)

#6

Rationalism before the Sequences

What was rationalism like before the Sequences and LessWrong? Eric S. Raymond explores the intellectual roots of the rationalist movement, including General Semantics, analytic philosophy, science fiction, and Zen Buddhism.

#7

Lies, Damn Lies, and Fabricated Options

When people disagree or face difficult decisions, they often include fabricated options - choices that seem possible but are actually incoherent or unrealistic. Learning to spot these fabricated options can help you make better decisions and have more productive disagreements.

1Elizabeth

My first reaction when this post came out was being mad Duncan got the credit for an idea I also had, and wrote a different post than the one I would have written if I'd realized this needed a post. But at the end of the day the post exists and my post is imaginary, and it has saved me time in conversations with other people because now they have the concept neatly labeled.

#16

Taboo "Outside View"

People use the term "outside view" to mean very different things. Daniel argues this is problematic, because different uses of "outside view" can have very different validity. He suggests we taboo "outside view" and use more specific, clearer language instead.

8Alex_Altair

This is a negative review of an admittedly highly-rated post. The positives first; I think this post is highly reasonable and well written. I'm glad that it exists and think it contributes to the intellectual conversation in rationality. The examples help the reader reason better, and it contains many pieces of advice that I endorse. But overall, 1) I ultimately disagree with its main point, and 2) it's way too strong/absolutist about it. Throughout my life of attempting to have true beliefs and take effective actions, I have quite strongly learned some distinction that maps onto the ideas of inside and outside view. I find this distinction extremely helpful, and specifically, remembering to use (what I call) the outside view often wins me a lot of Bayes points. When I read through the Big Lists O' Things, I have these responses; * I think many of those things are simply valid uses of the terms[1] * People using a term wrong isn't a great reason[2] to taboo that term; e.g. there are countless mis-uses of the concept of "truth" or "entropy" or "capitalism", but the concepts still carve reality * Seems like maybe some of these you heard one person use once, and then it got to go on the list? A key example of the absolutism comes from the intro: "I recommend we permanently taboo “Outside view,” i.e. stop using the word and use more precise, less confused concepts instead." (emphasis added). But, as described in the original linked sequence post, the purpose of tabooing a word is to remember why you formed a concept in the first place, and see if that break-down helps you reason further. The point is not to stop using a word. I think the absolutism has caused this post to have negative effects; the phrase "taboo the outside view" has stuck around as a meme, and in my memory, when people use it it has not tended to be good for the conversation. Instead, I think the post should have said the following. * The term "outside view" can mean many things that can

#19

Split and Commit

When you encounter evidence that seems to imply X, Duncan suggests explicitly considering both "What kind of world contains both [evidence] and [X]?" and "What kind of world contains both [evidence] and [not-X]?".

Then commit to preliminary responses in each of those possible worlds.

#23

Trapped Priors As A Basic Problem Of Rationality

Scott Alexander explores the idea of "trapped priors" - beliefs that become so strong they can't be updated by new evidence, even when that evidence should change our mind.

#28

The Rationalists of the 1950s (and before) also called themselves “Rationalists”

The rationalist scene based around LessWrong has a historical predecessor! There was a "Rationalist Association" founded in 1885 that published works by Darwin, Russell, Haldane, Shaw, Wells, and Popper. Membership peaked in 1959 with over 5000 members and Bertrand Russell as President.

#31

Feature Selection

In this short story, an AI wakes up in a strange environment and must piece together what's going on from limited inputs and outputs. Can it figure out its true nature and purpose?

#32

Cup-Stacking Skills (or, Reflexive Involuntary Mental Motions)

Duncan explores a concept he calls "cup-stacking skills" - extremely fast, almost reflexive mental or physical abilities developed through intense repetition. These can be powerful but also problematic if we're unaware of them or can't control them.

#34

Self-Integrity and the Drowning Child

"The Watcher asked the class if they thought it was right to save the child, at the cost of ruining their clothing. Everyone in there moved their hand to the 'yes' position, of course. Except Keltham, who by this point had already decided quite clearly who he was, and who simply closed his hand into a fist, otherwise saying neither 'yes' nor 'no' to the question, defying it entirely."

#37

Simulacrum 3 As Stag-Hunt Strategy

"Simulacrum Level 3 behavior" (i.e. "pretending to pretend something") can be an effective strategy for coordinating on high-payoff equilibria in Stag Hunt-like situations. This may explain some seemingly-irrational corporate behavior, especially in industries with increasing returns to scale.

5Raymond Arnold

This gave a satisfying "click" of how the Simulacra and Staghunt concepts fit together. Things I would consider changing: 1. Lion Parable. In the comments, John expands on this post with a parable about lion-hunters who believe in "magical protection against lions." That parable is actually what I normally think of when I think of this post, and I was sad to learn it wasn't actually in the post. I'd add it in, maybe as the opening example. 2. Do we actually need the word "simulacrum 3"? Something on my mind since last year's review is "how much work are the words "simulacra" doing for us? I feel vaguely like I learned something from Simulacra Levels and their Interactions, but the concept still feels overly complicated as a dependency to explain new concepts. If I read this post in the wild without having spent awhile grokking Simulacra I think I'd find it pretty confusing. But, meanwhile, the original sequences talked about "belief in belief". I think that's still a required dependency here, but, a) Belief in Belief is a shorter post, and I think b) I think this post + the literal words "belief in belief" helps grok the concept in the first place. On the flipside, I think the Simulacra concept does help point towards an overall worldview about what's going on in society, in a gnarlier way than belief-in-belief communicates. I'm confused here. Important Context A background thing in my mind whenever I read one of these coordination posts is an older John post: From Personal to Prison Gangs. We've got Belief-in-Belief/Simulacra3 as Stag Hunt strategies. Cool. They still involve... like, falsehoods and confusion and self-deception. Surely we shouldn't have to rely on that? My hope is yes, someday. But I don't know how to reliably do it at scale yet. I want to just quote the end of the prison gangs piece:

7Elizabeth

Most of the writing on simulacrum levels have left me feeling less able to reason about them, that they are too evil to contemplate. This post engaged with them as one fact in the world among many, which was already an improvement. I've found myself referring to this idea several times over the last two years, and it left me more alert to looking for other explanations in this class.

#41

Catching the Spark

Logan Strohl outlines a structured approach for tapping into genuine curiosity and embarking on self-driven investigations, inspired by the spirit of early scientific pioneers. They hopes this method can help people overcome modern hesitancy to make direct observations, and draw their own conclusions.

8Logan Strohl

* Oh man, what an interesting time to be writing this review! * I've now written second drafts of an entire sequence that more or less begins with an abridged (or re-written?) version of "Catching the Spark". The provisional title of the sequence is "Nuts and Bolts Of Naturalism". (I'm still at least a month and probably more from beginning to publish the sequence, though.) This is the post in the sequence that's given me the most trouble; I've spent a lot of the past week trying to figure out where I stand with it. * I think if I just had to answer "yes" or "no" to "do I endorse the post at this point", I'd say "yes". I continue to think it lays out a valuable process that can result in a person being much more in tune with what they actually care about, and able to see much more clearly how they're relating to a topic that they might want to investigate. * As I re-write the post for my new sequence, though, I have two main categories of objections to it, both of which seem to be results of my having rushed to publish it as a somewhat stand-alone piece so I could get funding for the rest of my work. * One category of objection I have is that it tries to do too much at once. It tries to give instructions for the procedure itself, demonstrate the procedure, and provide a grounding in the underlying philosophy/worldview. It's perhaps a noble goal to do all of that in one post, but I don't think I personally am actually capable of that, and I think I ended up falling short of my standards on all three points. If you've read my sequence Intro To Naturalism, you might possibly share my feeling that the philosophy parts of Catching the Spark are some kind of desperate and muddled. Additionally, I think the demonstration parts are insufficiently real and insufficiently diverse. When I wrote the post, I mostly looked back at my memories to find illustrative examples, rather than catching my examples in real time. A version of this with demonstrations that meet my stan

#43

Shoulder Advisors 101

Duncan discusses "shoulder advisors" – imaginary simulations of real friends or fictional characters that can offer advice, similar to the cartoon trope of a devil and angel on each shoulder, but more nuanced. He argues these can be genuinely useful for improving decision making and offers tips on developing and using shoulder advisors effectively.

AI ALIGNMENT FORUM
The Best of LessWrong
AF

Rationality

Optimization

World

Practical

AI Strategy

Technical AI Safety