tldr: a Demon Thread is a discussion where everything is subtly warping towards aggression and confusion (i.e. as if people are under demonic influence), even if people are well intentioned and on the same 'side.' You can see a demon thread coming in advance, but it's still hard to do anything about.

("Flame Wars" are similar but I felt the connotation was more like "everything has already gone to hell, and people aren't even pretending to be on the same side")

I kept wanting to reference this post when discussing internet discussion policy, and I kept forgetting that nobody has written it yet. So here it is.

Suggested Background Reading:

Politics is Hard Mode (Rob Bensinger)
Civility is Never Neutral (Ozy)
Writing that Provokes Comments (me)
Musings on Double Crux and Productive Disagreement (me)

If someone in the future linked you to this post, it's probably because a giant sprawling mess of angry, confused comments is happening - or is about to happen - and it's going to waste a lot of time, make people upset, and probably less likely to listen to each other about whatever the conversation ostensibly is about.

I have some ideas on what to do instead, which I discuss in this followup post.

But for now, this post is meant to open a discussion, explore the mechanics of how demon threads work, and then in the comments brainstorm solutions about how to handle them.

Wrong On the Internet

I find “Someone Is Wrong On the Internet” to be a weird, specific feeling.

It's distinct from someone being factually wrong - people can be wrong, point it out, and hash out their disagreements without a problem. But a common pattern I’ve witnessed (and experienced) is to notice someone being wrong in a way that feels distinctly bad, like if you don’t correct them, something precious will get trampled over.

This is when people seem most prone to jump into the comments, and it’s when I think people should be most careful.

Sometimes there actually is an important thing at stake.

There usually isn’t.

It often feels like there is, because our social intuitions were honed for tribes of a hundred or two, instead of a world of 7 billion. We live in a different world now. If you actually want to have an impact on society, yelling at each other on the internet is almost certainly not the best way to do so.

When there actually is something important at stake, I think there are usually better plans than “get into a giant internet argument.” Think about what your goals are. Devise a plan that actually seems like it might help.

Different situations call for different plans. For now, I want to talk about the common anti-pattern that often happens instead.

Demon Threads are explosive, frustrating, many-tentacled conversations that draw people in regardless of how important they are. They come in two forms:

Benign Demon Threads are mostly time wasting. Nobody gets that angry, it's just a frustrated mess of "you're wrong" "no you're wrong" and then people spend loads of digital ink arguing about something that doesn't matter much.
Malignant Demon Threads feed upon emotions of defensiveness, anger, tribal affiliation and righteousness - and inflame those emotions, drawing more people into the fire.

(A malignant demon thread is cousin to the flame war - people hurling pure insults at each other. What makes a malignant demon thread insideous is the way it can warp discussion even among people who are earnestly trying to communicate, seek truth and solve problems)

If you find yourself in a malignant demon thread, I think it's likely you are not only not helping, but are actually hurting your cause.

The Demon Seed

How to write so that people will comment [disclaimer: not necessarily good advice]

1. Be wrong

2. Be controversial

3. Write things people feel qualified to have opinions on.

4. Invoke social reality.

- Writing That Provokes Comments

In the comments on YouTube, or the worst parts of Facebook or tumblr, demon threads are not surprising. People write comments that inflame ideological warfare all the time. Internets be internets. People be people. What can you do?

The surprising thing is how this works in places where everyone should really know better. The powers of demons are devious and subtle.

There's an experiment — insert obligatory replication crisis disclaimer — where one participant is told to gently poke another participant. The second participant is told to poke the first participant the same amount the first person poked them.

It turns out people tend to poke back slightly harder than they were first poked.

Repeat.

A few iterations later, they are striking each other really hard.

I think something like this is at work in the mechanics of demon threads.

The Demon Seed is the first comment in what will soon become a demon thread. It might look pretty innocuous. Maybe it feels slightly rude, or slightly oblivious, or pushing a conversation that should be about concrete empirical facts slightly towards being about social consensus (or, vice versa?).

It feels 1% outside the bound of a reasonable comment.

And then someone waters the demon seed. They don't want to let the point stand, so they respond with what seems like a fair rebuke.

Maybe they're self-aware that they're feeling annoyed, so they intentionally dial back the aggression of their response. "Ah, actually this probably comes across as too hostile, so I'll tweak my wording to reduce hostility by 4%." But, actually, the words were 6% more hostile than they thought, and now they've escalated 2%.

Repeat 2-3 times. The demon seed is watered. Latent underlying disagreements about how to think properly... or ideal social norms... or which coalitions should be highest status... or pure, simple you're insulting me and I'm angry…

They have festered and now they are ready to explode.

Then someone makes a comment that pushes things over the edge, and a demon thread is born.

(It is, of course, possible to skip steps 1-4 and just write a blatantly rude, incendiary comment. I'm trying to describe how this happens even when everyone is well intentioned and mostly trusts each other)

From there, if you're lucky it's contained to two people. But often, well meaning bystanders will wander by and think "Ah! People are being wrong on the internet! Wrong about things I am qualified to have opinions on! I can help!"

And it grows.

Then people start linking it from elsewhere, or FB algorithms start sharing it because people are commenting so the thread must be important.

It grows further.

And it consumes days of people's attention and emotional energy. More importantly, it often entrenches people's current opinions, and it burns people's good will that they might have been willing to spend on honest, cooperative discourse.

Why Demon Threads are Bad

I think demon threads are not just a bad plan - I think they are often net negative plan.

The reason is best expressed in Conor Moreton's Idea Innoculation and Inferential Distance. [Edit: the full article is no longer available]

Inferential distance is the gap between [your hypotheses and world model], and [my hypotheses and world model]. It's just how far out we have to reach to one another in order to understand each other.

If you share political and intellectual and cultural foundations, it's (relatively) easy. If you have completely different values and assumptions, (say you get dropped off in the 15th century and need to argue with Christopher Columbus) it may be nigh impossible.

It's right in the name—inferential distance. It's not about the "what" so much as it is about the "how"—how you infer new conclusions from a given set of information. When there's a large inferential distance between you and someone else, you don't just disagree on the object level, you also often disagree about what counts as evidence, what counts as logic, and what counts as self-evident truth.

What makes this really bad is idea inoculation.

When a person is exposed to a weak, badly-argued, or uncanny-valley version of an idea, they afterwards are inoculated against stronger, better versions of that idea. The analogy to vaccines is extremely apt—your brain is attempting to conserve energy and distill patterns of inference, and once it gets the shape of an idea and attaches the flag "bullshit" to it, it's ever after going to lean toward attaching that same flag to any idea with a similar shape.

When you combine idea inoculation with inferential distance, you get a recipe for disaster—if your first attempt to bridge the gap fails, your second attempt will also have to overcome the person's rapidly developing resistance.

You might think that each successive attempt will bring you closer to the day that you finally establish common ground and start communicating, but alas—often, each attempt is just increasing their resistance to the core concept, as they build up a library of all the times they saw something like this defeated, proven wrong, made to look silly and naive.

A demon thread is a recipe for bad attempts at communicating. Lots of people are yelling at once. Their defenses are raised. There's a sense that if you give in, you or your people look like losers or villains.

This'll make people worse at listening and communicating.

Why the Internet Worse

"Demon threads" can happen in person, but they're worse online.

One obvious reason is that the internet is more anonymous. This reduces consequences to the person writing a comment, and makes the target of the comment easier to round off to a bad stereotype or an abstract representation of The Enemy.

Other things people do:

A. People end up writing long winded monologues without anyone interrupting them to correct basic, wrong assumptions.

i.e. "you're just wrong because you think X, therefore... [complicated argument]", without providing opportunity for someone to respond "no I don't actually think X at all". And then, having written out [complicated argument] you're already invested in it, despite it being built on faulty premises.

B. Lots of people are writing. Especially as the demon thread grows. After 24 hours of its existence, the thread will have so much content it's a huge investment to actually read everything that's been said.

C. The comments aren't necessarily displayed in order. Or, if they are, people aren't reading them in order, they're reading whatever it's largest or most interesting.

D. The internet is full of lots of other content competing for attention.

This all means that:

E. People are skimming. This is most true when lots of people are writing lengthy monologues, but even when the thread first begins, people's eyes may be bouncing around to different tabs or different threads within a page so they aren't even reading what's being said, not with the intentionality and empathy they would when confronted with a real person in front of them.

And they might first be reading the most explosive, recent parts of a thread rather than piecing together the actual order of escalation, which may make people look less reasonable than they were.

This all adds up to giant threads being a uniquely bad way to resolve nuanced, emotionally fraught issues.

Containment?

Demon threads are like wildfires. Maybe you can put them out, with coordinated effort. You can also try to ignore them and hope they burn themselves out.

But if you wanted to actually stop it, the best bet is to do so is before they're erupted in the first place.

I've developed a sense of what seeds look like. I'll see a comment, think "god, this is going to become a demon thread in like two hours", and then sure enough, two hours later people are yelling at each other and everything is awful and everyone involved seems really sure that they are helping somehow.

Some flags that a demon thread might be about to happen:

Flags Regarding: Tension and Latent Hostility

When you look a comment and want to respond, you feel a visceral sense of "you're wrong", or "ugh, those people [from group that annoy me]" or "important principle must be defended!" or “I am literally under attack.”
You feel physiological defensiveness or anger - you notice the hairs on the back of your neck or arms standing on end, or a tightness in your chest, or however those emotions manifest in your body.
People in the thread seem to be talking past each other.
For whatever reason, tensions seem to be escalating.

Flags Regarding: Social Stakes

The argument seems like it's about who should be high or low status, which people or groups are virtuous and which are not, etc.
The argument is about social norms (in particular if at stake is whether some people will end up feeling unwelcome or uncomfortable in a given community/space that is important to them - this is extremely threatening)
More generally - the argument touches in some way on social reality, in ways that might have ramifications beyond the immediate conversation (or that people are afraid might have such ramifications).

If some of the above seem true (in particular, at least one of the first group and at least one of the second), then I think it's worth stepping back and being very careful about how you engage, even if no comment seems especially bad yet.

Potential Solutions

The first line of defense is to notice what’s happening - recognize if you’re feeling defensive or angry or talking past each other. Brienne's Noticing Sequence is pretty good for this (as well as her particular posts on training the skills of Empathy and handling Defensiveness - these may not work for everyone but I found the underlying thought process useful).

But while noticing is necessary, it’s not sufficient.

Rather than list my first guesses here, I’ll be discussing them in the comments and following this up with a “best-seeming of the potential solutions” post.

Meanwhile, some factors to consider as you decide what to do:

How are you involved?

Are you one of the people initially arguing, or a bystander?
How much do you normally trust the people involved?
Is it possible to take the conversation private?

Are we on the demon seed or demon thread stage? Is there common knowledge about either?
What are the actual stakes?
What are the moderation tools available to you?
Are you in a venue where you have the ability to shape conversational norms?

Do you directly control them (i.e. personal blog or feed?)
Does anyone have direct ownership of the venue? (either technically, or culturally)
Is there anything you can do unilaterally to make the conversation better, or will it require help from others?

Are you building a site where you get to develop entire new tools to deal with this class of problem?

With that in mind…

In whatever venues you most find yourself demon-thread-prone, what sort of plans can you actually think of that might actually help?

Note: I have since written a followup post with a working example of what I think people should usually do instead of demon threads.