Motte-and-bailey is a pretty famous term in rationalist circles. To quote the canonical SSC post on the topic:

So the motte-and-bailey doctrine is when you make a bold, controversial statement. Then when somebody challenges you, you claim you were just making an obvious, uncontroversial statement, so you are clearly right and they are silly for challenging you. Then when the argument is over you go back to making the bold, controversial statement.

I think that this is a real thing that very definitely happens.

I also can't remember having been in a single discussion that would have been improved by someone using this concept, and several that were worsened by it.

What's wrong with pointing out a motte and bailey if you see one?

Well, consider this conversation, whose content is fictional, but whose structure is similar to several very frustrating conversations that I've had:

Person: LW rationality isn't worth looking at. It's involves basing all your thinking on explicit calculations, which never works.

Me: Actually, that's not the case! People in the rationalsphere have explicitly said that that you shouldn't do that, and that emotion and intuition are actually important inputs to your decision-making process as well. For instance, here's a link to Julia Galef on straw vulcans, and here's an article saying that you shouldn't always use probabilities, and...

Person: Right, but that's just the motte. What they actually recommend in practice is just doing an explicit calculation about everything.

I am now faced with an impossible situation. Given that this person has decided that all the explicit declarations of "use emotion and intuition too" are just the motte, there's practically nothing that I could do to change their mind. Any links to articles saying that will be dismissed as just being parts of the motte. I could try to challenge their claim by asking for links that would support their case - but no doubt that the person can dig up some links which on an uncharitable reading at least sound like the thing that the person is saying. And possibly the person's actual crux is that they've actually run into some people who did misinterpret LW rationality in that way, and now this person has decided that those people were actually representative of all LW rationality.

Now, in this particular example, I might still be able to change the other person's mind: it's not like there are lots of self-described LW rationalists who are saying "down with intuition, yay formally calculating everything".

But suppose that we were discussing something of which there were both sensible and crazy interpretations - held by different people. So:

group A consistently makes and defends sensible claim A1
group B consistently makes and defends crazy claim B1

and maybe even:

group C consistently makes crazy claim B1, but when challenged on it, consistently retreats to defending A1

Now we are at the worst possible situation. Suppose that I belong to group A, and want to defend my group against an accusation. I say that no, we don't believe in crazy claim B1, we actually consistently maintain claim A1, and always have. I have links to back this up.

The other person says that this is just a motte and bailey, digging up links of group C using A1 as a motte - and is entirely correct.

Maybe I could say "you are referring to people belonging to group C, I belong to group A, see, members of my group are better than that." But it's unlikely that everyone involved explicitly declares their allegiance to different named groups, and that all members of the explicitly named group behave the same.

And even if they did, maybe that was part of the motte-and-bailey as well. There could be one group whose mission it was to say outrageous things, and another group whose mission was to say the sensible thing, in a way where the sensible thing still ended up supporting the outrageous version but established plausible deniability. This kind of thing actually happens: e.g. the youth organizations of Finnish political parties often make more radical claims than the parent parties would. To some extent this is likely deliberate and to some extent it's likely just younger people being more radical - and it being impossible to say the extent to which it's deliberate, is of course exactly what you want for the sake of having plausible deniability.

Scott's original post actually noted that motte-and-bailey accusations might derail a conversation, and had a recommendation for how to deal with it:

This means people who know both terms are at constant risk of arguments of the form “You’re weak-manning me!” “No, you’re motte-and-baileying me!“.

Suppose we’re debating feminism, and I defend it by saying it really is important that women are people, and you attack it by saying that it’s not true that all men are terrible. Then I can accuse you of making life easy for yourself by attacking the weakest statement anyone vaguely associated with feminism has ever pushed. And you can accuse me if making life too easy for myself by defending the most uncontroversially obvious statement I can get away with.

So what is the real feminism we should be debating? Why would you even ask that question? What is this, some kind of dumb high school debate club? Who the heck thinks it would be a good idea to say “Here’s a vague poorly-defined concept that mind-kills everyone who touches it – quick, should you associate it with positive affect or negative affect?!”

Taboo your words, then replace the symbol with the substance. If you have an actual thing you’re trying to debate, then it should be obvious when somebody’s changing the topic. If working out who’s using motte-and-bailey (or weak man) is remotely difficult, it means your discussion went wrong several steps earlier and you probably have no idea what you’re even arguing about.

This is often good advice. But as Scott himself has also discussed, sometimes you actually need to defend a group against an accusation:

I always thought that having things like political parties was stupid. Instead of identifying as a liberal and getting upset when someone insulted liberals or happy when someone praised liberals, I should say “These are my beliefs. There are other people who believe approximately the same thing, but the differences are sufficient that I just want to be judged on my own individual beliefs alone.”

The problem is, that doesn’t work. It’s not my decision whether or not I get to identify with other liberals or not. If other people think of me as a liberal, then anything other liberals do is going to reflect, positively or negatively, on me. And I’m going to have to join in the fight to keep liberals from being completely discredited, or else the fact that I didn’t share any of the opinions they were discredited for isn’t going to save me. I will be Worst Argument In The World-ed and swiftly dispatched.

In terms of the example which started this article, many of us probably think that LW rationality is important and valuable and want to encourage other people to take a look at it. So if someone shows up to say "LW rationality isn't worth looking at, they might say the occasional reasonable-sounding thing but that's just the motte, the bailey is something quite stupid", well, I'll probably want to say something in response.

I don't know what the right approach here is, given that sometimes you do need to talk about the opinions of groups, and sometimes groups do actually engage in motte-and-bailey. I don't want to go as far as to say, never accuse people of it. But I do suggest that you should be very cautious about using that accusation, because you may be placing yourself in a situation where you will never change your mind.