Summary

In a list of guidelines for rational discourse, Duncan Sabien proposes that one should "[a]im for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth."
However, prediction markets illustrate fundamental reasons why rational discourse doesn't particularly look like "aiming for convergence." When market prices converge on the truth, it's because traders can only make money by looking for divergences where their beliefs are more accurate than the market's. Similarly, when discussions converge on the truth, it's because interlocutors can only advance the discussion by making points where the discussion-so-far has been wrong or incomplete. Convergence on the truth, if it happens, happens as a side-effect of correctly ironing out all existing mispricings/disagreements; it seems wrong to describe this as "aiming for convergence" (even if convergence would be the end result if everyone were reasoning perfectly).
Sabien's detailed discussion of the "aim for convergence on truth" guideline concerns itself with how to determine whether an interlocutor is "present in good faith and genuinely trying to cooperate." I don't think I understand how these terms are being used in this context. More generally, the value of "collaborative truth-seeking" is unclear to me: if I can evaluate arguments on their merits, the question of whether the speaker is "collaborative" with me does not seem intellectually substantive.

Mostly, I don't expect to disagree with heavily-traded prediction markets. If the market says it's going to rain on Saturday with 85% probability, then I (lacking any special meteorology knowledge) basically think it's going to rain on Saturday at 85% probability.

Why is this? Why do I defer to the market, instead of tarot cards, or divination sticks, or my friend Maddie the meteorology enthusiast?

Well, I don't expect the tarot cards to tell me anything about whether it will rain on Saturday, because there's no plausible physical mechanism by which information about the weather could influence the cards. Shuffling and dealing the cards should work the same in worlds where it will rain and worlds where it won't rain. Even if there is some influence (because whether it will rain affects the moisture and atmospheric pressure in the air, which affects my grip on the cards, which affects my shuffling motion?), it's not something I can detect from which cards are drawn.

I do expect my friend Maddie the meteorology enthusiast to tell me something about whether it will rain on Saturday. That's because she's always looking at the latest satellite cloud data and tinkering with her computer models, which is a mechanism by which information about the weather can influence her forecasts. The cloud data will be different in worlds where it will rain and worlds where it won't rain. If Maddie is pretty sharp and knows her stuff, maybe she can tell the difference.

And yet—no offense, Maddie—I expect the market to do even better. It's not just that the market has a lot of other pretty sharp people looking at the cloud data, and that maybe some of them are even sharper than Maddie, even though Maddie is my friend and my friends are the best.

It's that the market mechanism rewards people for being less wrong than the market. If the rain-on-Saturday market is trading at 85%, and Maddie's rival Kimber buys 100 shares of No, that doesn't mean Kimber thinks it's not going to rain. It means Kimber thinks 85% is too high. If Kimber thinks it's "actually" only going to rain with 80% probability, then she figures that a No share that pays out $1 if it doesn't rain should be worth 20¢. If it's currently trading for 15¢, it's worth buying for the "expected" profit of 5¢ per share—effectively, buying a dollar for 15¢ in the 20% of worlds where it doesn't rain—even though it's still probably going to rain. If she were risk-neutral and had enough money, Kimber would have an incentive to keep buying No shares from anyone willing to sell them for less than 20¢, until there were no such sellers left—at which point, the rain-on-Saturday market would be trading at 80%.

Conversely, if I can't tell whether 85% is too low or too high, then I can't expect to make money by buying Yes or No shares. There's no point in buying a dollar for 85¢ in 85% of worlds, or for 15¢ in 15% of worlds.

That's why I defer to the market. It's not that I'm aiming to converge my beliefs with those of market participants. It's not that market participants are trying to converge with each other, "cooperating" in some "collaborative truth-seeking" project. The market converges on truth (if it does) because market participants are trying to make money off each other, and it's not so easy to make money off of an aggregation of sharp people who are already trying to do the same. I would prefer to correctly diverge from the market—to get something right that the market is getting wrong, and make lots of money in the future when my predictions come true. But mostly, I don't know how.

Unfortunately, not everything can be the subject of a prediction market. Prediction markets work on future publicly observable measurements. We bet today on whether it will rain on Saturday (which no one can be sure about), expecting to resolve the bets on Saturday (when anyone can just look outside).

Most disputes of intellectual interest aren't like this. We do want to know whether Britain's coal reserves were a major cause of the Industrial Revolution, or whether Greg Egan's later work has discarded the human factor for mathematical austerity, but we can't bet without some operationalization for how to settle the bet, which is lacking in cases like these that require an element of "subjective" judgement.

Nevertheless, many of the principles regarding prediction markets and when to bet in them, approximately generalize to the older social technology of debates and when to enter them.

Mostly, I don't expect to enter heavily-argued debates. If prevailing opinion on the economic history subreddit says that Britain's coal reserves were a major cause of the Industrial Revolution, then I (lacking any special economic history knowledge) basically think that Britain's coal reserves were a major cause of the Industrial Revolution.

If Kimber's sister Gertrude leaves a comment pointing to data that cities closer to coalfields started growing faster in 1750, it's not because that comment constitutes the whole of Gertrude's beliefs about the causes of the Industrial Revolution. It means that Gertrude thinks that the city-growth/coal-proximity correlation is an important consideration that the discussion hadn't already taken into account; she figures that she can win status and esteem from her fellow economic–history buffs by mentioning it.

Conversely, if I don't know anything about economic history, then I can't expect to win status or esteem by writing "pro-coal" or "anti-coal" comments: there's no point in saying something that's already been said upthread, or that anyone can tell I just looked up on Wikipedia.

That's why I defer to the forum: because (hopefully) the forum socially rewards people for being less wrong than the existing discussion. The debate converges on truth (if it does) because debaters are trying to prove each other wrong, and it's not so easy to prove wrong an aggregation of sharp people who are already trying to do the same.

In a reference post on "Basics of Rationalist Discourse", Duncan Sabien proposes eleven guidelines for good discussions, of which the (zero-indexed) fifth is, "Aim for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth."

This advice seems ... odd. What's this "convergence" thing about, that differentiates this guideline from "aim for truth"?

Imagine giving the analogous advice to a prediction market user: "Aim for convergence on the correct probability, and behave as if your fellow traders are also aiming for convergence on the correct probability."

In some sense, this is kind of unobjectionable: you do want to make trades that bring the market price closer to your subjective probability, and in the process, you should take into account that other traders are also already doing this.

But interpreted another way, the advice is backwards: traders make money by finding divergences where their own beliefs are more accurate than the market's. Every trade is an expression of the belief that your counterparty is not aiming to converge on the correct probability—that there's a sucker at every table, and that this time it isn't you.

(This is with respect to the sense of "aiming" in which an archer "aiming" an arrow at a target might not hit it every time, but we say that their "aim" is good insofar as they systematically tend to hit the target, that any misses are best modeled by a random error term that can't be predicted. Similarly, the market might not always be right, but if you can predict when the market is wrong, the traders must not have been "aiming" correctly from your perspective.)

So why is the advice "behave as if your interlocutors are also aiming for convergence on truth", rather than "seek out conversations where you don't think your interlocutors are aiming to converge on truth, because those are exactly the conversations where you have something substantive to say instead of already having converged"?

(For example, the reason I'm writing the present blog post contesting Sabien's Fifth Guideline of "Aim for convergence on truth [...]" and not the First Guideline of "Don't say straightforwardly false things", is because I think the Fifth Guideline is importantly wrong, and the First Guideline seems fine.)

Sabien's guidelines are explicitly disclaimed to be shorthand that it sometimes makes sense to violate; the post helpfully includes another 900 words elaborating on how the Fifth Guideline should be interpreted. Unfortunately, the additional exposition does not seem to clarify matters. Sabien writes:

If you are moving closer to truth—if you are seeking available information and updating on it to the best of your ability—then you will inevitably eventually move closer and closer to agreement with all the other agents who are also seeking truth.

But this can't be right. To see why, substitute "making money on prediction markets" for "moving closer to truth", "betting" for "updating", and "trying to make money on prediction markets" for "seeking truth":

If you are making money on prediction markets—if you are seeking available information and betting on it to the best of your ability—then you will inevitably eventually move closer and closer to agreement with all the other agents who are also trying to make money on prediction markets.

But the only way to make money on prediction markets is by correcting mispricings, which necessarily entails moving away from agreement from the consensus market price. (As it is written, not every change is an improvement, but every improvement is necessarily a change.)

To be sure, most traders shouldn't bet in most markets; you should only bet when you think you see a mispricing. In the same way, most people shouldn't speak in most discussions; you should only speak up when you have something substantive to say. All else being equal, the more heavily-traded the market or the more well-trodden the discussion, the more worried you should be that the mispricing or opportunity to make a point that you thought you saw, was illusory. In any trade, one party has to be on the losing side; in any disagreement, at least one party has to be in the wrong; be wary if not afraid that it might be you!

But given that you're already in the (unusual!) situation of making a trade or prosecuting a disagreement, "aim for convergence on truth" doesn't seem like particularly useful advice, because the "for convergence" part isn't doing any work. And "behave as if your interlocutors [or counterparties] are also aiming for convergence on truth" borders on the contradictory: if you really believed that, you wouldn't be here!

(That is, disagreement is disrespect; the very fact that you're disagreeing with someone implies that you think there's something wrong with their epistemic process, and that they think there's something wrong with your epistemic process. Perhaps each of you could still consider the other to be "aiming for convergence on truth" if the problem is construed as a "capabilities failure" rather than an "alignment failure": that you each think the other is "trying" to get the right answer (whatever "trying" means), but just doesn't know how. Nevertheless, "don't worry; I'm not calling you dishonest, I'm just calling you stupid" doesn't hit the note of symmetrical mutual respect that the Fifth Guideline seems to be going for.)

Prediction markets, and betting more generally, are hallmarks of "rationalist" culture, something "we" (the target audience of a blog post on "rationalist discourse") generally encourage, rather than discourage. Why is this, if idealized Bayesian reasoners would never bet against each other, because idealized Bayesian reasoners would never disagree with each other? Why don't we condemn offers to bet as violations of a guideline to "behave as if your interlocutors are also aiming for convergence on truth"?

It's out of an appreciation that the process of bounded agents becoming less wrong, doesn't particularly look like the final outcome if everyone were minimally wrong. The act of sticking your neck (or your wallet) out at a particular probability disciplines the mind. Bayesian superintelligences need no discipline and would never have occasion to bet against each other, but you can't become a Bayesian superintelligence by imitating this surface behavior; clarifying real disagreements is more valuable than steering towards fake agreement. Every bet and every disagreement is the result of someone's failure. But the only way out is through.

Sabien's exposition on the Fifth Guideline expresses concern about how to distinguish "genuine bad faith" from "good faith and genuinely trying to cooperate", about the prevalence of "defection strategies" getting in the way of "treat[ing] someone as a collaborative truth-seeker".

My reply to this is that I don't know what any of those words mean. Or rather, I know how these words in my vocabulary map onto concepts in my ontology, but those meanings don't seem consistent with the way Sabien seems to be using the words.

In my vocabulary, I understand the word "cooperate" used in the proximity of the word "defect" or "defection" to indicate a Prisoner's Dilemma-like situation, where each party would be better off Defecting if their counterparty's behavior were held constant, but both parties prefer the Cooperate–Cooperate outcome over the Defect–Defect outcome (and also prefer Cooperate–Cooperate over taking turns alternating between Cooperate–Defect and Defect–Cooperate). Sabien's references to "running a tit-for-tat algorithm", "appear[ing] like the first one who broke cooperation", and "would-be cooperators hav[ing] been trained and traumatized into hair-trigger defection" would seem to suggest he has something like this in mind?

But, normatively, rationalist discourse shouldn't be a Prisoner's Dilemma-like situation at all. If I'm trying to get things right (every step of my reasoning cutting through to the correct answer in the same movement), I can just try to get things right unilaterally. I prefer to talk to people who I judge as also trying to get things right, if any are available—they probably have more to teach me, and are better at learning from me, than people who are motivatedly getting things wrong.

But the idiom of "cooperation" as contrasted to "defection", in which one would talk about the "first one who broke cooperation", in which one cooperates in order to induce others to cooperate, doesn't apply. If my interlocutor is motivatedly getting things wrong, I'm not going to start getting things wrong in order to punish them.

(In contrast, if my roommate refused to do the dishes when it was their turn, I might very well refuse when it's my turn in order to punish them, because "fair division of chores" actually does have the Prisoner's Dilemma-like structure, because having to do the dishes is in itself a cost rather than a benefit; I want clean dishes, but I don't want to do the dishes in the way that I want to cut through to the correct answer in the same movement.)

A Prisoner's Dilemma framing would make sense if we modeled discourse as social exchange: I accept a belief from you, if you accept a belief from me; I'll use cognitive algorithms that produce a map that reflects the territory, but only if you do, too. But that would be crazy. If people are natively disposed to think of discourse as a Prisoner's Dilemma in this way, we should be trying to disabuse them of the whole ontology, not induce them to "cooperate"!

Relatedly, the way Sabien speaks of "good faith and genuinely trying to cooperate" in the same breath—almost as if they were synonymous?—makes me think I don't understand what he means by "good faith" or "bad faith". In my vocabulary, I understand "bad faith" to mean putting on the appearance of being moved by one set of motives, while actually acting from another.

But on this understanding, good faith doesn't have anything to do with cooperativeness. One can be cooperative in good faith (like a true friend), adversarial in good faith (like an honorable foe), cooperative in bad faith (like a fair-weather friend who's only being nice to you now in order to get something out of you), or adversarial in bad faith (like a troll just saying whatever will get a rise out of you).

(In accordance with Sabien's Seventh Guideline ("Be careful with extrapolation, interpretation, and summary/restatement"), I should perhaps emphasize at this point that this discussion is extrapolating a fair amount from the text that was written; perhaps Sabien means something different by terms like "defection" or "bad faith" or "collaborative", than what I take them to mean, such that these objections don't apply. That's why my reply is, "I don't know what any of those words mean", rather than, "The exposition of the Fifth Guideline is wrong.")

Sabien gives this example of a request one might make of someone whose comments are insufficiently adhering to the Fifth Guideline:

"Hey, sorry for the weirdly blunt request, but: I get the sense that you're not treating me as a cooperative partner in this conversation. Is, uh. Is that true?"

Suppose someone were to reply:

"You don't need to apologize for being blunt! Let me be equally blunt. The sense you're getting is accurate: no, I am not treating you as a cooperative partner in this conversation. I think your arguments are bad, and I feel very motivated to explain the obvious counterarguments to you in public, partially for the education of third parties, and partially to raise my status at the expense of yours."

I consider this a good faith reply. It's certainly not a polite thing to say. But politeness is bad faith. (That's why someone might say in response to a compliment, "Do you really mean it, or are you just being polite?") Given that someone actually in fact thinks my arguments are bad, and actually in fact feels motivated to explain why to me in public in order to raise their status at expense of mine, I think it's fine for them to tell me so. How would me expecting them to lie about their motives help anyone? Complying with such an expectation really would be in bad faith!

I suppose such a person would not be engaging in the "collaborative truth-seeking" that the "Basics of Rationalist Discourse" guideline list keeps talking about. But it's not clear to me why I should care about that, when I can can just ... listen to the counterarguments and judge them on their merits, without getting distracted by the irrelevancy of whether the person seems "collaborative" with me?

In slogan form, you could perhaps say that I don't believe in collaborative truth-seeking; I believe in competitive truth-seeking. But I don't like that slogan, because in my ontology, they're not actually different things. "Attacking your argument because it sucks" sounds mean, and "Suggesting improvements to your argument to make it even better" sounds nice, but the nice/mean dimension is not intellectually substantive. The math is the same either way.