Robin Hanson's Futarchy is a proposal to let prediction markets make governmental decisions. We can view an operating Futarchy as an agent, and ask if it is aligned with the interests of its constituents. I am aware of two main failures of alignment: (1) since predicting rare events is rewarded in proportion to their rareness, prediction markets heavily incentivise causing rare events to happen (I'll call this the entropy-market problem); (2) it seems prediction markets would not be able to assign probability to existential risk, since you can't collect on bets after everyone's dead (I'll call this the existential risk problem). I provide three formulations of (1) and solve two of them, and make some comments on (2). (Thanks to Scott for pointing out the second of these problems to me; I don't remember who originally told me about the first problem, but also thanks.)

---

Futarchy and Agents

My motivation here is twofold. On the one hand, Futarchy is something which many people are enthusiastic about. There is a current effort to implement it for small-to-medium-size group coordination. Futarchy is just the idea of using prediction markets to make decisions that matter, and decision markets are already a thing. So, from one perspective, we're already on our way there. But, there are alignment problems which have not been addressed. The misaligned Futarchy problem seems less bad that the misaligned superhuman AI problem. Arguably, we are already embedded in misaligned markets, and Futarchy seems like an improvement on those. Furthermore, Futarchy is made of people; if a Futarchy becomes too evil, the people can just stop, right?...

Well, one way or the other, it seems like it would be a good idea to go ahead and solve the Futarchy alignment problem before there's a problem.

My second reason for being interested in this is as a case-study for alignment. Maybe the Futarchy alignment problem is an easier instance, which could inform us about the AI alignment problem? Or perhaps it could be more directly relevant, informing logical-induction decision theories which look something like Futarchies?

More specifically, the entropy-market problem is an instance of "directionality" in decision theory, as discussed in The Foundations of Epistemic Decision Theory by Jason Konek and Ben Levinstein. The idea is that it's not always beneficial to enforce rationality constraints. A failure of a decision-theoretic constraint suggests some failure of rationality somewhere, but some ways of enforcing the constraint are "backward". For example, Aumann's Agreement Theorem does not imply that you should accept the Modesty Argument; doing so may harm your epistemology rather than improve it. It seems to me that getting directionality right would yield more convincing arguments for decision theory (relating to my puzzling at the end of this post), since it would explain when enforcing a constraint should be compelling. More importantly, perhaps, it may yield insight into the sort of directionality we need to extend causal decision theory to logical counterfactuals.

So, how does a prediction market get directionality wrong?

Entropy-Market Problem

In a typical prediction market, if you can reliably predict an event which the market assigns probability $p$, you stand to make $1/p$ on your initial investment. This means potentially huge returns for making rare events happen. This seems like a bad thing by default; it is easier to commit unpredictable acts of destruction than of creation. Effectively, a prediction market is also a market for purchasing chaos and havoc in the world; the one who can make the unthinkable happen has a fat paycheck waiting for them.

I offer three possible formalizations of the problem. All three rest on the idea of asymmetric incentives to cause unlikely events, but give different notions of "symmetric incentives" to aim for.

1. The differential value of being able to make an event slightly more likely is the same as that of making the event slightly less likely.
2. The value of being able to make an event happen is always the same as the value of preventing it.
3. The value of being able to add $x$ to the log-odds of the event is the same as the value of subtracting $x$.

In all three cases, I am ignoring the question of how difficult the event is to cause or prevent. Instead, I am trying to equalize the ROI of manipulating the "true" probability of an event. This does not eliminate the value of betting on events which you can personally manipulate -- the "directionality" problem is not totally solved. However, both of my solutions at least don't give arbitrarily high ROI to manipulators. (As a consequence, they also avoid giving arbitrarily high ROI to legitimate prediction of events the market doesn't expect, which is unfortunate. The ROI can still be decent, though; the market is still incentivised to achieve calibrated probabilities overall.)

The first interpretation is based on the intuition that what's more important is the incentive with respect to shifting probabilities a little bit. We can never cause or prevent an event with complete certainty. So, perhaps it's better to be sure that little shifts in the probability of an event aren't incentivised one way or the other.

The second interpretation seems like the most obvious. It is based on the intuition that what we want to avoid incentivising is someone stepping in and forcing an unlikely event to happen. Put simply, we want a would-be assassin to have the same incentive as a would-be bodyguard, rather than infinitely more.

The third interpretation is somewhere between the two. It could be justified by the intuition that it's probably about as difficult to double the odds of an event as cut them in half. Unlike the first or second interpretation, it explicitly takes into account that it's likely a lot harder to move probabilities which are already near 1 or 0. I haven't figured out whether this interpretation is achievable yet, but I liked it enough that I wanted to include it.

Solution 1: Equalized Differential ROI

How do we change the ROI of the market? The method I'll employ is to lock away some money with a transaction. For example, normally a share of an event which the market assigns probability 0.1 would cost 10¢ and pay out $1 on success or $0 on failure. We can add an artificial cost of 50¢; now, it costs 60¢, pays $1.50 on success, and pays back your 50¢ on failure. The ROI of these transactions goes down because the cost of initial investment goes up. Suppose you had assigned probability 0.2 to the event in our example. For the unmodified bet formula, the expected value we assign to the bet would be 20¢, at 10¢ cost, so ROI = 1. (ROI is net income / investment.) In the new scheme, the expected value is 70¢ -- but at a cost of 60¢. So, the new ROI is 1/6. I will call this extra money set aside a "reserve".

Now, suppose we add a reserve of $1-2p$ for $p<0.5$, and $0$ otherwise.

If the market assigns probability $p<0.5$ to an event, and we can do something to make that event happen with probability $p+x$ ($x$ positive), then we can pay $p-(1-2p)=1-p$ for a contract which we assign expected worth $p+x+(1-2p)=1-p+x$. The ROI is $x/(1-p)$.

On the other hand, if we can make the event $x$ less probable, we would take the other side of the bet: pay $1-p$ for a bet which pays off $1 if the event doesn't happen. The expected value for us is $1-p+x$, for an ROI of $x/(1-p)$.

Obviously, this gets me the differential property I wanted; whether increasing the probability of the event or decreasing it, the limit of ROI/x as x gets very small is $1/(1-p)$. This is, of course, because the function is the same in both directions: we've got the much stronger property that adding or subtracting x is always equally valuable. I didn't put this property on my list of problem formulations, though, since it doesn't strike me as intuitively appealing; adding and subtracting $x$ are not intuitively similarly difficult when $p$ is close to 1 or 0, except when $x$ is very small.

Another issue with this solution is that setting aside a reserve to modify the ROI is an artificial warping of the market, which can be easily corrected by external creditors. You will receive the reserve back with certainty; so, a lender might happily lend you up to the quantity you've got in reserve, for a small fee. This problem is somewhat corrected in the next proposed solution.

Solution 2: Equalized Event-Forcing ROI

We've seen that we can artificially decrease ROI (fragile as the method is when external lenders arrive on the scene). Can we artificially increase ROI? Yes, but there is a cost. We can artificially decrease the cost of an investment by offering leverage. For example, suppose we offer leverage of half the cost. Where I would normally pay $p for a contract paying out $1 on an event, now I pay $p/2; the contract pays out $1-p/2, and if the event I bet on doesn't happen, I still need to pay back the $p/2.

We can extend the solution from the previous problem by interpreting negative reserve as leverage. Again, we use the formula $1-2p$ for the reserve (leverage when negative). The cost of a contract is now $1-p$ in all cases, and the payout is $1+(1-2p)=2-2p$ if the event happens (and $1-2p$ if it doesn't). If we can make the event happen, the ROI is always 100%, no matter what $p$ the market assigns! Obviously, if we can prevent the event from happening, we just take the opposite side of the bet, which also gives an ROI of 100%.

Once again, I got more than I asked for: not only are the values of making an event happen or preventing it identical to each other, they're also the same for all values of $p$. This means there's no special incentive for manipulation as $p$ gets small (which might be concerning even if the manipulation were even-handed with respect to causing and preventing the event).

An added bonus of this scheme is that we can dump the risk of leveraging bets on $p>1/2$ on the people taking the $p<1/2$ side of the bet: the reserve of the other side of the bet is always the amount we need for leverage, so why not use that? It's not a very nice thing to do to the people betting on unlikely events, granted, but it solves the problem mentioned in the previous section: now the lenders will be more hesitant to loan money based on what's in reserve. This makes my warped market incentives more robust to external forces.

Half the people in the market will be unhappy comparing the situation to a normal betting market, but the other half will be happy about the free leverage and increased ROI. So, this market won't automatically be out-competed by alternate betting markets. Granted, any solution of the kind I'm exploring here works by blocking some contracts which people would be happier to make than the ones offered. Perhaps, for a betting market tied to a futarchy, people will be understanding of the need to avoid incentivising disorder.

Obviously, there are more formulas and desirable properties one could play around with. (The reserve formula $1/2-p$ is somewhat fun.) Since it is a futarchy, the system could even adapt to purposefully increase ROI of betting on events which we'd like to happen, and decrease the ROI of the reverse; I haven't explored this yet, but it seems even more promising.

Forecasting Doom

What of the second problem I mentioned at the beginning, the existential risk problem? It would be a sad irony if the aspiring rationalist community worked hard to put a futarchy in place only to find that the new government still systematically under-allocated resources to deal with existential risk.

The inability of a prediction market to forecast existential risk is a special case of an inaccuracy in forecasting events correlated with changes in the value of money. If money will predictably be worth half as much in a certain outcome, the market underestimates the odds of that outcome by that very ratio, just as if the bet contracts failed to pay out half the time. (Observation due to Scott.) Actually, though, that problem is not so hard to solve. We know exactly how this effect skews predictions about the value of money itself; so, we can adjust the market-estimated probabilities accordingly. Similarly, we can adjust for the skew on an arbitrary event $A$ by looking at conditional bets on the value of money given $A$. (If such conditional bets are not traded enough, we might assume no adjustment is needed, or perhaps do something more clever.) None of this changes how the market itself operates; it's all an "overlay" to let you see the true probability estimates implied by the market.

Unfortunately, this doesn't do much good for existential risk. Bets on the value of money can't be re-scaled appropriately when that value is zero.

However, I think this problem is also fixable. In fact, logical-inductor decision theories don't run into any problem like this. Granted, the agents there aren't using money for anything but the prediction market, so are easier to give purely epistemic concerns. However, I don't think that's the best thing to point to. A more important factor is that a logical induction market doesn't adjudicate its own destruction by waiting for it to happen. It adjudicates with proofs instead. It can see versions of itself die in many alternate worlds, and to the extent it knows which world it resides in, it may see itself die as well.

We can adjudicate doom before doom actually happens. We "simply" need a sufficiently trusted underlying adjudication system which is capable of estimating the death toll taken by nuclear missiles, plagues, asteroids, nanotech disasters, and AIs and all variety of conditional bets about the logic of such events.

I think many people think of prediction markets as the last word in knowledge aggregation, but the need for an underlying knowledge system which is slower but more trusted is nothing new. If you were to make bets on fundamental physics, you would not make a contract which hinges directly on a property of a particular particle. Instead, you would have to bet on the appearance of a peer-reviewed article in a sufficiently trusted venue with such-and-such a result, or something of that kind. A prediction market can only adjudicate bets on a direct fact in so far as the fact is accessible to it -- represented in data accessible by the computer. Beyond that, you need witnesses, data-entry, something to get the knowledge to the system in a trusted way. The advantage of the prediction market is that it can tell you what to expect before the trusted knowledge system does.

Adjudicating existential risk is hard. A Futarchy would likely need to leave the question of how to do it somewhat open, perhaps first installing a very minimal system and developing better options over time. Coming up with high-quality underlying knowledge systems to do the adjudication seems like the direction to go, though. Fortunately, in many cases, we can solve this well with an open market -- different bets can hinge on different knowledge systems, based on the whims of those making the contracts. When it comes to the critical bets for a Futarchy's decision-making, though, a top-down choice must be made as to how to adjudicate.