Updatelessness doesn't solve most problems
In some discussions (especially about acausal trade and multi-polar conflict), I’ve heard the motto “X will/won’t be a problem because superintelligences will just be Updateless”. Here I’ll explain (in layman’s terms) why, as far as we know, it’s not looking likely that a super satisfactory implementation of Updatelessness exists, nor that superintelligences automatically implement it, nor that this would drastically improve multi-agentic bargaining. Epistemic status: These insights seem like the most robust update from my work with Demski on Logical Updatelessness and discussions with CLR employees about Open-Minded Updatelessness. To my understanding, most researchers involved agree with them and the message of this post. What is Updatelessness? This is skippable if you’re already familiar with the concept. It’s easier to illustrate with the following example: Counterfactual Mugging. I will throw a fair coin. * If it lands Heads, you will be able to freely choose whether to pay me $100 (and if so, you will receive nothing in return). * If it lands Tails, I will check whether you paid me the $100 in the Heads world[1], and if so, I will pay you $1000. In this picture, I am Ω (throwing the coin) and you are A (deciding whether to Pay). When you find yourself in the Heads world, one might argue, the rational thing to do is to not pay. After all, you already know the coin landed Heads, so you will gain nothing by paying the $100 (assume this game is not iterated, etc.). But if, before knowing how the coin lands, someone offers you the opportunity of committing to paying up in the Heads world, you will want to accept it! Indeed, you’re still uncertain about whether you’ll end up in the Heads or the Tails world (50% chance on each). If you don’t commit, you know you won’t pay if you find yourself in the Heads world (and so also won’t receive $1000 in the Tails world), so your expected payoff is $0. But if you commit, your payoff will be -$100 in the Heads worl