[todo: turn the rest of this into a finished doc, or delete it.

To the extent that "standing in front of a whiteboard trying to think of pseudocode" does work better to clear up conceptual confusion, it will be because this stage of the problem could iterate through more possible pieces of pseudocode, in a context that exposed the underlying problem as simply as possible without any added complications from demanding bounded solutions or realism.

Once you realize you currently can't write the code; what's the next step? Depending on how easy it is to build small programs that naturally represent the problem setup - and in the case of chess, this shouldn't be hard, the hard part is generating good moves, not representing the game of chess in running code - then it might make sense to take exploratory stabs at trying to compute something that makes the game play good chess.

What if you can't even represent the problem setup naturally in running code? Then

What about after you realize that you currently can't write code?

Doing unbounded analysis implies that you think your current state of understanding has basic confusions to be cleared up. If at this point somebody tried to write a paper about an algorithm that would, given hypercomputation, pilot a robotic car on an ideal 2-dimensional surface, it would be a legitimate reply to say "But we have actual running code for robotic cars, nor are we confused about how to do this; why do an unbounded analysis when we already have bounded ones?" In terms of the rocket alignment metaphor, spending a lot of time trying to figure out how to fire a cannonball so that it circles a perfectly spherical Earth and returns to its exact starting point, corresponds to a suspicion that you currently lack the mathematical language to even talk about landing a rocket on the Moon. Someone who expects that current vehicles only require one or two tweaks to get to the Moon, or that they can be piloted there by keeping the Moon in the viewport and steering intuitively, is more likely to object "Why are you talking about cannonballs, instead of the vehicles that reach the highest altitudes today?" The closer you think current algorithms are to supporting a Friendly AI, the less sympathetic you might be to the suggestion that very basic foundational work needs to be done and might need to be done in a simplified setting.

Or imagine, say, somebody claiming that an ideal chess program ought to evaluate the ideal goodness of each move, and giving their philosophical analysis in terms of a chess agent which knows the perfect goodness...

Faced with an "unbounded solution" you don't like, the next step is to say crisply exactly what is wrong with it in the form of a new desideratum for your solution. In this case, our reply would be that for Agent 1 to exactly simulate Agent 2, Agent 1 must be larger than Agent 2, and since we want to model stable self-modification, we can't introduce a requirement that Agent 2 be strictly weaker than Agent 1. More generally, we apply the insight of Vinge_principleVinge's Principle to this observation and arrive at the desiderata of Vinge_uncertainty and Vinge_reflection, which we also demand that an unbounded solution exhibit.

Faced with an "unbounded solution" you don't like, the next step is to say crisply exactly what is wrong with it in the form of a new desideratum for your solution. In this case, our reply would be that for Agent 1 to exactly simulate Agent 2, Agent 1 must be larger than Agent 2, and since we want to model stable self-modification, we can't introduce a requirement that Agent 2 be strictly weaker than Agent 1. More generally, we apply the insight of Vinge's Principle to this observation and arrive at the desiderata of Vinge_uncertaintyVingean uncertainty and Vinge_reflectionVingean reflection, which we also demand that an unbounded solution exhibit.

Summary

"Unbounded analysis" refers to determining the behavior of a computer program that would, to actually run, require an unphysically large amount of computing power, or sometimes hypercomputation. If we know how to solve a problem using unlimited computing power, but not with real-world computers, then we have an "unbounded solution" but no "bounded solution".

As a central example, consider computer chess. The first paper ever written on computer chess, by Claude Shannon in 1950, gave an unbounded solution for playing perfect chess by exhaustively searching the tree of all legal chess moves. (Since a chess game draws if no pawn is moved and no piece is captured for 50 moves, this is a finite search tree.) Shannon then passed to considering more bounded ways of playing imperfect chess, such as evaluating the worth of a midgame chess position by counting the balance of pieces, and searching a smaller tree up to midgame states. It wasn't until 47 years later, in 1997, that Deep Blue beat Garry Kasparov for the world championship, and there were multiple new basic insights along the way, such as alpha-beta pruning.

In 1836, there was a sensation called the Mechanical Turk, allegedly a chess-playing automaton. Edgar Allen Poe, who was also an amateur magician, wrote an essay arguing that the Turk must contain a human operator hidden in the apparatus (which it did). Besides analyzing the Turk's outward appearance to locate the hidden compartment, Poe carefully argued as to why no arrangement of wheels and gears could ever play chess in the first place, explicitly comparing the Turk to "the calculating machine of Mr. Babbage":

Arithmetical or algebraical calculations are, from their very nature, fixed and determinate. Certain data being given, certain results necessarily and inevitably follow [...] But the case is widely different with the Chess-Player. With him there is no determinate progression. No one move in chess necessarily follows upon any one other. From no particular disposition of the men at one period of a game can we predicate their disposition at a different period [...] Now even granting that the movements of the Automaton Chess-Player were in themselves determinate, they would be necessarily interrupted and disarranged by the indeterminate will of his antagonist. There is then no analogy whatever between the operations of the Chess-Player, and those of the calculating machine of Mr. Babbage [...] It is quite certain that the operations of the Automaton are regulated by mind, and by nothing else. Indeed this matter is susceptible of a mathematical demonstration, a priori.

(In other words: In an algebraical problem, each step follows with the previous step of necessity, and therefore can be represented by...

1. If we don't know how to solve a problem even given unbounded computation, that means we're confused about the nature of the work to be performed. It is often worthwhile to tackle this basic conceptual confusion in a setup that exposes the confusing part of the problem as simply as possible, and doesn't introduce further and distracting complications until the core issue is resolved.
2. Introducing 'realistic' complications can make it difficult to engage in cooperative discourse about which ideas have which consequences - AIXI was a watershed moment because it was formally specified and there was no way to say "Oh, I didn't mean that" when somebody pointed out that AIXI would try to seize control of its own reward channel.
3. Increasing the intelligence of an advanced agent may sometimes move its behavior closer to ideals and further from specific complications of an algorithm. Early, stupid chess algorithms had what seemed to humans like weird idiosyncracies tied to their specific algorithms. Modern chess programs, far beyond the human champions, can from an intuitive human perspective be seen as just making good chess moves.
4. Current AI algorithms are often incapable of demonstrating future phenomena that seem like they should predictably occur later, and whose interesting properties seem like they can be described using an unbounded algorithm as an example. E.g. convergent instrumental strategies arise from consequentialist agents reasoning about difficult domains like "My programmers have beliefs about whether I'll achieve their goals" and "What happens if they decide to press my shutdown button?", which is very far from something that naturally arises in current AI algorithms; but we can still talk about this using abstract notions like economic agency and Utility indifference. Similarly, if you're not allowed to do brute-force proof searches, then modern AI algorithms are very far from arriving at abstract beliefs about alternate versions of tehirtheir own code.

1. If we don't know how to solve a problem even given unbounded computation, that means we're confused about the nature of the work to be performed. It is often worthwhile to tackle this basic conceptual confusion in a setup that exposes the confusing part of the problem as simply as possible, and doesn't introduce further and distracting complications until the core issue is resolved.
2. Introducing 'realistic' complications can make it difficult to engage in cooperative discourse about which ideas have which consequences - AIXI was a watershed moment because it was formally specified and there was no way to say "Oh, I didn't mean that" when somebody pointed out that AIXI would try to seize control of its own reward channel.
3. Increasing the intelligence of an advanced agent may sometimes move its behavior closer to ideals and further from specific complications of an algorithm. Early, stupid chess algorithms had what seemed to humans like weird idiosyncracies tied to their specific algorithms. Modern chess programs, far beyond the human champions, can from an intuitive human perspective be seen as just making good chess moves.
4. Current AI algorithms are often incapable of demonstrating future phenomena that seem like they should predictably occur later, and whose interesting properties seem like they can be described using an unbounded algorithm as an example. E.g. convergent instrumental strategies arise from consequentialist agents reasoning about difficult domains like "My programmers"programmers have beliefs about whether I'll achieve their goals" and "What happens if they decide to press my shutdown button gets pressed"button?", which is very far from something that naturally arises in current AI algorithms; but we can still talk about this using abstract notions like economic agency and Utility indifference. Similarly, if you're not allowed to do brute-force proof searches, then modern AI algorithms are very far from arriving at abstract beliefs about alternate versions of tehir own code.

Unbounded analysis in computer science

All of these properties simplify the scenario, at the expense of realism.

As background for the state of mind in modern computer science, it should be remembered that individual computer scientists may sometimes be tempted to exaggerate how much their algorithm solves or how realistic it is - while not everyone gives into that temptation, at least some computer scientists do so at least some of the time. This means that there are a number of 'unbounded' analyses floating around which don't solve the real-world problem, may not even shed very much light on the real-world problem, and whose authors praise them as nearly complete solutions but for some drudge-work of mere computation. A reasonable reaction is to be suspicious of unbounded analyses, and scrutinize them closely to make sure that they...

In modern AI and especially in value alignment theory, there's a sharp divide between "problems we know how to solve using unlimited computing power", and "problems we can't state how to solve using computers larger than the universe". Not knowing how to do something with unlimited computing power reveals that you are confused about the structure of the problem.

Introduction: Poe, Shannon, and Deep Blue

In 1950, the very first paper ever written on computer chess, by Claude Shannon, gave the algorithm that would play perfect chess given unlimited computing power. In reality, computing power was limited, so computers did not play superhuman chess until 47 years after that.

So knowing Knowing how to do something in principle doesn't always mean you can do it in practice without additional hard work and insights. Nonetheless it remains true that

Nonetheless, if you don't know how to play chess using unlimited computing power, you definitely can't play chess using limited computing power. In 1830, Edgar Allen Poe, who was also an amateur magician, carefully argued that no automaton could ever play chess, since at each turn there are many possible moves, but machines can only make deterministic motions. Between Poe in 1830 and Shannon in 1950 there was a key increase in the understanding of computer chess, represented by the intermediate work of Turing, Church, and others.others - working with blackboards and mathematics rather than existing tabulating and calculating machines, since general computing machines had not yet been built.

If you ask any present researcher how to write a Python program that would be a nice AI if we could run it on a computer much larger than the universe, and they have the ability to notice when they are confused, they should notice the "stubbing the mind's toe" feeling of trying to write a program when we're confused about the nature of the computations we need to perform.

This doesn't necessarily prove that we can best proceed by grabbing our whiteboards and trying to figure out how to crisply model some aspects of the problem given unlimited computing power - try to directly tackle and reduce our own confusion. And if you do, there are certainly pitfalls to avoid. There's nonetheless some strong considerations pushing MIRI in the direction of trying to do unbounded analyses, which can be briefly summarized as follows:

1. If we don't know how to solve a problem even given unbounded computation, that means we're confused about the nature of the work to be performed. It is often worthwhile to tackle this basic conceptual confusion in a setup that exposes the confusing part of the problem as simply as possible, and doesn't introduce further and distracting complications until the core issue is resolved.
2. Introducing 'realistic' complications can make it difficult to engage in cooperative discourse about which ideas have which consequences - AIXI was a watershed moment because it was formally specified and there was no way to say "Oh, I didn't mean that" when somebody pointed out that AIXI would try to seize control of

However,So knowing how to do something in principle doesn't always mean you can do it in practice without additional hard work and insights. Nonetheless it remains true that if you don't know how to play chess using unlimited computing power, you definitely can't play chess using limited computing power. In 1830, Edgar Allen Poe, who was also an amateur magician, carefully argued that no automaton could ever play chess, since at each turn there are many possible moves, but machines can only make deterministic motions. Between Poe in 1830 and Shannon in 1950 there was a genuinekey increase in the understanding of computer chess, represented by the intermediate work of Turing, Church, and others.

Besides the role of mathematization in producing conceptual understanding, it also helps build cooperative discourse. Marcus Hutter's AIXI marks not only the first time that somebody described a short Python program that would be a strong AI if run on a hypercomputer, but also the first time that anyone was able to point to a formal specification and say "And that is why this AI design would wipe out the human species and turn all matter within its reach into configurations of insignificant value" and the reply wasn't just "Oh, well, of course that's not what I meant" because the specification of AIXI was fully nailed down. While not every key issue in value alignment theory is formalizable, there's a strong drive toward formalism not just for conceptual clarity but also to sustain a cooperative process of building up collective debate and understanding of ideas.