This seems a good opportunity for me to summarize my disagreements with both Paul and MIRI. In short, there are two axes along which Paul and MIRI disagree with each other, where I'm more pessimistic than either of them.
(One of Paul's latest replies to me on his AI control blog says "I have become more pessimistic after thinking it through somewhat more carefully." and "If that doesn’t look good (and it probably won’t) I will have to step back and think about the situation more broadly." I'm currently not sure how broadly Paul was going to rethink the situation or what conclusions he has since reached. What follows is meant to reflect my understanding of his positions up to those statements.)
One axis might be called "metaphilosophical paternalism" (a phrase I just invented, not sure if there's an existing one I should use), i.e., how much external support / error correction do humans need, or more generally how benign of an environment do we have to place them in, before we can expect them to eventually figure out their "actual" values (which implies correctly solving all relevant philosophical dependencies such as population ethics and philosophy of consciousness) and how hard will it be to design and provide such support / error correction.
MIRI's position seems to be that humans do need a lot of external support / error correction (see CEV) and this is a hard problem, but not so hard that it will likely turn out to be a blocking issue. Paul's position went from his 2012 version of "indirect normativity" which envisioned placing a human in a relatively benign simulated environment (although still very different from the kinds of environments where we have historical evidence of humans being able to make philosophical progress in) to his current ideas where humans live in very hostile environments, having to process potentially adversarial messages from superintelligent AIs under time pressure.
My own thinking is that we currently know very little about metaphilosophy, essentially nothing beyond that philosophy is some kind of computational / cognitive process implemented by (at least some) human brains, and there seems to be such a thing as philosophical truth or philosophical progress, but that is hard to define or even recognize. Without easy ways to check one'e ideas (e.g., using controlled experiments or mathematical proofs), human cognitive processes tend to diverge rather than converge. (See political and religious beliefs, for example.) If typical of philosophical problems in general, understanding metaphilosophy well enough to implement something like CEV will likely take many decades of work even after someone discovers a viable approach (which we don't yet have). Think of how confused we still are about how expected utility maximization applies in bargaining, or what priors really are or should be, many decades after those ideas were first proposed. I don't understand Paul and MIRI's reasons for being as optimistic as they each are on this issue.
The other axis of disagreement is how feasible it would be to create aligned AI that matches or beats unaligned AI in efficiency/capability. Here Paul is only trying to match unaligned AIs using the same mainstream AI techniques, whereas MIRI is trying to beat unaligned AIs in order to prevent them from undergoing intelligence explosion. But even Paul is more optimistic than I think is warranted. (To be fair, at least some within MIRI, such as Nate, may be aiming to beat unaligned AIs not because they're particularly optimistic about the prospects of doing so, but because they're pessimistic about what would happen if we merely match them.) It seems unlikely to me that alignment to complex human values comes for free. If nothing else, aligned AIs will be more complex than unaligned AIs and such complexity is costly in design, coding, maintenance, and security. Think of the security implications of having a human controller or a complex value extrapolation process at an AI's core, compared to something simpler like a paperclip maximizer, or the continuous challenges of creating improved revisions of AI design while minimizing the risk of losing alignment to a set of complex and unknown values.
Jessica's post lists searching for fundamental obstructions to aligned AI as a motivation for Paul's research direction. I think given that efficient aligned AIs almost certainly exist as points in mindspace, it's unlikely that we can find "fundamental" reasons why we can't build them. Instead they will likely just take much more resources (including time) to build than unaligned AIs, for a host of "messy" reasons. Maybe the research can show that certain approaches to building competitive aligned AIs won't succeed, but realistically such a result can only hope to cover a tiny part of AI design space, so I don't see why that kind of result would be particularly valuable.
Please note that what I wrote here isn't meant to be an argument against doing the kind of research that Paul and MIRI are doing. It's more of an argument for simultaneously trying to find and pursue other approaches to solving the "AI risk" problem, especially ones that don't require the same preconditions in order to succeed. Otherwise, since those preconditions don't seem very likely to actually obtain, we're leaving huge amounts of potential expected value on the table if we bank on just one or even both of these approaches.
Paul Christiano and "MIRI" have disagreed on an important research question for a long time: should we focus research on aligning "messy" AGI (e.g. one found through gradient descent or brute force search) with human values, or on developing "principled" AGI (based on theories similar to Bayesian probability theory)? I'm going to present my current model of this disagreement and additional thoughts about it.
I put "MIRI" in quotes because MIRI is an organization composed of people who have differing views. I'm going to use the term "MIRI view" to refer to some combination of the views of Eliezer, Benya, and Nate. I think these three researchers have quite similar views, such that it is appropriate in some contexts to attribute a view to all of them collectively; and that these researchers' views constitute what most people think of as the "MIRI view".
(KANSI AI complicates this disagreement somewhat; the story here is that we can use "messy" components in a KANSI AI but these components have to have their capabilities restricted significantly. Such restriction isn't necessary if we think messy AGI can be aligned in general.)
Intuitions and research approaches
I'm generally going to take the perspective of looking for the intuitions motivating a particular research approach or produced by a particular research approach, rather than looking at the research approaches themselves. I expect it is easier to reach agreement about the how compelling a particular intuition is (at least when other intuitions are temporarily ignored), than to reach agreement on particular research approaches.
In general, it's quite possible for a research approach to be inefficient while still being based on, or giving rise to, useful intuitions. So a criticism of a particular research approach is not necessarily a criticism of the intuitions behind it.
Terminology
Listing out intuitions
I'm going to list out a bunch of relevant intuitions. Usually I can't actually convey the intuition through text; at best I can write "what someone who has this intuition would feel like saying" and "how someone might go about gaining this intuition". Perhaps the text will make "logical" sense to you without feeling compelling; this could be a sign that you don't have the underlying intuition.
Background AI safety intuitions
These background intuitions ones that I think are shared by both Paul and MIRI.
1. Weak orthogonality. It is possible to build highly intelligent agents with silly goals such as maximizing paperclips. Random "minds from mindspace" (e.g. found through brute force search) will have values that significantly diverge from human values.
2. Instrumental convergence. Highly advanced agents will by default pursue strategies such as gaining resources and deceiving their operators (performing a "treacherous turn").
3. Edge instantiation. For most objective functions that naively seem useful, the maximum is quite "weird" in a way that is bad for human values.
4. Patch resistance. Most AI alignment problems (e.g. edge instantiation) are very difficult to "patch"; adding a patch that deals with a specific failure will fail to fix the underlying problem and instead lead to further unintended solutions.
Intuitions motivating the agent foundations approach
I think the following intuitions are sufficient to motivate the agent foundations approach to AI safety (thinking about idealized models of advanced agents to become less confused), and something similar to the agent foundations agenda, at least if one ignores contradictory intuitions for a moment. In particular, when considering these intuitions at once, I feel compelled to become less confused about advanced agents through research questions similar to those in the agent foundations agenda.
I've confirmed with Nate that these are similar to some of his main intuitions motivating the agent foundations approach.
5. Cognitive reductions are great. When we feel confused about something, there is often a way out of this confusion, by figuring out which algorithm would have generated that confusion. Often, this works even when the original problem seemed "messy" or "subjective"; something that looks messy can have simple principles behind it that haven't been discovered yet.
6. If you don't do cognitive reductions, you will put your confusion in boxes and hide the actual problem. By default, a lot of people studying a problem will fail to take the perspective of cognitive reductions and thereby not actually become less confused. The free will debate is a good example of this: most discussion of free will contains confusions that could be resolved using Daniel Dennett's cognitive reduction of free will. (This is essentially the same as the cognitive reduction discussed in the sequences.)
7. We should expect mainstream AGI research to be inefficient at learning much about the confusing aspects of intelligence, for this reason. It's pretty easy to look at most AI research and see where it's hiding fundamental confusions such as logical uncertainty without actually resolving them. E.g. if neural networks are used to predict math, then the confusion about how to do logical uncertainty is placed in the black box of "what this neural net learns to do". This isn't that helpful for actually understanding logical uncertainty in a "cognitive reduction" sense; such an understanding could lead to much more principled algorithms.
8. If we apply cognitive reductions to intelligence, we can design agents we expect to be aligned. Suppose we are able to observe "how intelligence feels from the inside" and distill these observations into an idealized cognitive algorithm for intelligence (similar to the idealized algorithm Daniel Dennett discusses to resolve free will). The minimax algorithm is one example of this: it's an idealized version of planning that in principle could have been derived by observing the mental motions humans do when playing games. If we implement an AI system that approximates this idealized algorithm, then we have a story for why the AI is doing what it is doing: it is taking action X for the same reason that an "idealized human" would take action X. That is, it "goes through mental motions" that we can imagine going through (or approximates doing so), if we were solving the task we programmed the AI to do. If we're programming the AI to assist us, we could imagine the mental motions we would take if we were assisting aliens.
9. If we don't resolve our confusions about intelligence, then we don't have this story, and this is suspicious. Suppose we haven't actually resolved our confusions about intelligence. Then we don't have the story in the previous point, so it's pretty weird to think our AI is aligned. We must have a pretty different story, and it's hard to imagine different stories that could allow us to conclude that an AI is aligned.
10. Simple reasoning rules will correctly generalize even for non-learning problems. That is, there's some way that agents can learn rules for making good judgments that generalize to tasks they can't get fast feedback on. Humans seem to be an existence proof that simple reasoning rules can generalize; science can make predictions about far-away galaxies even when there isn't an observable ground truth for the state of the galaxy (only indirect observations). Plausibly, it is possible to use "brute force" to find agents using these reasoning rules by searching for agents that perform well on small tasks and then hoping that they generalize to large tasks, but this can result in misalignment. For example, Solomonoff induction is controlled by malign consequentialists who have learned good rules for how to reason; approximating Solomonoff induction is one way to make an unaligned AI. If an aligned AI is to be roughly competitive with these "brute force" unaligned AIs, we should have some story for why the aligned AI system is also able to acquire simple reasoning rules that generalize well. Note that Paul mostly agrees with this intuition and is in favor of agent foundations approaches to solving this problem, although his research approach would significantly differ from the current agent foundations agenda. (This point is somewhat confusing; see my other post for clarification)
Intuitions motivating act-based agents
I think these following intuitions are all intuitions that Paul has that motivate his current research approach.
11. Almost all technical problems are either tractable to solve or are intractable/impossible for a good reason. This is based on Paul's experience in technical research. For example, consider a statistical learning problem where we are trying to predict a Y value from an X value using some model. It's possible to get good statistical guarantees on problems where the training distribution of X values is the same as the test distribution of X values, but when those distributions are distinguishable (i.e. there's a classifier that can separate them pretty well), there's a fundamental obstruction to getting the same guarantees: given the information available, there is no way to distinguish a model that will generalize from one that won't, since they could behave in arbitrary ways on test data that is distinctly different from training data. An exception to the rule is NP-complete problems; we don't have a good argument yet for why they can't be solved in polynomial time. However, even in this case, NP-hardness forms a useful boundary between tractable and intractable problems.
12. If the previous intuition is true, we should search for solutions and fundamental obstructions. If there is either a solution or a fundamental obstruction to a problem, then an obvious way to make progress on the problem is to alternate between generating obvious solutions and finding good reasons why a class of solutions (or all solutions) won't work. In the case of AI alignment, we should try getting a very good solution (e.g. one that allows the aligned AI to be competitive with unprincipled AI systems such as ones based on deep learning by exploiting the same techniques) until we have a fundamental obstruction to this. Such a fundamental obstruction would tell us which relaxations to the "full problem" we should consider, and be useful for convincing others that coordination is required to ensure that aligned AI can prevail even if it is not competitive with unaligned AI. (Paul's research approach looks quite optimistic partially because he is pursuing this strategy).
13. We should be looking for ways of turning arbitrary AI capabilities into equally powerful aligned AI capabilities. On priors, we should expect it to be hard for AI safety researchers to make capabilities advances; AI safety researchers make up only a small percentage of AI researchers. If this is the case, then aligned AI will be quite uncompetitive unless it takes advantage of the most effective AI technology that's already around. It would be really great if we could take an arbitrary AI technology (e.g. deep learning), do a bit of thinking, and come up with a way to direct that technology towards human values. There isn't a crisp fundamental obstruction to doing this yet, so it is the natural first place to look. To be more specific about what this research strategy entails, suppose it is possible to build built an unaligned AI system. We expect it to be competent; say it is competent for reason X. We ought to be able to either build an aligned AI system that also works for reason X, or else find a fundamental obstruction. For example, reason X could be "it does gradient descent to find weights optimizing a proxy for competence"; then we'd seek to build a system that works because it does gradient descent to find weights optimizing a proxy for competence and alignment.
14. Pursuing human narrow values presents a much more optimistic picture of AI alignment. See Paul's posts on narrow value learning, act-based agents, and abstract approval direction. The agent foundations agenda often considers problems of the form "let's use Bayesian VNM agents as our starting point and look for relaxations appropriate to realistic agents, which are naturalized". This leads to problems such as decision theory, naturalized induction, and ontology identification. However, there isn't a clear argument for why they are subproblems of the problem we actually care about (which is close to something like "pursuing human narrow values"). For example, perhaps we can understand how to have an AI pursue human narrow values without solving decision theory, since maybe humans don't actually have a utility function or a decision theory yet (though we might upon long-term reflection; pursuing narrow values should preserve the conditions for such long-term reflection). These research questions might be useful threads to pull on if solving them would tell us more about the problems we actually care about. But I think Paul has a strong intuition that working on these problems isn't the right way to make progress on pursuing human narrow values.
15. There are important considerations in favor of focusing on alignment for foreseeable AI technologies. See posts here and here. In particular, this motivates work related to alignment for systems solving learning problems.
16. It is, in principle, possible to automate a large fraction of human labor using robust learning. That is, a human can use X amount of labor to oversee the AI doing something like X3/2 amount of labor in a robust fashion. KWIK learning is a particularly clean (though impractical) demonstration of this. This enables the human to spend much more time overseeing a particular decision than the AI takes to make it (e.g. spending 1 day to oversee a decision made in 1 second), since only a small fraction of decisions are overseen.
17. The above is quite powerful, due to bootstrapping. "Automating a large fraction of human labor" is significantly more impressive than it first seems, since the human can use other AI systems in the course of evaluating a specific decision. See ALBA. We don't yet have a fundamental obstruction to any of ALBA's subproblems, and we have an argument that solving these subproblems is sufficient to create an aligned learning system.
18. There are reasons to expect the details of reasoning well to be "messy". That is, there are reasons why we might expect cognition to be as messy and hard to formalize as biology is. While biology has some important large-scale features (e.g. evolution), overall it is quite hard to capture using simple rules. We can take the history of AI as evidence for this; AI research often does consist of people trying to figure out how humans do something at an idealized level and formalize it (roughly similar to the agent foundations approach), and this kind of AI research does not always lead to the most capable AI systems. The success of deep learning is evidence that the most effective way for AI systems to acquire good rules of reasoning is usually to learn them, rather than having them be hardcoded.
What to do from here?
I find all the intuitions above at least somewhat compelling. Given this, I have made some tentative conclusions:
Acknowledgements
Thanks to Paul, Nate, Eliezer, and Benya for a lot of conversations on this topic. Thanks to John Salvatier for helping me to think about intuitions and teaching me skills for learning intuitions from other people.