Related to: Should I believe what the SIAI claims?; What I would like the SIAI to publish

The argument, that an AI can go FOOM (undergo explosive recursive self-improvement), requires various premises (P#) to be true simultaneously:

P1: The human development of artificial general intelligence will take place quickly.
P2: Any increase in intelligence does vastly outweigh its computational cost and the expenditure of time needed to discover it.
P3: AGI is able to create, or acquire, resources, empowering technologies or civilisatory support.
P4: AGI can undergo explosive recursive self-improvement and reach superhuman intelligence without having to rely on slow environmental feedback.
P5: Goal stability and self-preservation are not requirements for an AGI to undergo explosive recursive self-improvement.
P6: AGI researchers will be smart enough and manage to get everything right, including a mathematically precise definition of the AGI's utility-function, yet fail to implement spatio-temporal scope boundaries, resource usage and optimization limits.

Therefore the probability of an AI to go FOOM (P(FOOM)) is the probability of the conjunction (P#∧P#) of its premises:

P(FOOM) = P(P1∧P2∧P3∧P4∧P5∧P6)

Of course, there are many more premises that need to be true in order to enable an AI to go FOOM, e.g. that each level of intelligence can effectively handle its own complexity, or that most AGI designs can somehow self-modify their way up to massive superhuman intelligence. But I believe that the above points are enough to show that the case for a hard takeoff is not disjunctive, but rather strongly conjunctive.

Premise 1 (P1): If the development of AGI takes place slowly, a gradual and controllable development, we might be able to learn from small-scale mistakes, or have enough time to develop friendly AI, while having to face other existential risks.

This might for example be the case if intelligence can not be captured by a discrete algorithm, or is modular, and therefore never allow us to reach a point where we can suddenly build the smartest thing ever that does just extend itself indefinitely.

Premise 2 (P2): If you increase intelligence you might also increase the computational cost of its further improvement, the distance to the discovery of some unknown unknown that could enable another quantum leap, by reducing the design space with every iteration.

If an AI does need to apply a lot more energy to get a bit more complexity, then it might not be instrumental for an AGI to increase its intelligence, rather than using its existing intelligence to pursue its terminal goals or to invest its given resources to acquire other means of self-improvement, e.g. more efficient sensors.

Premise 3 (P3): If artificial general intelligence is unable to seize the resources necessary to undergo explosive recursive self-improvement (FOOM), then, the ability and cognitive flexibility of superhuman intelligence in and of itself, as characteristics alone, would have to be sufficient to self-modify its way up to massive superhuman intelligence within a very short time.

Without advanced real-world nanotechnology it will be considerable more difficult for an AI to FOOM. It will have to make use of existing infrastructure, e.g. buy stocks of chip manufactures and get them to create more or better CPU’s. It will have to rely on puny humans for a lot of tasks. It won’t be able to create new computational substrate without the whole economy of the world supporting it. It won’t be able to create an army of robot drones overnight without it either.

Doing so it would have to make use of considerable amounts of social engineering without its creators noticing it. But, more importantly, it will have to make use of its existing intelligence to do all of that. The AGI would have to acquire new resources slowly, as it couldn’t just self-improve to come up with faster and more efficient solutions. In other words, self-improvement would demand resources, therefore the AGI could not profit from its ability to self-improve, regarding the necessary acquisition of resources, to be able to self-improve in the first place.

Therefore the absence of advanced nanotechnology constitutes an immense blow to the possibility of explosive recursive self-improvement.

One might argue that an AGI will solve nanotechnology on its own and find some way to trick humans into manufacturing a molecular assembler and grant it access to it. But this might be very difficult.

There is a strong interdependence of resources and manufacturers. The AI won’t be able to simply trick some humans to build a high-end factory to create computational substrate, let alone a molecular assembler. People will ask questions and shortly after get suspicious. Remember, it won’t be able to coordinate a world-conspiracy, it hasn’t been able to self-improve to that point yet, because it is still trying to acquire enough resources, which it has to do the hard way without nanotech.

Anyhow, you’d probably need a brain the size of the moon to effectively run and coordinate a whole world of irrational humans by intercepting their communications and altering them on the fly without anyone freaking out.

If the AI can’t make use of nanotechnology it might make use of something we haven’t even thought about. What, magic?

Premise 4 (P4): Just imagine you emulated a grown up human mind and it wanted to become a pick up artist, how would it do that with an Internet connection? It would need some sort of avatar, at least, and then wait for the environment to provide a lot of feedback.

So, even if we're talking about the emulation of a grown up mind, it will be really hard to acquire some capabilities. Then how is the emulation of a human toddler going to acquire those skills? Even worse, how is some sort of abstract AGI going to do it that misses all of the hard coded capabilities of a human toddler?

Can we even attempt to imagine what is wrong about a boxed emulation of a human toddler, that makes it unable to become a master of social engineering in a very short time?

Can we imagine what is missing that would enable one of the existing expert systems to quickly evolve vastly superhuman capabilities in its narrow area of expertise?

Premise 5 (P5): A paperclip maximizer wants to guarantee that its goal of maximizing paperclips will be preserved when it improves itself.

By definition, a paperclip maximizer is unfriendly, does not feature inherent goal-stability (a decision theory of self-modifying decision systems), and therefore has to use its initial seed intelligence to devise a sort of paperclip-friendliness before it can go FOOM.

Premise 6 (P6): Complex goals need complex optimization parameters (the design specifications of the subject of the optimization process against which it will measure its success of self-improvement).

Even the creation of paperclips is a much more complex goal than telling an AI to compute as many digits of Pi as possible.

For an AGI, that was designed to design paperclips, to pose an existential risk, its creators would have to be capable enough to enable it to take over the universe on its own, yet forget, or fail to, define time, space and energy bounds as part of its optimization parameters. Therefore, given the large amount of restrictions that are inevitably part of any advanced general intelligence (AGI), the nonhazardous subset of all possible outcomes might be much larger than that where the AGI works perfectly yet fails to hold before it could wreak havoc.