The Main Sources of AI Risk?

byWei Dai2mo21st Mar 20196 comments


There are so many causes or sources of AI risk that it's getting hard to keep them all in mind. I propose we keep a list of the main sources (that we know about), such that we can say that if none of these things happen, then we've mostly eliminated AI risk (as an existential risk) at least as far as we can determine. Here's a list that I spent a couple of hours enumerating and writing down. Did I miss anything important?

  1. Insufficient time/resources for AI safety (for example caused by intelligence explosion or AI race)
  2. Insufficient global coordination, leading to the above
  3. Misspecified or incorrectly learned goals/values
  4. Inner optimizers
  5. ML differentially accelerating easy to measure goals
  6. Paul's "influence-seeking behavior" (a combination of 3 and 4 above?)
  7. AI generally accelerating intellectual progress in a wrong direction (e.g., accelerating unsafe/risky technologies more than knowledge/wisdom about how to safely use those technologies)
  8. Metaethical error
  9. Metaphilosophical error
  10. Other kinds of philosophical errors in AI design (e.g., giving AI a wrong prior or decision theory)
  11. Other design/coding errors (e.g., accidentally putting a minus sign in front of utility function, supposedly corrigible AI not actually being corrigible)
  12. Doing acausal reasoning in a wrong way (e.g., failing to make good acausal trades, being acausally extorted, failing to acausally influence others who can be so influenced)
  13. Human-controlled AIs ending up with wrong values due to insufficient "metaphilosophical paternalism"
  14. Human-controlled AIs causing ethical disasters (e.g., large scale suffering that can't be "balanced out" later) prior to reaching moral/philosophical maturity
  15. Intentional corruption of human values
  16. Unintentional corruption of human values
  17. Mind crime (disvalue unintentionally incurred through morally relevant simulations in AIs' minds)
  18. Premature value lock-in (i.e., freezing one's current conception of what's good into a utility function)
  19. Extortion between AIs leading to vast disvalue
  20. Distributional shifts causing apparently safe/aligned AIs to stop being safe/aligned
  21. Value drift and other kinds of error as AIs self-modify, or AIs failing to solve value alignment for more advanced AIs
  22. Treacherous turn / loss of property rights due to insufficient competitiveness of humans & human-aligned AIs
  23. Gradual loss of influence due to insufficient competitiveness of humans & human-aligned AIs
  24. Utility maximizers / goal-directed AIs having an economic and/or military competitive advantage due to relative ease of cooperation/coordination, defense against value corruption and other forms of manipulation and attack, leading to one or more of the above
  25. In general, the most competitive type of AI being too hard to align or to safely use
  26. Computational resources being too cheap, leading to one or more of the above

(With this post I mean to (among other things) re-emphasize the disjunctive nature of AI risk, but this list isn't fully disjunctive (i.e., some of the items are subcategories or causes of others), and I mostly gave a source of AI risk its own number in the list if it seemed important to make that source more salient. Maybe once we have a list of everything that is important, it would make sense to create a graph out of it.)