Yet almost everyone agrees the world will likely be importantly different by the time advanced AGI arrives.
Why do you think this? My default assumption is generally that the world won't be super different from how it looks today in strategically relevant ways. (Maybe it will be, but I don't see a strong reason to assume that, though I strongly endorse thinking about big possible changes!)
Maybe I was overconfident here. I was generalizing from the sample of people I'd talked to. Also, as you'll see by reading the entries on the list, I have a somewhat low bar for strategic relevance.
Epistemic status: I started this as an AI Impacts research project, but given that it’s fundamentally a fun speculative brainstorm, it worked better as a blog post.
The default, when reasoning about advanced artificial general intelligence (AGI), is to imagine it appearing in a world that is basically like the present. Yet almost everyone agrees the world will likely be importantly different by the time advanced AGI arrives.
One way to address this problem is to reason in abstract, general ways that are hopefully robust to whatever unforeseen developments lie ahead. Another is to brainstorm particular changes that might happen, and check our reasoning against the resulting list.
This is an attempt to begin the second approach.2 I sought things that might happen that seemed both (a) within the realm of plausibility, and (b) probably strategically relevant to AI safety or AI policy.
I collected potential list entries via brainstorming, asking others for ideas, googling, and reading lists that seemed relevant (e.g. Wikipedia’s list of emerging technologies,3 a list of Ray Kurzweil’s predictions4, and DARPA’s list of projects.5)
I then shortened the list based on my guesses about the plausibility and relevance of these possibilities. I did not put much time into evaluating any particular possibility, so my guesses should not be treated as anything more. I erred on the side of inclusion, so the entries in this list vary greatly in plausibility and relevance. I made some attempt to categorize these entries and merge similar ones, but this document is fundamentally a brainstorm, not a taxonomy, so keep your expectations low.
I hope to update this post as new ideas find me and old ideas are refined or refuted. I welcome suggestions and criticisms; email me (gmail kokotajlod) or leave a comment.
Asya Bergal and I made an interactive button to go with the list. The button randomly generates a possible future according to probabilities that you choose. It is very crude, but it has been fun to play with, and perhaps even slightly useful. For example, once I decided that my credences were probably systematically too high because the futures generated with them were too crazy. Another time I used the alternate method (described below) to recursively generate a detailed future trajectory, written up here. I hope to make more trajectories like this in the future, since I think this method is less biased than the usual method for imagining detailed futures.6
To choose probabilities, scroll down to the list below and fill each box with a number representing how likely you think the entry is to occur in a strategically relevant way prior to the advent of advanced AI. (1 means certainly, 0 means certainly not. The boxes are all 0 by default.) Once you are done, scroll back up and click the button.
A major limitation is that the button doesn’t take correlations between possibilities into account. The user needs to do this themselves, e.g. by redoing any generated future that seems silly, or by flipping a coin to choose between two generated possibilities that seem contradictory, or by choosing between them based on what else was generated.
Here is an alternate way to use this button that mostly avoids this limitation:
If you don’t want to choose probabilities yourself, click “fill with pre-set values” to populate the fields with my non-expert, hasty guesses.7
GENERATE FUTURE
Fill with pre-set values (default method)
Fill with pre-set values (alternate method)
Letters after list titles indicate that I think the change might be relevant to:
Each possibility is followed by some explanation or justification where necessary, and a non-exhaustive list of ways the possibility may be relevant to AI outcomes in particular (which is not guaranteed to cover the most important ones). Possibilities are organized into loose categories created after the list was generated.
1. Advanced science automation and research tools (TML, TAS, CHA, MIS)
Narrow research and development tools might speed up technological progress in general or in specific domains. For example, several of the other technologies on this list might be achieved with the help of narrow research and development tools.
2. Dramatically improved computing hardware (TML, TAS, POL, MIS)
By this I mean computing hardware improves at least as fast as Moore’s Law. Computing hardware has historically become steadily cheaper, though it is unclear whether this trend will continue. Some example pathways by which hardware might improve at least moderately include:
Dramatically improved computing hardware may:
3. Stagnation in computing hardware progress (TML, TAS, POL, MIS)
Many forecasters think Moore’s Law will be ending soon (as of 2020).18 In the absence of successful new technologies, computing hardware could progress substantially more slowly than Moore’s Law would predict.
Stagnation in computing hardware progress may:
4. Manufacturing consolidation (POL)
Chip fabrication has become more specialized and consolidated over time, to the point where all of the hardware relevant to AI research depends on production from a handful of locations.19 Perhaps this trend will continue.
One country (or a small number working together) could control or restrict AI research by controlling the production and distribution of necessary hardware.
5. Advanced additive manufacturing (e.g. 3D printing or nanotechnology) (TML, CHA)
Advanced additive manufacturing could lead to various materials, products and forms of capital being cheaper and more broadly accessible, as well as to new varieties of them becoming feasible and quicker to develop. For example, sufficiently advanced 3D printing could destabilize the world by allowing almost anyone to secretly produce terror weapons. If nanotechnology advances rapidly, so that nanofactories can be created, the consequences could be dramatic:20
6. Massive resource glut (TML, TAS, POL, CHA)
By “glut” I don’t necessarily mean that there is too much of a resource. Rather, I mean that the real price falls dramatically. Rapid decreases in the price of important resources have happened before.21 It could happen again via:
My impression is that energy, raw materials, and unskilled labor combined are less than half the cost of computing, so a decrease in the price of one of these (and possibly even all three) would probably not have large direct consequences on the price of computing.28 But a resource glut might lead to general economic prosperity, with many subsequent effects on society, and moreover the cost structure of computing may change in the future, creating a situation where a resource glut could dramatically lower the cost of computing.29
7. Hardware overhang (TML, TAS, POL)
Hardware overhang refers to a situation where large quantities of computing hardware can be diverted to running powerful AI systems as soon as the AI software is developed.
If advanced AGI (or some other powerful software) appears during a period of hardware overhang, its capabilities and prominence in the world could grow very quickly.
8. Hardware underhang (TML, TAS, POL)
The opposite of hardware overhang might happen. Researchers may understand how to build advanced AGI at a time when the requisite hardware is not yet available. For example, perhaps the relevant AI research will involve expensive chips custom-built for the particular AI architecture being trained.
A successful AI project during a period of hardware underhang would not be able to instantly copy the AI to many other devices, nor would they be able to iterate quickly and make an architecturally improved version.
9. Prediction tools (TML, TAS, POL, CHA, MIS)
Tools may be developed that are dramatically better at predicting some important aspect of the world; for example, technological progress, cultural shifts, or the outcomes of elections, military clashes, or research projects. Such tools could for instance be based on advances in AI or other algorithms, prediction markets, or improved scientific understanding of forecasting (e.g. lessons from the Good Judgment Project).
Such tools might conceivably increase stability via promoting accurate beliefs, reducing surprises, errors or unnecessary conflicts. However they could also conceivably promote instability via conflict encouraged by a powerful new tool being available to a subset of actors. Such tools might also help with forecasting the arrival and effects of advanced AGI, thereby helping guide policy and AI safety work. They might also accelerate timelines, for instance by assisting project management in general and notifying potential investors when advanced AGI is within reach.
10. Persuasion tools (POL, CHA, MIS)
Present technology for influencing a person’s beliefs and behavior is crude and weak, relative to what one can imagine. Tools may be developed that more reliably steer a person’s opinion and are not so vulnerable to the victim’s reasoning and possession of evidence. These could involve:
Strong persuasion tools could:
11. Theorem provers (TAS)
Powerful theorem provers might help with the kinds of AI alignment research that involve proofs or help solve computational choice problems.
12. Narrow AI for natural language processing (TML, TAS, CHA)
Researchers may develop narrow AI that understands human language well, including concepts such as “moral” and “honest.”
Natural language processing tools could help with many kinds of technology, including AI and various AI safety projects. They could also help enable AI arbitration systems. If researchers develop software that can autocomplete code—much as it currently autocompletes text messages—it could multiply software engineering productivity.
13. AI interpretability tools (TML, TAS, POL)
Tools for understanding what a given AI system is thinking, what it wants, and what it is planning would be useful for AI safety.31
14. Credible commitment mechanisms (POL, CHA)
There are significant restrictions on which contracts governments are willing and able to enforce–for example, they can’t enforce a contract to try hard to achieve a goal, and won’t enforce a contract to commit a crime. Perhaps some technology (e.g. lie detectors, narrow AI, or blockchain) could significantly expand the space of possible credible commitments for some relevant actors: corporations, decentralized autonomous organizations, crowds of ordinary people using assurance contracts, terrorist cells, rogue AGIs, or even individuals.
This might destabilize the world by making threats of various kinds more credible, for various actors. It might stabilize the world in other ways, e.g. by making it easier for some parties to enforce agreements.
15. Better coordination tools (POL, CHA, MIS)
Technology for allowing groups of people to coordinate effectively could improve, potentially avoiding losses from collective choice problems, helping existing large groups (e.g. nations and companies) to make choices in their own interests, and producing new forms of coordinated social behavior (e.g. the 2010’s saw the rise of the Facebook group)). Dominant assurance contracts,32 improved voting systems,33 AI arbitration systems, lie detectors, and similar things not yet imagined might significantly improve the effectiveness of some groups of people.
If only a few groups use this technology, they might have outsized influence. If most groups do, there could be a general reduction in conflict and increase in good judgment.
16. Deterioration of collective epistemology (TML, TAS, POL, CHA, MIS)
Society has mechanisms and processes that allow it to identify new problems, discuss them, and arrive at the truth and/or coordinate a solution. These processes might deteriorate. Some examples of things which might contribute to this:
This could cause chaos in the world in general, and lead to many hard-to-predict effects. It would likely make the market for influencing the course of AI development less efficient (see section on “Landscape of…” below) and present epistemic hazards for anyone trying to participate effectively.
17. New and powerful forms of addiction (TML, POL, CHA, MIS)
Technology that wastes time and ruins lives could become more effective. The average person spends 144 minutes per day on social media, and there is a clear upward trend in this metric.35 The average time spent watching TV is even greater.36 Perhaps this time is not wasted but rather serves some important recuperative, educational, or other function. Or perhaps not; perhaps instead the effect of social media on society is like the effect of a new addictive drug — opium, heroin, cocaine, etc. — which causes serious damage until society adapts. Maybe there will be more things like this: extremely addictive video games, or newly invented drugs, or wireheading (directly stimulating the reward circuitry of the brain).37
This could lead to economic and scientific slowdown. It could also concentrate power and influence in fewer people—those who for whatever reason remain relatively unaffected by the various productivity-draining technologies. Depending on how these practices spread, they might affect some communities more or sooner than others.
18. Medicine to boost human mental abilities (TML, CHA, MIS)
To my knowledge, existing “study drugs” such as modafinil don’t seem to have substantially sped up the rate of scientific progress in any field. However, new drugs (or other treatments) might be more effective. Moreover, in some fields, researchers typically do their best work at a certain age. Medicine which extends this period of peak mental ability might have a similar effect.
This could speed up the rate of scientific progress in some fields, among other effects.
19. Genetic engineering, human cloning, iterated embryo selection (TML, POL, CHA, MIS)
Changes in human capabilities or other human traits via genetic interventions38 could affect many areas of life. If the changes were dramatic, they might have a large impact even if only a small fraction of humanity were altered by them.
Changes in human capabilities or other human traits via genetic interventions might:
20. Landscape of effective strategies for influencing AI development changes substantially (CHA, MIS)
For a person at a time, there is a landscape of strategies for influencing the world, and in particular for influencing AI development and the effects of advanced AGI. The landscape could change such that the most effective strategies for influencing AI development are:
Here is a non-exhaustive list of reasons to think these features might change systematically over time:
A shift in the landscape of effective strategies for influencing the course of AI is relevant to anyone who wants to have an effective strategy for influencing the course of AI.40 If it is part of a more general shift in the landscape of effective strategies for other goals — e.g. winning wars, making money, influencing politics — the world could be significantly disrupted in ways that may be hard to predict.
21. Global economic collapse (TML, CHA, MIS)
This might slow down research or precipitate other relevant events, such as war.
22. Scientific stagnation (TML, TAS, POL, CHA, MIS)
There is some evidence that scientific progress in general might be slowing down. For example, the millennia-long trend of decreasing economic doubling time seems to have stopped around 1960.41 Meanwhile, scientific progress has arguably come from increased investment in research. Since research investment has been growing faster than the economy, it might eventually saturate and grow only as fast as the economy.42
This might slow down AI research, making the events on this list (but not the technologies) more likely to happen before advanced AGI.
23. Global catastrophe (TML, POL, CHA)
Here are some examples of potential global catastrophes:
A global catastrophe might be expected to cause conflict and slowing of projects such as research, though it could also conceivably increase attention on projects that are useful for dealing with the problem. It seems likely to have other hard to predict effects.
24. Shift in level of public attention on AGI (TML, POL, CHA, MIS)
The level of attention paid to AGI by the public, governments, and other relevant actors might increase (e.g. due to an impressive demonstration or a bad accident) or decrease (e.g. due to other issues drawing more attention, or evidence that AI is less dangerous or imminent).
Changes in the level of attention could affect the amount of work on AI and AI safety. More attention could also lead to changes in public opinion such as panic or an AI rights movement.
If the level of attention increases but AGI does not arrive soon thereafter, there might be a subsequent period of disillusionment.
25. Change in investment in AGI development (TML, TAS, POL)
There could be a rush for AGI, for instance if major nations begin megaprojects to build it. Or there could be a rush away from AGI, for instance if it comes to be seen as immoral or dangerous like human cloning or nuclear rocketry.
Increased investment in AGI might make advanced AGI happen sooner, with less hardware overhang and potentially less proportional investment in safety. Decreased investment might have the opposite effects.
26. New social movements or ideological shifts (TML, TAS, POL, MIS)
The communities that build and regulate AI could undergo a substantial ideological shift. Historically, entire nations have been swept by radical ideologies within about a decade or so, e.g. Communism, Fascism, the Cultural Revolution, and the First Great Awakening.47 Major ideological shifts within communities smaller than nations (or within nations, but on specific topics) presumably happen more often. There might even appear powerful social movements explicitly focused on AI, for instance in opposition to it or attempting to secure legal rights and moral status for AI agents.48 Finally, there could be a general rise in extremist movements, for instance due to a symbiotic feedback effect hypothesized by some,49 which might have strategically relevant implications even if mainstream opinions do not change.
Changes in public opinion on AI might change the speed of AI research, change who is doing it, change which types of AI are developed or used, and limit or alter discussion. For example, attempts to limit an AI system’s effects on the world by containing it might be seen as inhumane, as might adversarial and population-based training methods. Broader ideological change or a rise in extremisms might increase the probability of a massive crisis, revolution, civil war, or world war.
27. Harbinger of AGI (ALN, POL, MIS)
Events could occur that provide compelling evidence, to at least a relevant minority of people, that advanced AGI is near.
This could increase the amount of technical AI safety work and AI policy work being done, to the extent that people are sufficiently well-informed and good at forecasting. It could also enable people already doing such work to more efficiently focus their efforts on the true scenario.
28. AI alignment warning shot (ALN, POL)
A convincing real-world example of AI alignment failure could occur.
This could motivate more effort into mitigating AI risk and perhaps also provide useful evidence about some kinds of risks and how to avoid them.
29. Brain scanning (TML, TAS, POL, CHA, MIS)
An accurate way to scan human brains at a very high resolution could be developed.
Combined with a good low-level understanding of the brain (see below) and sufficient computational resources, this might enable brain emulations, a form of AGI in which the AGI is similar, mentally, to some original human. This would change the kind of technical AI safety work that would be relevant, as well as introducing new AI policy questions. It would also likely make AGI timelines easier to predict. It might influence takeoff speeds.
30. Good low-level understanding of the brain (TML, TAS, POL, CHA, MIS)
To my knowledge, as of April 2020, humanity does not understand how neurons work well enough to accurately simulate the behavior of a C. Elegans worm, though all connections between its neurons have been mapped50 Ongoing progress in modeling individual neurons could change this, and perhaps ultimately allow accurate simulation of entire human brains.
Combined with brain scanning (see above) and sufficient computational resources, this may enable brain emulations, a form of AGI in which the AI system is similar, mentally, to some original human. This would change the kind of AI safety work that would be relevant, as well as introducing new AI policy questions. It would also likely make the time until AGI is developed more predictable. It might influence takeoff speeds. Even if brain scanning is not possible, a good low-level understanding of the brain might speed AI development, especially of systems that are more similar to human brains.
31. Brain-machine interfaces (TML, TAS, POL, CHA, MIS)
Better, safer, and cheaper methods to control computers directly with our brains may be developed. At least one project is explicitly working towards this goal.51
Strong brain-machine interfaces might:
32. In vitro brains (TML, TAS, POL, CHA)
Neural tissue can be grown in a dish (or in an animal and transplanted) and connected to computers, sensors, and even actuators.53 If this tissue can be trained to perform important tasks, and the technology develops enough, it might function as a sort of artificial intelligence. Its components would not be faster than humans, but it might be cheaper or more intelligent. Meanwhile, this technology might also allow fresh neural tissue to be grafted onto existing humans, potentially serving as a cognitive enhancer.54
This might change the sorts of systems AI safety efforts should focus on. It might also automate much human labor, inspire changes in public opinion about AI research (e.g. promoting concern about the rights of AI systems), and have other effects which are hard to predict.
33. Weak AGI (TML, TAS, POL, CHA, MIS)
Researchers may develop something which is a true artificial general intelligence—able to learn and perform competently all the tasks humans do—but just isn’t very good at them, at least, not as good as a skilled human.
If weak AGI is faster or cheaper than humans, it might still replace humans in many jobs, potentially speeding economic or technological progress. Separately, weak AGI might provide testing opportunities for technical AI safety research. It might also change public opinion about AI, for instance inspiring a “robot rights” movement, or an anti-AI movement.
34. Expensive AGI (TML, TAS, POL, CHA, MIS)
Researchers may develop something which is a true artificial general intelligence, and moreover is qualitatively more intelligent than any human, but is vastly more expensive, so that there is some substantial period of time before cheap AGI is developed.
An expensive AGI might contribute to endeavors that are sufficiently valuable, such as some science and technology, and so may have a large effect on society. It might also prompt increased effort on AI or AI safety, or inspire public thought about AI that produces changes in public opinion and thus policy, e.g. regarding the rights of machines. It might also allow opportunities for trialing AI safety plans prior to very widespread use.
35. Slow AGI (TML, TAS, POL, CHA, MIS)
Researchers may develop something which is a true artificial general intelligence, and moreover is qualitatively as intelligent as the smartest humans, but takes a lot longer to train and learn than today’s AI systems.
Slow AGI might be easier to understand and control than other kinds of AGI, because it would train and learn more slowly, giving humans more time to react and understand it. It might produce changes in public opinion about AI.
36. Automation of human labor (TML, TAS, POL, CHA, MIS)
If the pace of automation substantially increases prior to advanced AGI, there could be social upheaval and also dramatic economic growth. This might affect investment in AI.
37. Major leak of AI research (TML, TAS, POL, CHA)
Edward Snowden defected from the NSA and made public a vast trove of information. Perhaps something similar could happen to a leading tech company or AI project.
In a world where much AI progress is hoarded, such an event could accelerate timelines and make the political situation more multipolar and chaotic.
38. Shift in favor of espionage (POL, CHA, MIS)
Espionage techniques might become more effective relative to counterespionage techniques. In particular:
More successful espionage techniques might make it impossible for any AI project to maintain a lead over other projects for any substantial period of time. Other disruptions may become more likely, such as hacking into nuclear launch facilities, or large scale cyberwarfare.
39. Shift in favor of counterespionage (POL, CHA, MIS)
Counterespionage techniques might become more effective relative to espionage techniques than they are now. In particular:
Stronger counterespionage techniques might make it easier for an AI project to maintain a technological lead over the rest of the world. Cyber wars and other disruptive events could become less likely.
40. Broader or more sophisticated surveillance (POL, CHA, MIS)
More extensive or more sophisticated surveillance could allow strong and selective policing of technological development. It would also have other social effects, such as making totalitarianism easier and making terrorism harder.
41. Autonomous weapons (POL, CHA)
Autonomous weapons could shift the balance of power between nations, or shift the offense-defense balances resulting in more or fewer wars or terrorist attacks, or help to make totalitarian governments more stable. As a potentially early, visible and controversial use of AI, they may also especially influence public opinion on AI more broadly, e.g. prompting anti-AI sentiment.
42. Shift in importance of governments, corporations, and other groups in AI development (POL, CHA)
Currently both governments and corporations are strategically relevant actors in determining the course of AI development. Perhaps governments will become more important, e.g. by nationalizing and merging AI companies. Or perhaps governments will become less important, e.g. by not paying attention to AI issues at all, or by becoming less powerful and competent generally. Perhaps some third kind of actor (such as religion, insurgency, organized crime, or special individual) will become more important, e.g. due to persuasion tools, countermeasures to surveillance, or new weapons of guerilla warfare.59
This influences AI policy by affecting which actors are relevant to how AI is developed and deployed.
43. Catastrophe in strategically important location (TML, POL, CHA, MIS)
Perhaps some strategically important location (e.g. tech hub, seat of government, or chip fab) will be suddenly destroyed. Here is a non-exhaustive list of ways this could happen:
If it happens, it might be strategically disruptive, causing e.g. the dissolution and diaspora of the front-runner AI project, or making it more likely that some government makes a radical move of some sort.
44. Change in national AI research loci (POL, CHA)
For instance, a new major national hub of AI research could arise, rivalling the USA and China in research output. Or either the USA or China could cease to be relevant to AI research.
This might make coordinating AI policy more difficult. It might make a rush for AGI more or less likely.
45. Large war (TML, POL, CHA, MIS)
This might cause short-term, militarily relevant AI capabilities research to be prioritized over AI safety and foundational research. It could also make global coordination on AI policy difficult.
46. Civil war or regime change in major relevant countries (POL, CHA, MIS)
This might be very dangerous for people living in those countries. It might change who the strategically relevant actors are for shaping AI development. It might result in increased instability, or cause a new social movement or ideological shift.
47. Formation of a world government (POL, CHA)
This would make coordinating AI policy easier in some ways (e.g. there would be no need for multiple governing bodies to coordinate their policy at the highest level), however it might be harder in others (e.g. there might be a more complicated regulatory system overall).
18 June 2020.
(Edited to add text)