While I agree that the COVID response was worse than it could have been, I think there are several important disanalogies between the COVID-19 pandemic and a soft AI takeoff scenario:
1. Many new problems arose during this pandemic for which we did not have historical experience, e.g. in supply chains. (Perhaps we had historical precedent in the Spanish flu, but that was sufficiently long ago that I don’t expect those lessons to generalize, or for us to remember those lessons.) In contrast, I expect that with AI alignment the problems will not change much as the AI systems become more powerful. Certainly the effects of misaligned powerful AI systems will change dramatically and be harder to mitigate, but I expect the underlying causes of misalignment will not change much, and that’s what we need to gain consensus about and find solutions for. EDIT: Note that with COVID we failed even at existing, known problems (see Raemon's comment thread below), so this point doesn't really explain away our failure with COVID.
2. It looks like most institutions took action in about 2 months (mid-Jan to mid-March). While this was (I assume) too slow for COVID, it seems more than sufficient for AI alignment under soft takeoff, where I expect we will have multiple years for the relevant decision-making. However, unlike COVID, there probably won’t be a specific crisis mode that leads to quick(er) decision-making: while there might be warning signs that suggest that action needs to be taken for AI systems, it may not lead to action specifically targeted at AI x-risk.
3. It seems that the model in this post is that we should have learned from past epidemics, and applied it to solve this pandemic. However, in AI alignment, the hope is to learn from failures of narrow AI systems, and use that to prevent failures in more powerful AI systems. I would be pretty pessimistic if AI alignment was banking on noticing failures in powerful AI systems and then quickly mobilizing institutions to mitigate those failures, rather than preventing the failures in the first place. The analogous actions for COVID would be things we could have done before we knew about COVID to mitigate some unknown future pandemic. I don’t know enough about epidemiology to say whether or not there were cost-effective actions that should have been taken ahead of time, but note that any such argument should be evaluated ex ante (i.e. from the perspective where we don’t know that COVID-19 would happen).
Separately, it seems like humanity has in fact significantly mitigated the effects of COVID (something like we reduced deaths to a fraction of what they "could have been"), so if you want to take an extremely outside view approach, you should predict that with AI alignment we'll mitigate the worst effects but there will still be some pretty bad effects, which still argues for not-extinction. (I don't personally buy this reasoning; I mention it as a response to people who say "look at all of our civilization's failures, therefore we should predict failure at AI alignment too".)
I am mostly confused, but I expect that if I learned more I would say that it wasn't a fair characterization of the FDA.
(Cross-posted from personal blog. Summarized in Alignment Newsletter #104. Thanks to Janos Kramar for his helpful feedback on this post.)
Epistemic status: fairly speculative, would appreciate feedback
As the covid-19 pandemic unfolds, we can draw lessons from it for managing future global risks, such as other pandemics, climate change, and risks from advanced AI. In this post, I will focus on possible implications for AI risk. For a broader treatment of this question, I recommend FLI's covid-19 page that includes expert interviews on the implications of the pandemic for other types of risks.
A key element in AI risk scenarios is the speed of takeoff - whether advanced AI is developed gradually or suddenly. Paul Christiano's post on takeoff speeds defines slow takeoff in terms of the economic impact of AI as follows: "There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles." It argues that slow AI takeoff is more likely than fast takeoff, but is not necessarily easier to manage, since it poses different challenges, such as large-scale coordination. This post expands on this point by examining some parallels between the coronavirus pandemic and a slow takeoff scenario. The upsides of slow takeoff include the ability to learn from experience, act on warning signs, and reach a timely consensus that there is a serious problem. I would argue that the covid-19 pandemic had these properties, but most of the world's institutions did not take advantage of them. This suggests that, unless our institutions improve, we should not expect the slow AI takeoff scenario to have a good default outcome.
We can hope that the transformative technological change involved in the slow takeoff scenario will also help create more competent institutions without these weaknesses. We might expect that institutions unable to adapt to the fast pace of change will be replaced by more competent ones. However, we could also see an increasingly chaotic world where institutions fail to adapt without better institutions being formed quickly enough to replace them. Success in the slow takeoff scenario depends on institutional competence and large-scale coordination. Unless more competent institutions are in place by the time general AI arrives, it is not clear to me that slow takeoff would be much safer than fast takeoff.