YouTube link

The Future of Life Institute is one of the oldest and most prominant organizations in the AI existential safety space, working on such topics as the AI pause open letter and how the EU AI Act can be improved. Metaculus is one of the premier forecasting sites on the internet. Behind both of them lie one man: Anthony Aguirre, who I talk with in this episode.

Topics we discuss:

Anthony, FLI, and Metaculus
The Alignment Workshop
FLI’s current activity
AI policy
Work FLI funds

Daniel Filan (00:09): Hello, everyone. This is one of a series of short interviews that I’ve been conducting at the Bay Area Alignment Workshop, which is run by FAR.AI. Links to what we’re discussing, as usual, are in the description. A transcript is, as usual, available at axrp.net. And, as usual, if you want to support the podcast, you can do so at patreon.com/axrpodcast. Well, let’s continue to the interview.

(00:29): All right, well, I am currently speaking with Anthony Aguirre.

Anthony Aguirre (00:32): Nice to be here.

Anthony, FLI, and Metaculus

Daniel Filan (00:33): For those who aren’t as familiar with you and what you do, can you say a little bit about yourself?

Anthony Aguirre (00:39): I’m a theoretical physicist by original training. I still hold a professorship at UC Santa Cruz, and I’ve studied a whole range of theoretical physics things in the early universe and gravity and foundations of quantum mechanics and all sorts of fun stuff. But about 10 years ago or so, I, as well as another colleague, Max Tegmark, and a few others, started to think about the long-term future, and in particular, transformative technologies and the role that they would be playing, and decided to start up a non-profit thinking about transformative technologies and what could be done to steer them in maybe a more favorable direction. At that point, AI was just starting to work a little bit and things like AGI were clearly decades away, if not centuries in some people’s opinion. So this was a more theoretical and pleasantly abstract set of things to think about.

(01:36): But nonetheless, we felt like it was coming and started a bunch of activities and that was the birth of the Future of Life Institute. And I’m now, 10 years later, executive director of the Future of Life Institute pretty much full-time. And so I’ve got sort of a split between Future of Life Institute, my academic appointment and my students there, and a couple of other hats that I wear, including Metaculus.

Daniel Filan (01:58): How does Metaculus fit into all of that? I imagine once you’re both a professor and helping run FLI, I’m surprised you had the time to start this other thing.

Anthony Aguirre (02:07): Well, I started Metaculus at the same time as FLI, so I didn’t know what I was getting into at that point.

Daniel Filan (02:13): Classic way people do things.

Anthony Aguirre (02:14): Yes. So Metaculus actually started with FLI for a reason, because thinking about the future and how we could steer it in more positive directions, I felt like, well, then we have to know a little bit about what might happen in the future, and conditionally, if we did X and Y, what might that do? And also how do we build an ability to make predictions, and also how do I identify people that are really good at making predictions and modeling the world? And so it sort of started up at some level to be a thing that would be of service to the Future of Life Institute, but also everybody else who’s thinking about the future and wants to do better planning and better decision-making.

Daniel Filan (02:54): How useful to FLI has Metaculus been?

Anthony Aguirre (02:57): Surprisingly little, I have found, actually. I think the big lesson that I’ve taken from Metaculus is that the ability to say whether… Once you’ve really carefully defined a question, like will X happen by date Y, and you know exactly what X is and you’ve defined it so that everybody agrees what X is and so on… Once you’ve just done that, whether that thing is 70% likely or 80% likely, nobody cares. It barely ever matters. Maybe if you were in something very quantitative, if you were working for a hedge fund or something, you care about 70 versus 80. But at some level, nobody cares. And whether it’s 70 or 80 doesn’t really change what you do in almost any decision.

(03:44): But the process of getting to the point of knowing exactly what X is and having a well-defined question and keeping track of who makes good predictions and who doesn’t, thinking about what is it that I want to decide or what decision do I want to take and what do I need to know in order to make that decision, and turning that into very concrete operational questions that can happen or not happen, those things are really valuable. So I think that the interesting pieces - some to FLI and some elsewhere - haven’t been so much the outputs, the actual predictions, but going through the process of making the questions: if this is what we really want to understand, how do we decompose that into well-defined things that can go on Metaculus? Almost independent of how those things actually turn out.

Daniel Filan (04:38): So you say 70% versus 80% doesn’t really matter. I imagine 10% versus 90% might matter. Does it ever come up that a thing turns out to be 90% and you thought it was 10% or vice versa?

Anthony Aguirre (04:50): Very occasionally. I mean, I would say it’s rare. I would say the Ukraine war was one where Metaculus had a much higher prediction [of war] than I think most people had and was right, was useful to some people. I think there were some people who actually moved out of Ukraine because of the Metaculus prediction. We felt…

Daniel Filan (05:09): That’s pretty cool.

Anthony Aguirre (05:09): …pretty good about that. It’s not many people who take Metaculus that seriously. But I think there are a few. So I think there are some… 1% versus 10% is also, of course, a huge difference, which is less appreciated among… if you’re not thinking about probabilities all the time. But it is also a huge difference, and I think there are some of those. I think a lot of it is also, once the reputation accrues, then it gets taken more seriously. So I think a lot of people in the AI community take seriously, at some level, Metaculus predictions about AGI arrival because Metaculus does have a track record.

(05:49): They know it’s a bunch of people that are high knowledge and thinking really carefully, technically about it, and it’s a way of aggregating lots of wisdom of the right sort of people. So I think the fact that there’s an AGI question that is at 2030 versus 2035 on Metaculus does make quite a big difference in the sort of things that AI safety is thinking about. Or a probability of 30% versus 70%, say, in human-level AGI by date X, is a reasonably big deal.

(06:28): I think there are some examples where the outputs really matter, but there are very few where I’ve said like, “oh, I think this is probably 90% probable,” and I put it into Metaculus, and Metaculus is like, “no, 10%.” Usually the numbers are not that surprising.

The Alignment Workshop

Daniel Filan (06:46): Fair enough. So before we get into the stuff you do at FLI, we’re currently at this alignment workshop being run by FAR.AI. How are you finding the workshop?

Anthony Aguirre (06:55): The workshop has been enjoyable in the way that these workshops are, which is that lots of people that I like and respect and want to talk to are here. It usually is less about the program than about the actual in-person physical gathering and who you can corner in various breaks. So from that perspective it’s been great, had great conversations.

FLI’s current activity

Daniel Filan (07:15): Awesome. So let’s talk about FLI. So you’re currently the executive director. At a high level, what’s FLI up to these days?

Anthony Aguirre (07:24): So FLI is always evolving. I think we started out primarily as a more academic group. We funded the first technical AI safety research grants back in 2015. We also were a convener, so we tried to bring together academics and non-profit people and industry people and a little bit policy makers and get them talking to each other before any of them were talking to each other.

Daniel Filan (07:54): Yeah. You ran the precursors to these alignment workshops. They were in nice tropical islands.

Anthony Aguirre (08:01): Yes.

Daniel Filan (08:01): I went to one of those and it was quite fun.

Anthony Aguirre (08:03): The Puerto Rico one, probably.

Daniel Filan (08:04): Yeah.

Anthony Aguirre (08:05): Well, there were two, but probably the second one. So we did a number of convenings partly to get technical people together, but also partly socially to get different groups together who weren’t talking together. Those groups are talking together a lot more now that they’re not in… They’re also talking together a lot less than they used to, in the sense that there were lots of heads of labs that you could get in together in one room and have them smiling and agreeing with each other in a way that doesn’t really happen nowadays, now that they’re sort of bitter rivals and at some level enemies, some of them. So that was a different time.

(08:42): For the last few years, I would say we’ve been focused both… Well, first, we’ve gotten more focused on AI as opposed to other transformative technologies. We still have a long history in nuclear, and we’re thinking about nuclear issues, but more where nuclear intersects AI or where bio intersects AI or where weaponry intersects AI. So AI is obviously the thing that is looming most quickly in all of our minds. So we’ve focused on that, I hope, I think appropriately. I mean, if it doesn’t turn out to be AI, it’ll be because there’s a quick pandemic first or a nuclear war first, and we’ll hope that that doesn’t happen.

Daniel Filan (09:20): Fingers crossed.

Anthony Aguirre (09:21): So in terms of AI, we’re doing some work still supporting technical research, a lot more work doing policy research and policy engagement and some level of advocacy. So we participated heavily in the EU AI Act, we participated in a number of sessions in the US Congress, gave testimony on Capitol Hill and part of the Schumer hearings. And so have basically had a presence of really trying to bring expertise about AI safety to policymakers in various forms.

(10:00): So there’s that policy expert and informative role. We’ve also taken on some level of policy advocacy role, I would say, especially starting with the Open Letter [on an AI pause] of early 2023, that was a bit more of a strong position that we took. We have taken those before. We’ve taken positions on autonomous weapons that we should basically not be developing them, we’ve taken positions on nuclear weapons, but those were, I would say, less directly contrary to the aims of the major companies that were doing things than the pause letter was. So I think since the pause letter, we’ve taken a little bit more of an advocacy role with a point of view about AI and AI risk, and pushed a little bit more for the things that we feel are needed, given the level of risk.

AI policy

Daniel Filan (11:04): So just at a high level, what do you think is needed? What policies do you advocate for?

Anthony Aguirre (11:11): Broadly speaking? What we’ve written formally and what you can find on our website, we believe that there should be ultimately a mandatory licensing system for AI systems at the frontier level. And that should require more and more stringent safety guarantees as those systems get more and more powerful and sort of zero for very lightweight systems or very narrow systems.

(11:38): We think that, at the moment, AGI is being developed in a rather unsafe race dynamic by large, un-overseen companies, and that this is not good. So we still call for a pause. We still think we should pause AGI development now until we have a much better system in place for managing the risks of it.

(12:07): And we also think that you should not develop superintelligence until or unless you can make really strong guarantees, like provable safety guarantees about the system, which are not possible to make now. So we should not be developing superintelligence any time soon until safety techniques are radically more capable than they are now.

(12:27): So those are like: baseline, we believe that. We are also quite concerned about issues of concentration of power, whether that power is concentrated in a handful of AI companies that have huge economic dominance, or a government that uses AI to suppress dissent and surveil and all of those things, or an AI itself. So in a sense of, not necessarily one AI taking over everything, but just lots of deferring decisions to a network of AI systems so that human decision-making and agency is largely lost.

Daniel Filan (13:06): Talking about the licensing thing first: you mentioned some sort of licensing scheme where for more powerful models, you have to do things to get a license. What are you imagining being the requirements for a license?

Anthony Aguirre (13:25): Yeah, I think, again, it should ratchet up as the systems become more powerful and potentially threatening. But I think for the systems we have now… So I would say it’s sort of the same thing that is happening. There should be evaluations of their capabilities and their risks, but it should be either done or checked by a disinterested third party, and there should be a stamp of approval on that before the system is actually deployed. So the evaluations are often done before the system is deployed, but if they found anything dangerous, it’s unclear what would happen at this point. So they should actually be required, they should actually have teeth, and they should involve an independent third party.

(14:15): And this is more or less what we do with every other product that might cause danger in our society. We do it with airplanes, we do it with cars, we do it with drugs. [With] everything else, you develop the thing, you make a case that the thing is safe at some reasonable and quantitatively-defined level, and then you roll it out to the consumer. So this is not some crazy new thing. It just feels weird because we’ve had an unbelievably unregulated tech space. And I think in many ways that has been fine. In some ways it’s been problematic. But once we’re now getting to systems that are actually potentially quite dangerous, we need to be adopting some of the techniques we developed for actually dangerous other systems.

Daniel Filan (15:00): So you mentioned in the limit of superintelligence, you would want some sort of guaranteed provable guarantees of safety. To me, this is reminiscent of Yoshua Bengio’s line of work: I guess [it] was “guaranteed safe AI”. There was some position paper. I’m afraid I didn’t actually read it, but have you been in contact with them, or collaborating on that?

Anthony Aguirre (15:23): Sure, yeah. So there’s a whole program that Max Tegmark has been pushing [that] Yoshua’s been involved in. Steve Omohundro has been pushing, Davidad has his own version of this. So I think there are a few very ambitious programs that say, okay, what would it look like to actually have safety when there’s a system that is potentially much more intelligent than us? That is not a problem that obviously even has a solution. At first blush when you say, “how do I control or ensure is safe for me, a system far more intelligent than me?”, the most obvious answer is “you don’t.” And if you have 10 kindergartners and they bring in a corporate CEO to help solve problems for them, there’s no sense in which those kindergartners are going to be controlling that CEO. The CEO is just more knowledgeable, more worldly, more wily, more persuasive, everything more effective than the 10 kindergartners. And so I think that’s a problem that is unsolvable for those kindergartners.

(16:32): Superintelligence may be an unsolvable problem in that same way, or it might not. I think we don’t know. So I think the requirement should be very high that we really believe that the problem is solved and we can really reassure ourselves that the problem is solved before we go ahead with that, because it’s not obvious that the problem is solvable, and I think we’re doing something rather unwise by going ahead and assuming that there will be a solution in time to superintelligent alignment or control when it’s not at all obvious that even in principle it’s possible, let alone that we know how to do it in practice.

Work FLI funds

Daniel Filan (17:09): I’m wondering… So maybe this relates to FLI’s role as a research funder. So you guys support a bunch of PhD fellowships, I guess you also run grants rounds. Is there a particular focus in the kinds of work that you want to fund, or is it more broad-based?

Anthony Aguirre (17:31): It’s pretty broad. And we decide on different things as priorities and then try to put both institutional resources and fiscal resources behind them. So I think the AI safety fellowships are part of the idea that we need to field-build in technical AI safety, and lots of people agree with that, and that’s our contribution to that. We just ran a request for proposals on concentration of power because we feel like that’s something that lots of people talk about and worry about, but aren’t really doing much about, certainly not at the research level. So, looking for things that aren’t necessarily going to happen by themselves and really do need independent or philanthropic dollars. So that’s an example of that.

(18:20): Others could be more niche, technical projects. So we’re funding things in compute security and governance now. Things that probably will come into being on their own, but probably much later than we would like them to. There, the idea is to accelerate the timeline for things that everybody agrees are good. Everybody agrees security is good, to a first approximation, but everybody also agrees that our security is [redacted] in most cases, so trying to make that better for high-powered AI. So it’s a mixture of different things, where we decide that there’s some thing that we see is underfunded but is important and just design some sort of program to do that.

Daniel Filan (19:03): One thing you said is that you support fiscally, as well as in some other way. I forget what exactly you said.

Anthony Aguirre (19:12): Yeah, so we give away grants, but we also do joint programs with things. Future of Life Institute also has a sister organization, Future of Life Foundation, the role of which is to incubate new organizations. So that will look like seed funding, but also looks like finding leadership and designing what the institution does and providing operational support in early days and things like that. Or we might have an organization that we’re collaborating with and we might help them out with communications or coordinate with them on media production or whatever. So that’s the sort of thing, just some of our institutional resources going to help some other projects that other people are doing.

Daniel Filan (19:58): Gotcha. Makes sense. And the Future of Life Foundation, am I right that that’s somewhat of a new organization?

Anthony Aguirre (20:03): That’s pretty new: it now has a staff of two, just recently doubled. So that’s just getting started, really. But it has sort of fully launched one thing which is CARMA. Don’t ask me what the acronym means because I will get it wrong, but it is a technical AI policy shop that Richard Mallah is leading.

(20:28): It has also taken over a project that was originally funded out of Future of Life Institute, now called Wise Ancestors, which is looking at non-human extinction and what we can do about that, and can we sort of back up some of the genetic data to the hard drive as well as prevent some things from going extinct. So that’s a little bit of a different angle on extinction and x-risk, but one that we found could be interesting and useful, and there’s a bunch of things in the hopper, but yeah, it’s just getting started.

Daniel Filan (21:00): Gotcha. If there’s some founder who is maybe interested in doing something and is curious what kinds of organizations you want to kickstart, is there a list on some website that they can look at?

Anthony Aguirre (21:12): There’s not a public list, but I would totally encourage them to get in touch with either myself or Josh Jacobson. Flf.org will give them the contact information for that. We’re eager to meet people who are excited about founding new organizations and would love to talk with them about what they’re thinking, what we’re thinking, if there’s an interesting match there.

Daniel Filan (21:32): Great. Well, thanks very much for chatting with me today.

Anthony Aguirre (21:35): Yeah, thanks for having me. It’s a pleasure.

Daniel Filan (21:36): This episode was edited by Kate Brunotts, and Amber Dawn Ace helped with transcription. The opening and closing themes are by Jack Garrett. Financial support for this episode was provided by the Long-Term Future Fund, along with patrons such as Alexey Malafeev. To read a transcript of the episode, or to learn how to support the podcasts yourself, you can visit axrp.net. Finally, if you have any feedback about this podcast, you can email me at feedback@axrp.net.