I'm giving this +1 review point despite not having originally been excited about this in 2024. Last year, I and many others were in a frame where alignment plausibly needed a brilliant idea. But since then, I've realized that execution and iteration on ideas we already have is highly valuable. Just look at how much has been done with probes and steering!

Ideas like this didn't match my mental picture of the "solution to alignment", and I still don't think it's in my top 5 directions, but with how fast AI safety has been growing, we can assign 10 researchers to each of 20 "neglected approach"es like this, so it deserves +1 point.

The post has an empirical result that's sufficient to concretize the idea and show it has some level of validity, which is necessary. Adam Jones has a critique. However, the only paper on this so far didn't make it to a main conference and only has 3 cites, so the impact isn't large (yet).

I didn't believe the theory of change at the time and still don't. The post doesn't really make a full case for it, and I doubt it really convinced anyone to work on this for the right reasons.

• No BOTEC
• Models of social dynamics are handwavy
• In alignment and other safety work, we have force multipliers like neglectedness and influencing government. What's the multiplier here? Or is there no intervention that has multiplier effects in the long-term multipolar risk space?
• Why wouldn't the effect just get swamped by the dozens of medical AI startups that are raising billions of dollars as we speak?

It may do a good job of giving the author's perspective, but given all these gaps it's not very memorable today in explaining the risks OP cares about-- even if we do end up worrying about them in 5-10 years.

To clarify I'm not very confident that AI will be aligned; I still have a >5% p(takeover doom | 10% of AI investment is spent on safety). I'm not really sure why it feels different emotionally but I guess this is just how brains are sometimes.

I'm glad to see this post come out. I've previously opined that solving these kinds of problems is what proves a field has become paradigmatic:

Paradigms gain their status because they are more successful than their competitors in solving a few problems that the group of practitioners has come to recognize as acute. ––Thomas Kuhn

It has been proven many times across scientific fields that a method that can solve these proxy tasks is more likely to achieve an application. The approaches sketched out here seem like a particularly good fit for a large lab like GDM, because the North Star can be somewhat legible and the team has enough resources to tackle a series of proxy tasks that are relevant and impressive. Not that it would be a bad fit elsewhere either.

Seems difficult for three reasons:

1. Many other companies are trying to build an AGI so a company trying to do this would have to first win the race and disempower everyone else; this means they will have fewer resources for safety
2. The AI will have to act antagonistically to humanity, so it could not be at all corrigible
3. The AI will need to act correctly until 2125, which is much farther out of distribution than any AI we have observed

Given these difficulties I'd put it below 50/50, but this challenge seems significantly harder than the one I think we will actually face, which is more like having an AI we can defer to that doesn't try to sabotage the next stage of AI research, for each stage until the capability level that COULD disempower 2025 humanity, plus maybe other things to keep the balance of power stable.

Also I'm not sure what "drawn from the same distribution" means here, AI safety is trying dozens of theoretical and empirical directions, plus red teaming and model organisms can get much more elaborate with impractical resource investments, so things will look very qualitatively different in a couple of decades.

Thanks for writing this, I've suspected for a while that we're ahead, which is great but a bit emotionally difficult when I'd spent basically my whole career with the goal of heroically solving an almost impossible problem. And this is a common view among AI safety people, e.g. Ethan Perez said recently he's focused on problems other than takeover due to not being as worried about it.

I do expect some instrumental pressure towards misaligned power-seeking, but the number of tools we have to understand, detect, and prevent it is now large enough that it seems we'll be fine until the Dyson sphere is under construction, at which point things are a lot more uncertain but probably we'll figure something out there too.

Agree that your research didn't make this mistake, and MIRI didn't make all the same mistakes as OpenAI. I was responding in context of Wei Dai's OP about the early AI safety field. At that time, MIRI was absolutely being uncooperative: their research was closed, they didn't trust anyone else to build ASI, and their plan would end in a pivotal act that probably disempowers some world governments and possibly ends up with them taking over the world. Plus they descended from a org whose goal was to build ASI before Eliezer realized alignment should be the focus. Critch complained as late as 2022 that if there were two copies of MIRI, they wouldn't even cooperate with each other.

It's great that we have the FLI statement now. Maybe if MIRI had put more work into governance we could have gotten it a year or two earlier, but it took until Hendrycks got involved for the public statements to start.

We absolutely do need to "race to build a Friendly AI before someone builds an unFriendly AI". Yes, we should also try to ban Unfriendly AI, but there is no contradiction between the two. Plans are allowed (and even encouraged) to involve multiple parallel efforts and disjunctive paths to success.

Disagree, the fact that there needs to be a friendly AI before an unfriendly AI doesn't mean building it should be plan A, or that we should race to do it. It's the same mistake OpenAI made when they let their mission drift from "ensure that artificial general intelligence benefits all of humanity" to being the ones who build an AGI that benefits all of humanity.

Plan A means it would deserve more resources than any other path, like influencing people by various means to build FAI instead of UFAI.

Also mistakes, from my point of view anyway

• Attracting mathy types rather than engineer types, resulting in early MIRI focusing on less relevant subproblems like decision theory, rather than trying lots of mathematical abstractions that might be useful (e.g. maybe there could have been lots of work on causal influence diagrams earlier). I have heard that decision theory was prioritized because of available researchers, not just importance.
• A cultural focus on solving the full "alignment problem" rather than various other problems Eliezer also thought to be important (eg low impact), and lack of a viable roadmap with intermediate steps to aim for. Being bottlenecked on deconfusion is just cope, better research taste would either generate a better plan or realize that certain key steps are waiting for better AIs to experiment on
• Focus on slowing down capabilities in the immediate term (e.g. plans to pay ai researchers to keep their work private) rather than investing in safety and building political will for an eventual pause if needed

The Interwebs seem to indicate that that's only if you give it a laser spot to aim at, not with just GPS.

Good catch.

Agree grenade sized munitions won't damage buildings, I think the conversation is drifting between FPVs and other kinds of drones, and also between various settings, so I'll just state my beliefs.

• Fiber FPVs with 40km range and 2kg payload (either kamikaze or grenade/mine dropping), which can eventually be countered by a large number of short range guns if they fly at low altitude. It's not clear to me if the 40km range ones need to be larger
• Heavy bomber drones can be equipped with fiber (or Starlink for US allies) and carry 15kg+ payload, enough to damage buildings and sensitive industrial equipment. They can do this while flying above the range of small guns and need dedicated antiaircraft guns
• Fixed wing can carry even larger payloads with longer range and higher altitude, but are still pretty slow, except for the ones with jet engines
• Drones equipped with GPS will know their position to within ~10 meters like the GPS only variant of Excalibur. It seems possible to constrain the payload's horizontal velocity by 1 m/s on average, and the drop time from 1500m is 17 seconds, giving an error of 17 m. The overall error would be sqrt(10^2 + 17^2) = 20 m. If GPS is jammed, it's not obvious they can do the first part, but probably they can still use cameras or something
• All of the above are extremely threatening for both organized warfare and terrorism against an opponent without effective cheap air defense.
• Even with the next evolution of air defense including radar-acoustic fusion to find threats, the limited reliability of ~all types of existing air defense and large number of drone configurations makes me guess that drones will remain moderately threatening in some form. Given that Hezbollah was previously firing unguided rockets with CEP in the hundreds of meters, some kind of drone that can target with CEP around 20 meters could be more cost effective for them if they cannot procure thousands of cheap guided missiles. If they could drop six individual grenades on six people from a bomber drone even in the presence of air defense, that would be even more effective, but it seems unlikely
• Excalibur is made by the US, which has no incentive to reduce costs, and so its $70k price tag is more of a "maximum the army is willing to pay" situation. This is true to some extent with Skyranger so maybe someone motivated will build smart ammunition that costs $40 per round and make it cost effective.