We at the MIRI Technical Governance Team have proposed an international agreement to halt the development of superintelligence until it can be done safely.

Some people object that, even if the agreement is adopted by some countries, ASI can still be developed in other countries that do not halt development. They argue that this would undermine the stability of the agreement and even prevent it from being adopted in the first place, because countries do not want to give up their lead only to be overtaken by other, less safety-conscious countries that are racing ahead in contravention of the pause.

TL;DR: An ASI nonproliferation agreement can be effective even without complete worldwide adoption. A coalition that controls advanced chips, fabs, and cloud access can deter, detect, and if needed, disrupt illicit R&D programs. Our proposed agreement couples domestic pauses with outward export controls, verification, and do‑not‑assist rules applicable to people and firms. Carrots—including access to supervised compute and safe applications—reduce incentives to free‑ride; sticks raise the cost of staying outside. The result is slower progress among unsanctioned ASI programs, higher odds of detecting dangerous training, and a stable path to widening participation.

Introduction

Yudkowsky and Soares’ 2025 book If Anyone Builds It, Everyone Dies argues that the development of advanced superintelligence, if done using the present-day methods of machine learning, would lead to the extinction of humanity. Superintelligence cannot be safely created before substantially more progress is made in the field of AI alignment, and before governance is improved to ensure it is not used for malicious purposes.

Therefore, we have proposed an international agreement to halt the development of superintelligent AI until it can be done safely. This agreement builds a coalition that makes unsanctioned frontier training anywhere infeasible by controlling chips, training, and research, while keeping legal pressure and technical options ready for targeted disruption of prohibited projects. As the countries leading in AI research, compute, and model capabilities, the U.S. and China would necessarily be key players, along with countries such as Taiwan and the Netherlands that control crucial links in the AI supply chain.

A common objection is that, even if the agreement is adopted by some countries, ASI can still be developed in other countries that do not halt research and development within their borders. This could undermine the stability of the agreement and even prevent it from being adopted in the first place, because countries concerned with extinction do not want to sacrifice their advantages only to be overtaken by other, less safety-conscious countries that are racing ahead in contravention of the pause.

In this post, I explain how the agreement anticipates and addresses this threat.

How non-party states could undermine the agreement

A single deployment of unaligned superintelligence would be the most dangerous event humanity has ever faced, more catastrophic than even nuclear war. If anyone builds it, everyone is at risk. It only takes one uncautious non-party AI project that crosses the line of extinction-level capabilities to nullify the caution and sacrifice of all party states.

Worse yet, under partial adoption, the most cautious and responsible actors slow down while the reckless race ahead, raising the odds of takeover, extinction, or war, which necessitates ever more invasive and destabilizing countermeasures. Without global coverage, a pause cannot fully eliminate the risk of an ASI-triggered catastrophe.

AI alignment is a very difficult problem that may take years to solve. A unilateral halt by the country leading in AI is most likely insufficient, because the country in second place will simply catch up and then continue development as before.

The most obvious way a non-party state can undermine the agreement is by intentionally developing ASI, but there are many other ways non-parties can create challenges for the coalition:

Non-party states add costs for party states

Even non-party states that ultimately pose no direct threat to the coalition’s agenda still burden it to some extent because they require costly monitoring. This disincentivizes would-be parties from joining and sharing in those costs, especially in the early days of the agreement, when there are more non-party states to monitor and fewer parties collaborating to monitor them. Smaller countries are the most likely to be hesitant about joining because of these expenses, as the costs are large relative to their economies, and they have less preexisting intelligence infrastructure of their own.

Non-party states raise the danger of covert projects in party states

The existence of non-signatory countries provides opportunities for party states to secretly collaborate with them (e.g. offshore their research to unmonitored areas), which heightens the risk of covert programs and unravels the agreement by undermining trust and cooperation.

Experienced researchers may relocate to non-party states

Talent flight exacerbates all these other problems. The population of researchers who can advance frontier AI is relatively small. If even a few of them relocate to non-party jurisdictions, ASI programs there gain disproportionate capability. Parties will rightly worry that unimpeded relocation of AI researchers would drain the talent pool of their home countries while providing dangerous expertise to non-party states.

Non-party progress toward ASI may peel off existing signatories

If a state believes it can reach superintelligence before other nations can stop it, it may decide to take the risk and dash for the finish line. If other nations become aware of this, they are likely to sabotage this unsanctioned AI project, but if they have their own computing hardware and are not certain the unsanctioned project can be stopped, they may abandon compliance with the agreement and join the race instead.

Non-party progress toward ASI necessitates more invasive controls

An AI model may be developed in a non-party state that is more capable than what is otherwise available in signatory states, which makes inference more dangerous.

Even if party states control most chip production, non-party states may continue R&D of computing hardware and AI algorithms, which lowers the size of clusters needed to advance the frontier of dangerous AI capabilities. At the extreme, there is some risk that algorithmic progress in non-party states can nullify the compute advantage of the coalition, though this would take many years of research by a compute-poor rogue state.

Free-rider dynamics sap domestic support in member states

Less safety-conscious firms in non-party states can undercut the profitability of firms in party states subject to compliance costs. Disadvantaged firms would be incentivized to lobby their governments against joining the coalition.

Firms in non-coalition countries will also lobby against joining because they will have strong incentives to push closer to the edge of ASI—they stand to gain huge profits if they succeed, but they are not on the hook for the damage caused by a misaligned superintelligence that kills all of humanity.

This reluctance to join the coalition could be contagious, as more responsible firms in would-be party states will be wary of competing with firms from outside the coalition which have the above mindset.

How the agreement combats these issues

The architecture of the agreement is designed to grow the coalition and hold it together while quickly shutting down prohibited ASI projects. It couples a domestic halt to ASI progress with an outward posture of nonproliferation toward other signatories and non-signatories. The ideal level of adherence to the agreement is at least equal to that of the Nuclear Non-Proliferation Treaty, and our recommended rollout phases include transparency, confidence-building, development of verification mechanisms, and gradual establishment of commitments, to bring in the international community and achieve this level of safety worldwide.

Incentives for non-parties to become parties

The best way to preempt problems raised by non-parties is to convert such states into parties. We recommend proverbial carrots and sticks to attract non-parties to join the coalition.

Membership in the agreement should be made economically worthwhile. A natural carrot here is access to the coalition’s computing clusters for use in socially beneficial AI applications. Lower-income countries should also be supported with technical assistance and cost-sharing by the wealthier Parties. We envision major economies, including the U.S. and China, joining the agreement early on, and these founding parties can lay much of the groundwork of monitoring in the early days when this task is costliest. This reduces the barrier to entry for smaller countries that join later.

Remaining outside the agreement should be economically costly. One straightforward stick is import bans on AI services developed in non-party countries, which might otherwise undercut the profitability of coalition firms by cutting corners on safety and monitoring. Non‑parties are also denied access to supervised chips and manufacturing capabilities (Arts. I, VI). They will face restrictions on inputs and raw materials for the AI supply chain, disruption or sabotage of their prohibited research activities, and trade restrictions such as blacklisting of AI services developed in those countries.

North Korea and Iran are good historical examples of how the international community can deal with countries that try to acquire dangerous technology in contravention of worldwide norms. North Korea withdrew from the Non-Proliferation Treaty and developed nuclear weapons, and was hit with crippling sanctions from most of the world in response. Iran is attempting to develop nuclear weapons despite being an NPT signatory state, so its nuclear program has repeatedly been sabotaged by those who fear the consequences in the Middle East if it were to acquire them. The situation with AI development is analogous in many ways and different in others. We don’t know all the specific methods and incentives that will ultimately be applied, but we expect that parties will encourage others to join, as was the case for nuclear non-proliferation.

For reasons that will become clear in subsequent sections, it is vital that countries currently possessing very large numbers of advanced AI chips join the coalition. This can be done through inducements by the leading Parties: most countries with many advanced chips are very dependent on trade and technological exchange with the U.S., China, or both, meaning that economic incentives and access to AI technology from these Parties will be a major incentive. If these are not sufficient, standard tools of diplomatic pressure—such as trade restrictions, economic sanctions, and visa bans—should be applied (Art. XII).

Export and production controls

Many of the potential challenges posed by non-parties are moot if they simply don’t have the chips.

The agreement’s export and production controls deny non‑parties access to AI chips, as well as the specialized manufacturing equipment that bottlenecks the chip supply chain (Arts. I, V–VII). Non-states lacking such chips in large quantities would be unable to complete training runs of frontier models on reasonable time scales, and would not be able to operate existing high-end models in a cost-competitive manner.

Concretely, the AI chip supply chain is extremely narrow, and the chips are so complex that it is extremely difficult for anyone outside of the existing chain to replicate it. The vast majority of chips are designed by U.S. companies, mostly NVIDIA. The processors are almost all fabricated by TSMC, on their 4-6nm process nodes in only a couple of plants. Extreme ultraviolet photolithography machines, essential equipment for etching sufficiently small transistors to make chips as powerful and efficient as today’s, are made exclusively by the Dutch firm ASML. The market for high-bandwidth memory, another crucial component, is dominated by SK Hynix, Samsung, and Micron, which are Korean and American companies. China is trying to replicate each of these steps, but progress is slow due to the extreme complexity of modern computing hardware. If the coalition were to include China, the U.S., and their close allies, it would be virtually impossible for any outside country to produce state-of-the-art AI hardware.

Because chips break down with use and become obsolete with age, non-parties can expect their stock of strategically and commercially relevant AI hardware to gradually diminish over time. This burnout, in conjunction with export and production controls on non-party states, and monitoring of newly produced chips (Art. VI), means that the agreement should eventually cover a large majority of the relevant hardware.

Chip centralization and chip-use verification

The coalition, by requiring each member to consolidate chips into domestically controlled Covered Chip Clusters (CCCs) and then verifying how they are used, not only ensures that parties aren’t pursuing ASI projects, but also that non-parties are not doing such work on the coalition’s chips through remote access (Arts. V–VII).

When the number of AI chips outside the coalition is small, the potential for unmonitored training by anyone, anywhere is greatly reduced.

Research controls, whistleblower protection, and intelligence collection

The agreement’s restrictions on research that advances towards ASI or potentially undermines verification include “do‑not‑assist” obligations that travel with people and organizations. (Art. VIII–IX) These obligations reduce the surface for algorithmic workarounds and “offshored” labs (Arts. VIII–IX) by making it unlawful for Parties and their persons to fund, host, supervise, teach, publish, equip, or otherwise enable Banned Research anywhere (Art. VIII §4).

Research leading to dangerous AI capabilities must be clearly flagged as off-limits everywhere, and the coalition must use its diplomatic and economic influence to dissuade non-parties from such work just as it disincentivizes large AI training runs.

As is the case for nuclear research, individuals conducting dangerous AI research need to know they risk being arrested if they are undertaking dangerous AI research that crosses red lines declared by party states, even if they try to evade the research restrictions by moving to a non-party state. AI researchers with sensitive industry knowledge should be discouraged from traveling to rogue states—something the U.S. government already does for employees of important scientific research offices.

The agreement’s complementary Research Restriction Verification tools—including interviews with researchers and embedded auditors at selected organizations—deter covert relocation schemes (Art. IX §2). These are paired with whistleblower protection and asylum for those who expose violations (Art. X). Finally, the coalition would provide funded placements into safer research activities for former AI developers, which would also make it more expensive for rogue states to hire them.

Intelligence collection on non-compliant states and intel-sharing collaboration among the coalition fuses parties’ national technical means with whistleblower reports and enables challenge inspections on 24‑hour timelines when credible concerns arise (Art. X).

Protective actions

Under the protective actions protocols defined in the agreement, any actor—party or non-party—credibly moving towards ASI is subject to gradually escalating protective actions (sanctions → cyber interdiction → hardware seizure or disablement). Models, hardware, and development programs can be targeted for observation and, in extreme cases, destruction, with due measures taken to minimize the scope and collateral harm of any protective operations. (Arts. XI–XII)

Review and revision of threshold and definitions

The agreement provides mechanisms for revision of thresholds and definitions as the technical frontier shifts (Arts. III(d), XIII–XIV), as could happen due to ongoing developments in non-party states. These mechanisms include an allowance for the CTB to immediately adjust FLOP thresholds, CCC size, and Restricted Research boundaries when inaction poses security risk (effective for 30 days unless the Council objects).

Withdrawal guardrails

States wanting to leave the coalition present an outsized risk of pursuing dangerous ASI development, since the coalition is expected to include the countries holding most of the world’s AI hardware. Under the agreement, any withdrawing state must, over 12 months, help neutralize its CCCs and ASI‑enabling assets; after withdrawal, it remains subject to Protective Actions if it races for ASI (Art. XV).

Conclusion

Partial adoption is not a deal-breaker. A coalition that controls chip manufacturing equipment, monitors large clusters, and verifies chip usage can deter, detect, and disrupt dangerous AI development. By design, the agreement not only prevents dangerous R&D within member states and makes it difficult elsewhere, its phased rollout builds international trust and provides a stable pathway for full participation.

The chip consolidation and use verification, export/production controls, challenge inspections, and graduated ladder of protective actions make the option of withholding from the agreement and developing ASI on your own a visible, difficult, losing bet. On the other hand, joining not only reduces AI risk but offers economic benefits such as access to the coalition’s compute resources and beneficial AI technology.

One of the greatest challenges is that the longer we wait to implement this agreement, the harder it will be to prevent rogue ASI research due to algorithmic and hardware progress. Our recommendation is simple: start building the coalition now, and keep the door open to all those who wish to join in averting the risk of an AI catastrophe.

If you are interested in this work and want to join our team, we are hiring researchers.