This post is a follow-up to What if Alignment is Not Enough?, which summarized the core arguments of Substrate Needs Convergence. This post further explores the intuitions and worldview driving this theory, particularly the aspects of it that are the most contentious.

Executive Summary: It is not enough to consider what a system does, but also how its actions affect the world, including how those effects change the system itself, up to the point of stable equilibrium. At the scale of AGI, unintended consequences are inevitable, which gives evolutionary forces room to work, making growth of the system an implied attractor state, eventually overriding conflicting initial goals, such as compatibility with human survival. The only way to be safe from AGI is to not build it.

Terminology

SNC = Substrate Needs Convergence. The theory that AGI will necessarily destroy all biological life as a result of evolution-like pressures.
AGI = Artificial General Intelligence. Highly autonomous systems that outperform humans at most economically valuable work. SNC is primarily concerned about AI at this level.
ASI = Artificial Super Intelligence. AI that possesses the theoretical upper limit of cognitive ability, but is still bounded by physical laws. The fate of the world will likely be sealed before AI reaches this level, but the concept is nonetheless useful as a simplifying abstraction for thought experiments.

Challenging the implicit frame of the super-optimizer

AI safety is a novel field. As such, it is premature to characterize any positive viewpoint as representing a consensus. That said, based on the comments of the prior post in this series and responses to SNC more generally, there seems to be a set of consistent underlying beliefs among the people who engage with SNC enough to comment, but reject the conclusions. I venture to characterize this belief as being that ASI will behave as a persistent super-optimizer. That is, whatever world state is implied by the ASI’s initial starting values is a reasonable approximation of what the world state will be.

The super-optimizer frame admits that the value specification of an ASI is not the sum total of the values it will act on. The theory of Instrumental Convergence suggests that future AI will be motivated by instrumental needs such ...

Posts

Wikitag Contributions

Comments