This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

Additional thanks to Jamie Bernardi, Oly Sourbut and Shawn Hu, for their thoughts and feedback on this post.

TL;DR

The Natural Abstraction Hypothesis, proposed by John Wentworth, states that there exist abstractions (relatively high-dimensional summaries which capture information relevant for prediction) which are "natural" in the sense that we should expect a wide variety of cognitive systems to converge on using them.

If this hypothesis (which I will refer to as NAH) is true in a very strong sense, then it might be the case that we get "alignment by default", where an unsupervised learner finds a simple embedding of human values, and a supervised learner can be trained with access to this learned model. However, even if the NAH is true in a weaker sense, this has important implications, for areas such as:

Deceptive alignment - since a natural abstraction will be easier for an objective to point at
Instrumental convergence - since as I will discuss below, the formalisation for an abstraction resembles the idea of instrumental goals
Interpretability - since the extent to which human-interpretable abstractions are also "natural" has direct implications for how much of a trade-off incentivising interpretability will be

Several different fields can provide evidence for & against different forms of the NAH, for instance:

Studying the abstractions formed by different cultures can help us understand whether humans generally seem to converge on the same abstractions
Current empirical study of neural networks, primarily in the context of interpretability research, can help us understand whether they are forming human-interpretable abstractions

Since the NAH is primarily an empirical claim about the kinds of abstractions we expect to see forming in the real world, I will conclude by discussing some promising ways the hypothesis could be tested. This will include John Wentworth's own research agenda (which involves a feedback loop of training cognitive models on systems that contain abstractions, and examining then proving theorems about which abstractions are learned by which agents in which environments), as well as two other potentially promising directions: further interpretability work, and neuro...

Posts

Wiki Contributions

Comments