Random Tag
Contributors
You are viewing revision 0.0.2, last edited by Issa Rice

Optimization daemons are optimizers that result from heavy optimization pressure on a different system. For example, natural selection is an optimization process (that optimizes for reproductive fitness) that produced humans (who are capable of pursuing goals that no longer correlate reliably with reproductive fitness). In this case, humans are optimization daemons of natural selection. In the context of AI alignment, the concern is that an artificial general intelligence exerting optimization pressure may produce daemons that break alignment.1

History

Wei Dai brings up a similar idea in an SL4 thread.2

The optimization daemons article on Arbital was published probably in 2016.3...

(Read More)