To be an Artificial General Intelligence (AGI) an AI needs to meet or exceed skilled human performance for most tasks, specifically including Science/Technology/Engineering/Mathematics (STEM) research. I am here ignoring the possibility of building an existentially-dangerous AI that isn’t capable of doing STEM work: certainly it is hard to see how an AI could go Foom without that capability. So this analysis doesn’t cover, for example, AIs that are dangerously superhuman only at manipulating humans, but are not an AGI.

I am only going to consider AGIs that are value learners: i.e. they attempt to simultaneously learn to predict human preferences, and meanwhile act in light of their current understanding of human preferences and aim to produce outcomes that they are confident that (on some average or aggregate) humans will prefer. If we build a STEM-capable AGI that isn’t a value learner, and instead has any hardwired utility function that was written by humans, then I confidently predict Doom — let’s not do that!

Clearly a major convergent instrumental strategy for any value learner is to understand human preferences: the better the AI understands these, the more likely it is to be able to get them right and do the right thing. Relevant knowledge is power. So doing Alignment Research is a convergent instrumental strategy for any STEM-capable AGI value learner — exactly as we would want it to be.

Required Capabilities for STEM Work

In order to do STEM, such a AGI needs to be able to use the scientific method. That is to say, it needs to be able to hold in its mind multiple conflicting hypotheses about or models of the world, with uncertainty as to which is correct, and perform something approximating Bayesian updates on them. Note that, unlike AIXI, any real AGI will not have unlimited computational resources, so it cannot use a universal prior for its Bayesian priors — it will need to instead use fallible informed guesses of priors, and rely on the approximately-Bayesian updates to converge these guesses towards 100% for the best hypothesis and 0% for competing ones. Fortunately after a few good Bayesian updates, the result tends to be only weakly dependent on the prior, so this uncertainty tends to decrease.

Our STEM-capable AGI needs to be creative enough able to create new hypotheses — genetic algorithms and LLMs already show significant signs of m...

Entropy