I recommend reading Scalable AI control before reading this post.
In particular, in the "Hard values, easy values" section of this post, Paul writes:
My statement of the control problem is only really meaningful because there are instrumental subgoals that are shared (or are extremely similar) between many different values, which let us compare the efficacy with which agents pursue those different values. Performance on these very similar subgoals should be used as the performance metric when interpreting my definition of AI control problem.
In fact even if we only resolved the problem for the similar-subgoals case, it would be pretty good news for AI safety. Catastrophic scenarios are mostly caused by our AI systems failing to effectively pursue convergent instrumental subgoals on our behalf, and these subgoals are by definition shared by a broad range of values.
Convergent instrumental subgoals are mostly about gaining power. For example, gaining money is a convergent instrumental subgoal. If some individual (human or AI) has convergent instrumental subgoals pursued well on their behalf, they will gain power. If the most effective convergent instrumental subgoal pursuit is directed towards giving humans more power (rather than giving alien AI values more power), then humans will remain in control of a high percentage of power in the world.
If the world is not severely damaged in a way that prevents any agent (human or AI) from eventually colonizing space (e.g. severe nuclear winter), then the percentage of the cosmic endowment that humans have access to will be roughly close to to the percentage of power that humans have control of at the time of space colonization. So the most relevant factors for the composition of the universe are (a) whether anyone at all can take advantage of the cosmic endowment, and (b) the long-term balance of power between different agents (humans and AIs).
I expect that ensuring that the long-term balance of power favors humans constitutes most of the AI alignment problem, and that other parts of the AI alignment problem (e.g. ensuring AIs are beneficial in the short term, ensuring that AI systems don't cause global catastrophic risks that cause the cosmic endowment to become unavailable to any agent) will be easier to solve after thinking about this part of the problem. So I'm going to focus on power acquisition for now.
Priors and multidimensional power
Convergent instrumental subgoals aren't totally convergent, since power is multidimensional, and some types of power are more useful for some values.
Suppose the sun is either going to turn green or blue in 10 years. No one knows which color it will turn; people disagree, and it seems like their beliefs about the sun are irreconcilable because they result from different priors. The people who predict the sun will turn green (or equivalently, care more about futures in which the sun is green) buy more green-absorbing solar panels, while those who predict the sun will turn blue will buy more blue-absorbing solar panels. How could we measure how much power different people have?
In situations like this, it seems wrong to reduce power to a single scalar; there are at least 2 scalars involved in this situation (how much power someone has in futures where the sun turns green, versus in futures where the sun turns blue).
For the AI to gain power on the user's behalf, it should gain the kind of power the user cares about. If the user thinks the sun will turn green, then the AI should buy green-absorbing solar panels.
What if the user hasn't made up their mind about which color the sun will be, or it's hard for the AI to elicit the user's beliefs for some other reason? Then the AI could pursue a conservative strategy, in which the user does not lose power in either possible world. In the case of solar panels, if 60% of the solar panels that everyone buys absorb green, then the AI should invest 60% of the user's solar panel budget in green-absorbing solar panels and 40% in blue-absorbing solar panels. This way, the user generates the same percentage of the energy in each possible world, and thus has the same amount of relative power. This is suboptimal compared to if the user had more defined beliefs about the sun, but the user isn't any worse off than they were before, so this seems fine.
I think this is an important observation! It means that it isn't always necessary for an AI system to have good priors about hard-to-verify facts (such as the eventual color of the sun), as long as it's possible to estimate the effective priors of the agents who the user wants to be competitive with. In particular, if there is some "benchmark" unaligned AI system, and it is possible to determine that AI system's effective prior over facts like the color of the sun, then it should be possible to build an aligned AI system to use a similar prior and thereby be competitive with the unaligned AI system in all possible futures.
This doesn't only apply to priors, it also applies to things like discount rates (which are kind of like "priors about which times matter") and preferences about which parts of the universe are best to colonize (which are kind of like "priors about which locations matter"). In general, it seems like "estimating what types of power a benchmark system will try acquiring and then designing an aligned AI system that acquires the same types of power for the user" is a general strategy for making an aligned AI system that is competitive with a benchmark unaligned AI system.