Some problems to work on regarding goal-directed intelligence. Conjecture 5 is especially important for deconfusing basic questions in alignment, as it stands in opposition to Stuart Armstrong's thesis about the impossibility to deduce preferences from behavior alone.

1. Conjecture. Informally: It is unlikely to produce intelligence by chance. Formally: Denote $\Pi$ the space of deterministic policies, and consider some $\mu \in \Delta \Pi$. Suppose $\mu$ is equivalent to a stochastic policy ${\pi }^{\ast }$. Then, $E_{\pi \sim \mu }\left[g\right(\pi \left)\right]=O\left(C\right({\pi }^{\ast }\left)\right)$.

2. Find an "intelligence hierarchy theorem". That is, find an increasing sequence $\left\{g_n\right\}$ s.t. for every $n$, there is a policy with goal-directed intelligence in $(g_n,g_{n+1})$ (no more and no less).

3. What is the computational complexity of evaluating $g$ given (i) oracle access to the policy or (ii) description of the policy as a program or automaton?

4. What is the computational complexity of producing a policy with given $g$?

5. Conjecture. Informally: Intelligent agents have well defined priors and utility functions. Formally: For every $(U,\zeta )$ with $C\left(U\right)<\infty$ and $D_{KL}\left({\zeta }_0||\zeta \right)<\infty$, and every $ϵ>0$, there exists $g\in (0,\infty )$ s.t. for every policy $\pi$ with intelligence at least $g$ w.r.t. $(U,\zeta )$, and every $(\tilde{U},\tilde{\zeta })$ s.t. $\pi$ has intelligence at least $g$ w.r.t. them, any optimal policies ${\pi }^{\ast },{\tilde{\pi }}^{\ast }$ for $(U,\zeta )$ and $(\tilde{U},\tilde{\zeta })$ respectively satisfy $E_{\zeta {\tilde{\pi }}^{\ast }}\left[U\right]\ge E_{\zeta {\pi }^{\ast }}\left[U\right]-ϵ$.