All Posts
Saturday, September 26th 2020Sat, Sep 26th 2020
Saturday, September 26th 2020
Sat, Sep 26th 2020
No posts for September 26th 2020
Shortform
8Vanessa Kosoy2dAn AI progress scenario which seems possible and which I haven't seen discussed:
an imitation plateau.
The key observation is, imitation learning algorithms[1]
[#fn-CnejrBKHaCL3cWxrN-1] might produce close-to-human-level intelligence even
if they are missing important ingredients of general intelligence that humans
have. That's because imitation might be a qualitatively easier task than general
RL. For example, given enough computing power, a human mind becomes realizable
from the perspective of the learning algorithm, while the world-at-large is
still far from realizable. So, an algorithm that only performs well in the
realizable setting can learn to imitate a human mind, and thereby indirectly
produce reasoning that works in non-realizable settings as well. Of course,
literally emulating a human brain is still computationally formidable, but there
might be middle scenarios where the learning algorithm is able to produce a
good-enough-in-practice imitation of systems that are not too complex.
This opens the possibility that close-to-human-level AI will arrive while we're
still missing key algorithmic insights to produce general intelligence directly.
Such AI would not be easily scalable to superhuman. Nevertheless, some
superhuman performance might be produced by sped-up simulation, reducing noise
in human behavior and controlling the initial conditions (e.g. simulating a
human on a good day). As a result, we will have some period of time during which
AGI is already here, automation is in full swing, but there's little or no
further escalation. At the end of this period, the missing ingredients will be
assembled (maybe with the help of AI researchers) and superhuman AI (possibly a
fast takeoff) begins.
It's interesting to try and work out the consequences of such a scenario, and
the implications on AI strategy.
--------------------------------------------------------------------------------
1. Such as GPT-n ↩︎ [#fnref-CnejrBKHaCL3cWxrN-1]
12dReasoning about learned policies via formal theorems on the power-seeking
incentives of optimal policies
One way instrumental subgoals might arise in actual learned policies: we train a
proto-AGI reinforcement learning agent with a curriculum including a variety of
small subtasks. The current theorems show sufficient conditions for
power-seeking tending to be optimal in fully-observable environments; many
environments meet these sufficient conditions; optimal policies aren't hard to
compute for the subtasks. One highly transferable heuristic would therefore be
to gain power in new environments, and then figure out what to do for the
specific goal at hand. This may or may not take the form of an explicit
mesa-objective embedded in e.g. the policy network.
Later, the heuristic has the agent seek power for the "real world" environment.
(Optimal Farsighted Agents Tend to Seek Power [https://arxiv.org/abs/1912.01683]
is rather dated and will be updated soon.)