All Posts

Sorted by Magic (New & Upvoted)

Monday, September 28th 2020
Mon, Sep 28th 2020

No posts for September 28th 2020

Saturday, September 26th 2020
Sat, Sep 26th 2020

No posts for September 26th 2020
Shortform
8Vanessa Kosoy2dAn AI progress scenario which seems possible and which I haven't seen discussed: an imitation plateau. The key observation is, imitation learning algorithms[1] [#fn-CnejrBKHaCL3cWxrN-1] might produce close-to-human-level intelligence even if they are missing important ingredients of general intelligence that humans have. That's because imitation might be a qualitatively easier task than general RL. For example, given enough computing power, a human mind becomes realizable from the perspective of the learning algorithm, while the world-at-large is still far from realizable. So, an algorithm that only performs well in the realizable setting can learn to imitate a human mind, and thereby indirectly produce reasoning that works in non-realizable settings as well. Of course, literally emulating a human brain is still computationally formidable, but there might be middle scenarios where the learning algorithm is able to produce a good-enough-in-practice imitation of systems that are not too complex. This opens the possibility that close-to-human-level AI will arrive while we're still missing key algorithmic insights to produce general intelligence directly. Such AI would not be easily scalable to superhuman. Nevertheless, some superhuman performance might be produced by sped-up simulation, reducing noise in human behavior and controlling the initial conditions (e.g. simulating a human on a good day). As a result, we will have some period of time during which AGI is already here, automation is in full swing, but there's little or no further escalation. At the end of this period, the missing ingredients will be assembled (maybe with the help of AI researchers) and superhuman AI (possibly a fast takeoff) begins. It's interesting to try and work out the consequences of such a scenario, and the implications on AI strategy. -------------------------------------------------------------------------------- 1. Such as GPT-n ↩︎ [#fnref-CnejrBKHaCL3cWxrN-1]
1Alex Turner2dReasoning about learned policies via formal theorems on the power-seeking incentives of optimal policies One way instrumental subgoals might arise in actual learned policies: we train a proto-AGI reinforcement learning agent with a curriculum including a variety of small subtasks. The current theorems show sufficient conditions for power-seeking tending to be optimal in fully-observable environments; many environments meet these sufficient conditions; optimal policies aren't hard to compute for the subtasks. One highly transferable heuristic would therefore be to gain power in new environments, and then figure out what to do for the specific goal at hand. This may or may not take the form of an explicit mesa-objective embedded in e.g. the policy network. Later, the heuristic has the agent seek power for the "real world" environment. (Optimal Farsighted Agents Tend to Seek Power [https://arxiv.org/abs/1912.01683] is rather dated and will be updated soon.)

Friday, September 25th 2020
Fri, Sep 25th 2020

No posts for September 25th 2020

Thursday, September 24th 2020
Thu, Sep 24th 2020

No posts for September 24th 2020

Load More Days