If you haven't already, take a look at this post by johnswentworth to understand what this is all about: https://www.lesswrong.com/posts/KnPN7ett8RszE79PH/demons-in-imperfect-search

The short version is that while systems that use perfect search, such as AIXI, have many safety problems, a whole new set of problems arises when we start creating systems that are not perfect searchers. Patterns can form that exploit the imperfect nature of the search function to perpetuate themselves. johnswentworth refers to such patterns as "demons".

After reading that post I decided to see if I could observe demon formation in a simple model: gradient descent on a not-too-complicated mathematical function. It turns out that even in this very simplistic case, demon formation can happen. Hopefully this post will give people an example of demon formation where the mechanism is simple and easy to visualize.

Model

The function we try to minimize using gradient descent is called the loss function. Here it is:

$L (\to x) = - x_{0} + ϵ n \sum j = 1 x_{j} \cdot {splotch}_{j} (\to x)$

Let me explain what some of the parts of this loss mean. Each function ${splotch}_{j} (\to x)$ is periodic with period 2π in every component of $\to x$ . I decided in this case to make my splotch functions out of a few randomly chosen sine waves added together.

$ϵ$ is chosen to be a small number so in any local region, $ϵ \sum_{j = 1}^{n} x_{j} \cdot {splotch}_{j} (\to x)$ will look approximately periodic: A bunch of hills repeating over and over again with period 2π across the landscape. But over large enough distances, the relative weightings of various splotches do change. Travel a distance of 20π in the $x_{7}$ direction, and ${splotch}_{7}$ will be a larger component of the repeating pattern than it was before. This allows for selection effects.

The $- x_{0}$ term means that the vector $\to x$ mainly wants to increase its $x_{0}$ component. But the splotch functions can also direct its motion. A splotch function might have a kind of ridge that directs some of the $x_{0}$ motion into other components. If ${splotch}_{7}$ tends to direct motion in such a way that $x_{7}$ , increases, then it will be selected for, becoming stronger and stronger as time goes on.

Results

I used ordinary gradient descent, with a constant step size, and with a bit of random noise added in. Figure 1 shows the value of x0 as a function of time, while figure 2 shows the values of x1,x2,…x16 as a function of time.

Fig 1:

Fig 2:

There are three phases to the evolution: In t

...

Programming