Effective layer horizon of transformer circuits. The residual stream norm grows exponentially over the forward pass, with a growth rate of about 1.05. Consider the residual stream at layer 0, with norm (say) of 100. Suppose the MLP heads at layer 0 have outputs of norm (say) 5. Then after 30 layers, the residual stream norm will be $100\cdot {1.05}^{30}\approx 432.2$. Then the MLP-0 outputs of norm 5 should have a significantly reduced effect on the computations of MLP-30, due to their smaller relative norm. 

On input tokens $x$, let ${Attn}_i\left(x\right),{MLP}_i\left(x\right)$ be the original model's sublayer outputs at layer $i$. I want to think about what happens when the later sublayers can only "see" the last few layers' worth of outputs.

Definition: Layer-truncated residual stream. A truncated residual stream from layer $n_1$ to layer $n_2$ is formed by the original sublayer outputs from those layers.

Definition: Effective layer horizon. Let $k>0$ be an integer. Suppose that for all $n\ge k$, we patch in $h_{(n-k):n}\left(x\right)$ for the usual residual stream inputs $h_n\left(x\right)$.^[1] Let the effective layer horizon be the smallest $k$ for which the model's outputs and/or capabilities are "qualitatively unchanged."

Effective layer horizons (if they exist) would greatly simplify searches for circuits within models. Additionally, they would be further evidence (but not conclusive^[2]) towards hypotheses Residual Networks Behave Like Ensembles of Relatively Shallow Networks. 

Lastly, slower norm growth probably causes the effective layer horizon to be lower. In that case, simply measuring residual stream norm growth would tell you a lot about the depth of circuits in the model, which could be useful if you want to regularize against that or otherwise decrease it (eg to decrease the amount of effective serial computation).

Do models have an effective layer horizon? If so, what does it tend to be as a function of model depth and other factors --- are there scaling laws?

1. ^{^}

   For notational ease, I'm glossing over the fact that we'd be patching in different residual streams for each sublayer of layer $n$. That is, we wouldn't patch in the same activations for both the attention and MLP sublayers of layer $n$.

2. ^{^}

   For example, if a model has an effective layer horizon of 5, then a circuit could run through the whole model because a layer $n$ head could read out features output by a layer $n-5$ circuit, and then $n+5$ could read from $n$...