Nice work! I'm not sure I fully understand what the "gated-ness" is adding, i.e. what the role the Heaviside step function is playing. What would happen if we did away with it? Namely, consider this setup:

Let $f$ and $\hat{x}$  be the encoder and decoder functions, as in your paper, and let $x$ be the model activation that is fed into the SAE.

The usual SAE reconstruction is $\hat{x}\left(f\right(x\left)\right)$, which suffers from the shrinkage problem.

Now, introduce a new learned parameter $t\in R^{n_{features}}$, and define an "expanded" reconstruction $y_{expanded}=\hat{x}(t\odot f(x\left)\right)$, where $\odot$ denotes elementwise multiplication.

Finally, take the loss to be:

$L=||{\hat{x}}_{copy}\left(f\right(x\left)\right)-x|{|}_2^2+||y_{expanded}-x|{|}_2^2+\lambda ||f\left(x\right)|{|}_1$.

where ${\hat{x}}_{copy}$ ensures the decoder gets no gradients from the first term. As I understand it, this is exactly the loss appearing in your paper. The only difference in the setup is the lack of the Heaviside step function.

Did you try this setup? Or does it fail for an obvious reason I missed?

The typical noise on feature $f_1$ caused by 1 unit of activation from feature $f_2$, for any pair of features $f_1$, $f_2$, is (derived from Johnson–Lindenstrauss lemma)

 $ϵ=\sqrt{\frac{8\ln \left(m\right)}{n}}$  ^[1]

1. ... This is a worst case scenario. I have not calculated the typical case, but I expect it to be somewhat less, but still same order of magnitude

Perhaps I'm misunderstanding your claim here, but the "typical" (i.e. RMS) inner product between two independently random unit vectors in $R^n$ is $n^{-1/2}$. So I think the $\sqrt{8\ln m}$ shouldn't be there, and the rest of your estimates are incorrect.

This means that we can have at most  $l<n/\left(8\ln \right(m\left)\right)$ simultaneously active features

This conclusion gets changed to $l<n$.