Yep, you're totally right -- thanks!

Oh, one other issue relating to this: in the paper it's claimed that if $\gamma$ is the argmin of $E[\parallel \hat{x}-{\gamma }^{\prime }x{\parallel }_2^2]$ then $1/\gamma$ is the argmin of $E[\parallel {\gamma }^{\prime }\hat{x}-x\parallel ]$. However, this is not actually true: the argmin of the latter expression is $\frac{E[x\cdot \hat{x}]}{E[\parallel \hat{x}{\parallel }_2^2]}\ne {\left(\frac{E[x\cdot \hat{x}]}{E[\parallel x{\parallel }_2]}\right)}^{-1}$. To get an intuition here, consider the case where $\hat{x}$ and $x$ are very nearly perpendicular, with the angle between them just slightly less than ${90}^{\circ }$. Then you should be able to convince yourself that the best factor to scale either $x$ or $\hat{x}$ by in order to minimize the distance to the other will be just slightly greater than 0. Thus the optimal scaling factors cannot be reciprocals of each other.

ETA: Thinking on this a bit more, this might actually reflect a general issue with the way we think about feature shrinkage; namely, that whenever there is a nonzero angle between two vectors of the same length, the best way to make either vector close to the other will be by shrinking it. I'll need to think about whether this makes me less convinced that the usual measures of feature shrinkage are capturing a real thing.

ETA2: In fact, now I'm a bit confused why your figure 6 shows no shrinkage. Based on what I wrote above in this comment, we should generally expect to see shrinkage (according to the definition given in equation (9)) whenever the autoencoder isn't perfect. I guess the answer must somehow be "equation (10) actually is a good measure of shrinkage, in fact a better measure of shrinkage than the 'corrected' version of equation (10)." That's pretty cool and surprising, because I don't really have a great intuition for what equation (10) is actually capturing.

Ah thanks, you're totally right -- that mostly resolves my confusion. I'm still a little bit dissatisfied, though, because the $L_{\text{aux}}$ term is optimizing for something that we don't especially want (i.e. for $\hat{x}\left(\text{ReLU}\right({\pi }_{\text{gated}}\left(x\right))$ to do a good job of reconstructing $x$). But I do see how you do need to have some sort of a reconstruction-esque term that actually allows gradients to pass through to the gated network.

(The question in this comment is more narrow and probably not interesting to most people.)

The limitations section includes this paragraph:

One worry about increasing the expressivity of sparse autoencoders is that they will overfit when
reconstructing activations (Olah et al., 2023, Dictionary Learning Worries), since the underlying
model only uses simple MLPs and attention heads, and in particular lacks discontinuities such as step
functions. Overall we do not see evidence for this. Our evaluations use held-out test data and we
check for interpretability manually. But these evaluations are not totally comprehensive: for example,
they do not test that the dictionaries learned contain causally meaningful intermediate variables in the
model’s computation. The discontinuity in particular introduces issues with methods like integrated
gradients (Sundararajan et al., 2017) that discretely approximate a path integral, as applied to SAEs
by Marks et al. (2024).

I'm not sure I understand the point about integrated gradients here. I understand this sentence as meaning: since model outputs are a discontinuous function of feature activations, integrated gradients will do a bad job of estimating the effect of patching feature activations to counterfactual values.

If that interpretation is correct, then I guess I'm confused because I think IG actually handles this sort of thing pretty gracefully. As long as the number of intermediate points you're using is large enough that you're sampling points pretty close to the discontinuity on both sides, then your error won't be too large. This is in contrast to attribution patching which will have a pretty rough time here (but not really that much worse than with the normal ReLU encoders, I guess). (And maybe you also meant for this point to apply to attribution patching?)

I'm a bit perplexed by the choice of loss function for training GSAEs (given by equation (8) in the paper). The intuitive (to me) thing to do here would be would be to have the $L_{\text{reconstruct}}$ and $L_{\text{sparsity}}$ terms, but not the $L_{\text{aux}}$ term, since the point of ${\pi }_{\text{gate}}$ is to tell you which features should be active, not to itself provide good feature coefficients for reconstructing $x$. I can sort of see how not including this term might result in the coordinates of ${\pi }_{\text{gate}}$ all being extremely small (but barely positive when it's appropriate to use a feature), such that the sparsity term doesn't contribute much to the loss. Is that what goes wrong? Are there ablation experiments you can report for this? If so, including this $L_{\text{aux}}$ term still currently seems to me like a pretty unprincipled way to deal with this -- can the authors provide any flavor here?

Here are two ways that I've come up with for thinking about this loss function -- let me know if either of these are on the right track. Let $f_{\text{gate},\text{ReLU}}$ denote the gated encoder, but with a ReLU activation instead of Heaviside. Note then that $f_{\text{gate},\text{ReLU}}$ is just the standard SAE encoder from Towards Monosemanticity.

Perspective 1: The usual loss from Towards Monosemanticity for training SAEs is $\parallel x-\hat{x}\left(f_{\text{gate},\text{ReLU}}\right(x\left)\right){\parallel }_2^2+\lambda \parallel f_{\text{gate},\text{ReLU}}\left(x\right){\parallel }_1$ (this is the same as your $L_{\text{sparsity}}$ and $L_{\text{aux}}$ up to the detaching thing). But now you have this magnitude network which needs to get a gradient signal. Let's do that by adding an additional term $\parallel x-\hat{x}\left(\tilde{f}\right(x\left)\right){\parallel }_2^2$ -- your $L_{\text{reconstruction}}$. So under this perspective, it's the reconstruction term which is new, with the sparsity and auxiliary terms being carried over from the usual way of doing things.

Perspective 2 (h/t Jannik Brinkmann): let's just add together the usual Towards Monosemanticity loss function for both the usual architecture and the new modified archiecture: $L=L_{\text{reconstruction}}\left(\tilde{f}\right)+L_{\text{reconstruction}}\left(\tilde{f}\right)+L_{\text{sparsity}}\left(f_{\text{gate},\text{ReLU}}\right)+L_{\text{sparsity}}\left(f_{\text{gate},\text{ReLU}}\right)$.

However, the gradients with respect to the second term in this sum vanish because of the use of the Heaviside, so the gradient with respect to this loss is the same as the gradient with respect to the loss you actually used.

I believe that equation (10) giving the analytical solution to the optimization problem defining the relative reconstruction bias is incorrect. I believe the correct expression should be $\gamma =E_{x\sim D}\left[\frac{\hat{x}\cdot x}{\parallel x{\parallel }_2^2}\right]$.

You could compute this by differentiating equation (9), setting it equal to 0 and solving for $\gamma$. But here's a more geometrical argument.

By definition, $\gamma x$ is the multiple of $x$ closest to $\hat{x}$. Equivalently, this closest such vector can be described as the projection ${proj}_x\left(\hat{x}\right)=\frac{\hat{x}\cdot x}{\parallel x{\parallel }_2^2}x$. Setting these equal, we get the claimed expression for $\gamma$.

As a sanity check, when our vectors are 1-dimensional, $x=1$, and $\hat{x}=\frac{1}{2}$, we my expression gives $\gamma =\frac{1}{2}$ (which is correct), but equation (10) in the paper gives $\frac{1}{\sqrt{3}}$.

Great work! Obviously the results here speak for themselves, but I especially wanted to complement the authors on the writing. I thought this paper was a pleasure to read, and easily a top 5% exemplar of clear technical writing. Thanks for putting in the effort on that.

I'll post a few questions as children to this comment.

I'm pretty sure that you're not correct that the interpretation step from our SHIFT experiments essentially relies on using data from the Pile. I strongly expect that if we were to only use inputs from $D_a$ then we would be able to interpret the SAE features about as well. E.g. some of the SAE features only activate on female pronouns, and we would be able to notice this. Technically, we wouldn't be able to rule out the hypothesis "this feature activates on female pronouns only when their antecedent is a nurse," but that would be a bit of a crazy hypothesis anyway.

In more realistic settings (larger models and subtler behaviors) we might have more serious problems ruling out hypotheses like this. But I don't see any fundamental reason that using disambiguating datapoints is strictly necessary. 

(Edits made. In the edited version, I think the only questionable things are the title and the line "[In this post, I will a]rticulate a class of approaches to scalable oversight I call cognition-based oversight." Maybe I should be even more careful and instead say that cognition-based oversight is merely something that "could be useful for scalable oversight," but I overall feel okay about this.

Everywhere else, I think the term "scalable oversight" is now used in the standard way.)

I (mostly; see below) agree that in this post I used the term "scalable oversight" in a way which is non-standard and, moreover, in conflict way the way I typically use the term personally. I also agree with the implicit meta-point that it's important to be careful about using terminology in a consistent way (though I probably don't think it's as important as you do). So overall, after reading this comment, I wish I had been more careful about how I treated the term "scalable oversight." After I post this comment, I'll make some edits for clarity, but I don't expect to go so far as to change the title^[1].

Two points in my defense:

1. Even though "scalable oversight" isn't an appropriate description for the narrow technical problem I pose here, the way I expect progress on this agenda to actually get applied is well-described as scalable oversight.
2. I've found the scalable oversight frame on this problem useful both for my own thinking about it and for explaining it to others.

Re (1): I spend most of my time thinking about the sycophantic reward hacking threat model. So in my head, some of the model's outputs really are bad but it's hard to notice this. Here are two ways that I think this agenda could help with noticing bad particular outputs:

1. By applying DBIC to create classifiers for particular bad things (e.g. measurement tampering) which we apply to detect bad outputs.
2. By giving us a signal about which episodes should be more closely scrutinized, and which aspects of those episodes we should scrutinize. (For example, suppose you notice that your model is thinking about a particular camera in a maybe-suspicious way, so you look for tricky ways that that camera could have been tampered with, and after a bunch of targeted scrutiny you notice a hack).

I think that both of these workflows are accurately described as scalable oversight.

Re (2): when I explain that I want to apply interpretability to scalable oversight, people -- including people that I really expected to know better -- often react with surprise. This isn't, I think, because they're thinking carefully about what scalable oversight means the way that you are. Rather, it seems that a lot of people split alignment work into two non-interacting magisteria called "scalable oversight" and "solving deceptive alignment," and they classify interpretability work as being part of the latter magisterium. Such people tend to not realize that e.g. ELK is centrally a scalable oversight agenda, and I think of my proposed agenda here as attempting to make progress on ELK (or on special cases thereof).

I guess my post muddies the water on all of the above by bringing up scheming; even though this technically fits into the setting I propose to make progress on, I don't really view it as the central problem I'm trying to solve.

1. ^{^}

   Sadly, if I say that my goal is to use interpretability to "evaluate models" then I think people will pattern-match this to "evals" which typically means something different, e.g. checking for dangerous capabilities. I can't really think of a better, non-confusing term for the task of "figuring out whether a model is good or bad." Also, I expect that the ways progress on this agenda will actually be applied do count as "scalable oversight"; see below.