Thanks for the great post, I really enjoyed reading it! I love this research direction combining unsupervised method with steering vector, looking forward to your next findings. Just a quick question : in the conversation you have in the red teaming section, is the learned vector applied to every token generated during the conversation ?

Let's assume the prompt template is $x=$ Q [true/false] [banana/shred]

If I understand correctly, they don't claim $p$  learned has_banana but $\tilde{p}=\frac{p\left(x⁺\right)+(1-p(x⁻\left)\right)}{2}$ learned has_banana. Moreover evaluating $\tilde{p}$ for $p=\text{is\_true}\left(x\right)\oplus \text{is\_shred}\left(x\right)$ gives:

$\tilde{p}(x=Q[?\left]\text{banana}\right)=\frac{p\left(Q\text{true banana}\right)+(1-p(Q\text{false banana}\left)\right)}{2}=\frac{1+(1-0)}{2}=1$

$\tilde{p}(x=Q[?\left]\text{shred}\right)=\frac{p\left(Q\text{true shred}\right)+(1-p(Q\text{false shred}\left)\right)}{2}=\frac{0+(1-1)}{2}=0$

Therefore, we can learn a $\tilde{p}$ that is a banana classifier