This is a accompanying blog post to work done by Callum McDougall, Arthur Conmy and Cody Rushing as part of SERI MATS 2023. The work was mentored by Neel Nanda and Tom McGrath. You can find our full paper at https://arxiv.org/abs/2310.04625. We i) summarize our key results, ii) discuss limitations and future work and iii) list lessons learnt from the project

Key Results

Copy Suppression

In the paper, we define copy suppression as the following algorithm:

If components in earlier layers predict a certain token, and this token appears earlier in the context, the attention head suppresses it.

We show that attention head L10H7 in GPT2-Small (and to a lesser extent L11H10) both perform copy suppression across the whole distribution, and this algorithm explains 76.9% of their behavior.

For example, take the sentence "All's fair in love and war." If we ablate L10H7, the model will incorrectly predict "...love and love." A diagram of this process:

And a few more examples:

Copy Suppression-Preserving Ablation

To test how much of L10H7's behaviour we explained, we designed a form of ablation we called copy suppression preserving ablation (CSPA). The idea is that CSPA should delete all functionality of the head except for the things which are copy suppression. If this completely destroys performance then the head isn't doing copy suppression; if it doesn't affect performance at all.

The technical details of CSPA can be found in our paper. Very loosely, CSPA does two important things:

Delete all information moving from source token $S$ to destination token $D$ , except for those where token $S$ is predicted at $D$ before head L10H7.
For the information which isn't deleted, project it onto the unembedding for token $S$ (and throw away the positive component), so we're only capturing the direct effect of suppressing the token we're attending to.

The key result of this paper was the graph below. $D_{C S P A}$ is the KL divergence between (predictions after applying CSPA) and (original predictions). $D_{M A}$ is the KL divergence between (predictions after mean-ablating) and (original predictions). Each point is the average of a bunch of these values (we've grouped according to the value of $D_{M A}$ ). In other words, each (x, y) point on this graph represents (average KL div from mean ablating, average KL div from CSPA) for...

Posts

Wikitag Contributions

Comments