TL;DR: gpt2-small's head L1H5 directs attention to semantically similar tokens and actively suppresses self-attention. The head computes attention purely based on token identity, independent of position. This mechanism is driven by a symmetric bilinear form with negative eigenvalues, which enables suppression. We cluster tokens semantically, interpret the weights to explain the attention scores, and steer self-suppression by tuning eigenvalues.
work performed as part of ARENA 5.0 Capstone project.

Within gpt2-small lies an unusual component: attention head L1H5^[1] which fires on semantically similar concepts. It excels at connecting related concepts: the token cat attends to dog, and red attends to green and blue. Normally, this would be unsurprising, as we would expect embedding vectors to already cluster based on topic/semantic categories. But oddly enough, for this head, tokens do not attend to themselves. For example, the token dog will attend to other animals in the context, but it will not attend to itself or other instances of the token dog.

This behaviour is too specific to be an accident. This short research project aims to find the mechanistic explanation for this semantic grouping and self-avoidance, with the goal to develop useful techniques for analyzing attention patterns along the way.

The Behaviour: Three Simple Rules

Before diving into the mechanistic analysis, let's establish exactly what GPT-2 head L1H5 is doing. After analyzing its attention patterns across hundreds of sequences, the behaviour can be summarized by three rules:

Semantic Clustering: A token attends to other tokens in the sequence that belong to the same semantic category. Examples:
- cat attends to dog and horse, (animals)
- red attends to blue, green and yellow. (colours)
- Monday attends to Tuesday and Friday. (days of the week).
Self-Suppression: A token does not attend to itself, even when it appears multiple times in the sequence.
Fallback to Beginning: If no other token in the sequence belongs to the same semantic category, the token attends to the <bos> (beginning of sequence) token.

These rules hold remarkably consistently across different types of semantic categories, colours, months, days of the week, numbers, names, even verbs and logical operators. Broadly, the behaviour of this head as a semantic head was already mentioned in We Inspected Every Head In GPT-2 Sma...

Posts

Wikitag Contributions

Comments