Summary

Contrast-Consistent Search (CCS) is a method for finding truthful directions within the activation spaces of large language models (LLMs) in an unsupervised way, introduced in Burns et al., 2022. However, all experiments in that study involve training datasets that are balanced with respect to the ground-truth labels of the questions used to generate contrast pairs.^[1] This allows for the possibility that CCS performance is implicitly dependent on the balance of ground-truth labels, and therefore is not truly unsupervised.

In this post, we demonstrate that the imbalance of ground-truth labels in the training dataset can prevent CCS, or any contrast-pair-based unsupervised method, from consistently finding truthful directions in an LLM's activation space.

This post is a distillation of a more detailed write-up.

Normalization of Contrast-Pair Representations

The performance of CCS, as well as the other unsupervised methods introduced in Burns et al., 2022, is driven by representing contrast pairs such that truthfulness is isolated as the most salient feature. Here is a basic overview of the process:

We begin with a question, $q_{i}$ :

$q_{i}$ = "Is a cat a mammal?"

From any question $q_{i}$ , a contrast pair may be formed:

$x_{i}^{+}$ = "Is a cat a mammal? Yes"

$x_{i}^{-}$ = "Is a cat a mammal? No"

In this case, the ground-truth label for $q_{i}$ is positive, as the positive completion $x_{i}^{+}$ is the true statement among the pair.

Let $ϕ (x)$ be a feature extractor that maps a series of tokens, $x$ , to a vector. In Burns et al., 2022, $ϕ (x)$ is the mapping produced by inputting $x$ into an LLM and extracting the hidden state output by the last layer and at the last token's residual stream.

Then, for the contrast pair $(x_{i}^{+}, x_{i}^{-})$ , we can generate an unnormalized representation pair of vectors, $(ϕ (x_{i}^{+}), ϕ (x_{i}^{-}))$ . The unnormalized representations of this contrast pair may encode for two primary differences:

$x_{i}^{+}$ ends in "Yes", while $x_{i}^{-}$ ends in "No"
One of ( $x_{i}^{+}$ , $x_{i}^{-}$ ) is true, while the other is false

For any unsupervised method such as CCS to work, we need to isolate the feature encoded by 2 from that of 1. This is where mean normalization of the representations comes in. (Note: 1 and 2 are not the only possible differences. The problem of models representing additional "truth-like" features is discussed in ...

Posts

Wikitag Contributions

Comments