This post is the result of my attempts to understand what’s going on in these two posts from summer 2021:

Paul Christiano: https://www.alignmentforum.org/posts/QqwZ7cwEA2cxFEAun/teaching-ml-to-answer-questions-honestly-instead-of

Evan Hubinger: https://www.alignmentforum.org/posts/gEw8ig38mCGjia7dj/answering-questions-honestly-instead-of-predicting-human

The underlying problem is similar in some ways to Eliciting Latent Knowledge, although not quite the same. According to Paul, understanding the two-headed algorithm (described in these posts) is likely to be fairly useful to people working on ELK. A similar algorithm is described in the ELK report, under Strategy: penalize reporters that work with many different predictors and in related appendices.

I initially found the algorithm pretty confusing, so I wrote this post for anyone else who's confused by it.

I start with the basic strategy underlying Paul’s post, but in a simpler / more general setting without any ML. Then I add back in the ML context, followed by the specific context of the problem Paul and Evan discuss in their posts. (This is not quite the same as the ELK context.)

Along the way, I outline a simple application of this strategy: finding the second-simplest “meaningfully distinct” solution to a problem. I note that Paul’s application is sort of a special case of this.

Simple abstract version: conditioning on G(a, b)

Suppose we have a probability distribution over some discrete space A × B:

$p : A \times B \to R$

This is shown in the figure below -- dots represent probability mass:

Suppose there’s some point (a⁺, b⁺) in A × B, and we want to obtain it by sampling from this distribution. Unfortunately, it’s not very high probability, so we probably won’t succeed:

Conditioning on D(·, ·)

An obvious strategy, if sampling is very cheap, is to sample repeatedly until we get (a⁺, b⁺).

Unfortunately, we don’t even know how to recognize it if we had it -- there are other points that look very similar. Worse still, some of them are higher in probability.

Aside: rejection sampling might sound too expensive to be worth thinking about. We really care about it here as a proxy for processes like gradient descent, which are doing something that looks like “sample from a distribution conditional on a constraint” -- the constraint being “loss is low” and the distribution being an architecture- and optimizer-depend

...

Posts

Wiki Contributions

Comments