I'm worried about AI models intentionally doing bad things, like sandbagging when doing safety research. In the regime where the AI has to do many of these bad actions in order to cause an unacceptable outcome, we have some hope of identifying examples of the AI doing the bad action (or at least having some signal at distinguishing bad actions from good ones). Given such a signal we could:

Directly train the policy to not perform bad actions.
Train activation monitors to detect bad actions. These monitors could be used in a variety of ways such as:
1. Rejecting actions flagged as bad. Either resample a response from the same policy, or using some different process.
2. Look into bad actions further, to try to incriminate the AI. Incrimination based strategies are out of scope for this post: I'll be solely interested in the question of: What should we expect the sample efficiency of training policies and monitors to be at removing bad behavior? This theoretical analysis isn't supposed to replace empirical research on the matter, although I hope that it can generate some guiding intuition and good hypotheses to test.

Learning theory basic background

Let's start with the following problem setup:

We have a function $f : D \to {0, 1}$ from some function family $F$ of VC dimension $d$ .
- The VC dimension of a function family measures how "expressive/complex" the family is. Formally, it's the largest number $d$ such that there's some set $X$ of $d$ points so that for any $2$ -coloring $χ$ of $X$ , there exists a function $f \in F$ such that $f (X) = χ (X)$ . As an example, the family of "union of $k$ intervals" has VC dimension $2 k$ .
We receive $N$ labelled samples $x, f (x)$ .
We'd like to output a function $g$ to minimize ${Pr}_{x \sim D} [f (x) \neq g (x)]$ ; this quantity that we're trying to minimize is called our error. If it helps, you could think of $f$ as an AI model that must make a binary decision (e.g., maybe $f$ is a monitor flagging actions as bad or not).

The Fundamental Theorem of Statistical Learning states that it's possible to achieve error at most $O (\frac{d}{N} log N)$ .

There's a related version of the problem, where the labels are noisy. In the noisy case, there is some amount of irreducible error that any predictor will incur. However, the extra expected error beyond the irreducible error is at most $O (\sqrt{d / N})$ in the noisy case.

SGD doesn't do optimal Bayesian learning, but these learning theory results seem like a reasonable first ...