Thanks! To clarify:

• An important consideration is that alignment fakers might have exactly the same observable behavior on the training distribution as intent-aligned AIs. That's why I object to your sentence "(..) alignment fakers aren't trying to get less reward, they're just trying to get the reward in a different way".
• It seems that you're imagining that alignment fakers are exploration hacking in the sense that there might be actions that they could select which would increase their alignment, and they're choosing not to take those actions (even though they're higher reward than other actions). It's unclear to my why you think this.
  • Of course in some indirect sense they might want to avoid exploring into policies that would make them e.g. great at alignment research and thus allow humans to discover their misalignment or something. But presumably that's not what you mean.
  • Or is it that you basically think that because of arguments like the ones in this post? (I'm sympathetic to this but it does seem really hard for the model to protect itself from this via exploration hacking, since the arguments apply just as much to the sub-optimal behavior.)

I've now looked at the "Early Findings" section. I think this is great stuff and I'm excited to see your further work. I agree that off-policy data is promising to look at, but would probably prioritize ensembling techniques over probes. (Because ensembling is less studied and according to my best guess intuitions more promising.)

Thanks for working on this (in my opinion) very important problem! (And happy to advise in some capacity if helpful.)

Some comments on the conceptual work (I plan to comment on the empirical work later):
 

(This is slightly broader than some usage of the term, and in particular includes cases where the model still takes some high-reward actions, but chooses them in a way that leads to an altered training outcome. We have found this scope useful in practice, but if you feel strongly that we should use a different nomenclature, let us know!)

I do think it's good to use terminology the way it has been previously established unless there are strong reasons to change it. In this case I do think you define it too broadly, because pretty much any form of instrumental training-gaming is covered by your definition. And for empirical projects it's good to stay focused.

There are several ways models might strategically use exploration during training to shape their future behavior in ways aligned with their current preferences:

Nitpick: I'm not sure how the taxonomy that follows is helpful. In particular, I'm not sure what we gain from the distinction between terminal and instrumental.

1. Misaligned models might successfully resist alignment training. One good example of this would be reliable alignment faking. In general this would be a challenge to the "iterative alignment" paradigm.

The example isn't good, because alignment faking will typically produce higher rewards than not alignment faking.

Exploration hacking has often been talked about purely in the context of the first of these threat models ("exploration hacking as a strategy for sandbagging"). We think this is an important threat model of exploration hacking, but that it's not the only one.

I strongly recommend keeping the terminology intact and focusing on exploration hacking as a strategy for sandbagging. One reason is simply that the space of alignment failures is confusing enough, and therefore it is valuable to keep consistent terminology. Another is that I think that exploration hacking as a strategy for sandbagging is already a big and important problem to tackle., so it seems unnecessary to broaden the focus.
 

Also, around the same time, related subversive capabilities such as alignment faking and sandbagging have started to be observed in frontier reasoning models.

Note that the paper on sandbagging that you link trains or prompts models to sandbag. It is not evidence for frontier models naturally sandbagging.

Alignment faking. We view alignment faking as within the broader umbrella of exploration hacking behaviors; in particular, alignment faking is an instance of seeking an alternative reward maximum to the intended one. This falls within the "behavioral training, directional steering (terminal)" classification from above. Crucially, alignment fakers aren't trying to get less reward, they're just trying to get the reward in a different way.

No, this is a misunderstanding. Alignment fakers need not be reward seekers. I recommend reading this post by Joe Carlsmith.

For example, it could exploration hack during training:

1. It could strategically avoid exploring high-reward regions such that they never get reinforced
2. It could strategically explore in ways that don’t lead to general updates in behavior
3. It could condition its behavior on the fact that it is in a training environment and not real deployment such that it learns “don’t scheme when observed” rather than “never scheme”

Nitpick: The third item is not an instance of exploration hacking.

I think it's interesting that @Lukas Finnveden seems to think

compared to what you emphasize in the post, I think that a larger fraction of the benefits may come from the information value of learning that the AIs are misaligned.

in contrast to this comment. It's not literally contradictory (we could have been more explicit about both possibilities), but I wonder if this indicates a disagreement between you and him.

Agreed. 

(In the post I tried to convey this by saying

(Though note that with AIs that have diminishing marginal returns to resources we don’t need to go close to our reservation price, and we can potentially make deals with these AIs even once they have a substantial chance to perform a takeover.)

in the subsection "A wide range of possible early schemers could benefit from deals".)

[Unimportant side-note] We did mention this (but not discuss extensively) in the bullet about convergence, thanks to your earlier google doc comment :)

We could also try to deliberately change major elements of training (e.g. data used) between training runs to reduce the chance that different generations of misaligned AIs have the same goals.