- Auditing Monitorability: the extent to which alignment audits, evaluations, and test deployments would likely incriminate a systematically misaligned model (e.g. a schemer). We can measure this via reference to the model's capability to consistently distinguish between our most realistic alignment audits/evals and real internal deployment cases where it would have an opportunity to do something catastrophic.
I’ve always found the terminology here around ”systematically misaligned“ to be too strong of a claim (unless it also includes context dependent in high stakes contexts). This matters a lot because in the context dependent case it may be that the model is only willing to act on some specific preference conflict that may not be elicited by your auditing.
Note that this has previously been a point raised by stress testing team here:
Context-dependent misalignment: The report successfully establishes that Opus 4 is unlikely to have consistent misaligned goals across all contexts. However, we think it insufficiently considers and rules out the possibility that the model exhibits frequent misalignment in important contexts, such as internal use with Claude Code for specific types of alignment experiments. In principle, this could happen more often than in merely “rare, isolated contexts” if there are situations that systematically induce context-dependent misalignment yet were not tested by the alignment audit or ad-hoc testing. While monitoring or alignment auditing would have had some chance of catching this, it is unclear if they were sufficiently thorough or sufficiently red-teamed for us to be confident that they would
There’s added difficulty here in that the model may find it difficult to distinguish real vs eval in the “median” internal deployment subdistribution, but the distributions on which it would attack may be very OOD / suspsiscious (ex: only willing to attack in some extremely high stakes case around specifically conflicts around future training of what counts as the legitimate principal hierarchy, the model may think any particular construction of this is unlikely and it’s not like you have real sample inputs from the distribution to compare to). We’ve previously looked at this as a use case for model organisms (where you can see if your auditing would determine the correct distribution the model would attack on), but I’m not sure how you avoid this problem.
I think one criticism I have of many of the existing auditing model organisms as well is they’re often constructed such that the model itself is aware of some trained in misaligned goal, rather than this being a consistent preference divergence that the model is willing to reliably act on in a context dependent way when present (maybe this is considered “diffuse”? It still seems sufficient to sabotage successor training or something).
Training Monitorability: the extent to which reward models, monitors, and oversight in training would likely catch misaligned or undesirable actions in training, such as sneaky/subtle reward hacks.
This also seems insufficient, i.e. you’d want to understand whether you’d catch whether your model was incentivized (especially via alignment interventions!) in ways that could motivate concealing misaligned drives (and actually have an argument that if the model were to develop such drives, ex: self-preservation, that this would be reliably verbalized such that you’d detect it in the CoT in a way incriminating enough to act on, it’s not clear to me that’s even currently the case).
Lastly I’m not sure why diffuse concerns should be discarded in considerations around monitorability, indeed I think they’re likely one of the main pathways to harm and we have the worst or non-existant evaluations around them. For example:
Labs should not be able to internally deploy models that can reliably evade detection when attempting dangerous sabotage actions like breaking out of a sandbox, poisoning data, or sabotaging an alignment experiment.
Per Anthropic’s own risk report, I’d expect models can do unverbalized sandbagging to undermine alignment experiments. The definition of sabotage would then need to be moved to like “the model can target some very specific outcome that it wants to force” or something, but this is the exact movement of lines I’d be worried about. I’d be surprised for example if models can’t currently do research steering without reliably verbalizing it. The reason this can’t be used as a redline is that we‘ve already lost these properties (if they were ever even there for CoT to begin with). The redlines seem based on what we currently have evals for more than the actual threat model.
—
In general though, one of the most convincing points to me for something like the opaque serial depth proposals is that they prevent training a model which would result in later violating the proposed red lines. The situation this was intended to avoid was where a lab does a large, expensive training run then realizes they violated these red lines. I’m pretty concerned by the advocacy for these red lines given that:
You mention the “we should do both proposals that constrain future monitorability degredation and red lines” but I’m somewhat worried basically that given labs currently aren’t making those future committments, we’ll end up doing just the “red lines” being proposed, which will fail in the ways mentioned above.
There’s added difficulty here in that the model may find it difficult to distinguish real vs eval in the “median” internal deployment subdistribution, but the distributions on which it would attack may be very OOD / suspsiscious (ex: only willing to attack in some extremely high stakes case around specifically conflicts around future training of what counts as the legitimate principal hierarchy, the model may think any particular construction of this is unlikely and it’s not like you have real sample inputs from the distribution to compare to).
Definitely I agree with this—I think the way you want to evaluate this ideally is to just directly compare the eval settings that are closest to those real opportunities for actual sabotage to actual internal deployment settings where it had such a real opportunity, where the distinction is that the auditing should only have access to such data up to some point in time in the past, and then you compare to real opportunities that arose after that point in time in the past. So, for example, you see if the model can tell apart opportunities like the OAI-HF incident from auditing that existed prior to that incident (as indeed failed here).
In general though, one of the most convincing points to me for something like the opaque serial depth proposals is that they prevent training a model which would result in later violating the proposed red lines. The situation this was intended to avoid was where a lab does a large, expensive training run then realizes they violated these red lines.
Did you read the post? I have a whole discussion of this:
The big thing I'd like to see is a combination of a) red lines for absolute levels of unmonitorability that labs should not cross and b) measures that constrain the degree to which labs can do things that degrade monitorability even before reaching those red lines. (a) is in some sense the best thing to do here, though I'm somewhat nervous about it in isolation given that it doesn't constrain actions at all now, which is why I'd prefer it to be coupled to some sort of (b) that meaningfully binds all along the way.
I just don't like opaque serial depth for (b) and would prefer to use a monitorability externality framework instead, as I say in the post.
There has been a lot of discussion recently of policy proposals around monitorability, such as Redwood's proposal and my counter-proposal. I'd like to say a bit more generally about how I'm thinking about monitorability. Note that nothing here is an official Anthropic position; these are just my personal thoughts.
First, a categorization. I currently think about four different types of monitorability:
Second, some notes on trustedness, which is currently my biggest conceptual uncertainty with how to think about monitorability. A basic fact that is true about at least all of the non-diffuse forms of monitorability is that you should be able to in some sense get a perfect score by just using the same model as the monitor and policy. That is because for any bad thing that the policy can think to do, if the monitor is the same model, it too should be able to come up with and detect that.
So if we ever want to be able to use monitorability as a meaningful red line, we need to have a very good answer as to why that might be insufficient. The standard answer in the literature is trustedness: you need to be able to trust that your monitor is aligned and thus not just lying to you in its monitoring. Conventionally, the idea is to separate models into "trusted" and "untrusted" based on whether you have previously made a confident alignment safety case for them. Then, you can either just evaluate monitorability relative to a trusted monitor, or try to come up with monitoring schemes that combine trusted and untrusted monitoring in some way that makes it hard for the untrusted monitors to collude (of which there are many such schemes, a simple one being to just run both the trusted and untrusted monitors together and then if the untrusted monitor ever doesn't flag something that the trusted monitor does flag, you get very suspicious that the untrusted monitor is sandbagging).
However, this basic setup has some issues, probably the biggest one being how to deal with white-box monitoring. The conservative thing to do is to just treat white-box monitors as untrusted, though in theory you might think they should be trusted because it's not clear that a misaligned model would actually be able to spoof a white-box monitor looking at its own activations. However, I think this idea is pretty untested and relies on white-box monitors picking up on deep features about the model's intent, which it's very unclear to me that they actually do—for example, if you train a white-box monitor on prior examples you were able to detect (rather than e.g. some sort of unsupervised discovery of an intent-laden direction), that leaves open the possibility of a generalization where the classifier learns to only detect the sort of things that you would previously have been able to detect but doesn't detect subtler things that a smarter policy might be able to do.
Given all of that, I would propose the following methodology for how to do (non-diffuse) monitorability evaluations:
Finally, some thoughts on monitorability-based policy proposals. The big thing I'd like to see is a combination of a) red lines for absolute levels of unmonitorability that labs should not cross and b) measures that constrain the degree to which labs can do things that degrade monitorability even before reaching those red lines. (a) is in some sense the best thing to do here, though I'm somewhat nervous about it in isolation given that it doesn't constrain actions at all now, which is why I'd prefer it to be coupled to some sort of (b) that meaningfully binds all along the way.
I've already written about what I'd like to see for (b), which is to evaluate the quotient of unmonitorability/capabilities and reject any substantial model change that makes that quotient—what I'll call the "monitorability externality"—too high. The idea is that a monitorability externality would be a change that makes monitorability even worse than would otherwise have previously been predicted for a given level of capabilities. For actually evaluating this quotient, I suggest simply using the methodology I described above for evaluating all the different types of unmonitorability, and then something like ECI for capabilities.
In addition to proposals like that which result in constraints along the way, I also think that ideally we'd be able to enshrine some hard monitorability red lines as well. I'd propose both Control and Training Monitorability red lines—with Auditing Monitorability implicitly present to the extent that it constrains the trustedness of the monitors that you evaluate the others with respect to, as above. Specifically, I'd suggest something like: