Here's an AI safety case I sketched out in a few minutes. I think it'd be nice if more (single-AI) safety cases focused on getting good circuits / shards into the model, as I think that's an extremely tractable problem:

Premise 0 (not goal-directed at initialization): The model prior to RL training is not goal-directed (in the sense required for x-risk).

Premise 1 (good circuit-forming): For any $ϵ>0,$ we can select a curriculum and reinforcement signal which do not entrain any "bad" subset of circuits B such that
1A the circuit subset B in fact explains more than $ϵ$ percent of the logit variance^[1] in the induced deployment distribution
1B if the bad circuits had amplified influence over the logits, the model would (with high probability) execute a string of actions which lead to human extinction

Premise 2 (majority rules): There exists $K>0$ such that, if a circuit subset doesn't explain at least $K\cdot ϵ$ of the logit variance, then the marginal probability on x-risk trajectories^[2] is less than $ϵ$.
(NOTE: Not sure if there should be one $K$ for all $ϵ>0$?)

Conclusion:  The AI very probably does not cause x-risk.  
"Proof": Let the target probability of xrisk be $\delta$. Select a reinforcement curriculum such that it has $ϵ<\delta /K$ chance of executing a doom trajectory.

By premise 0, the AI doesn't start out goal-directed. By premise 1, RL doesn't entrain influential bad circuits --- so the overall logit variance explained is less than $\delta /K$. By premise 2, the overall probability on bad trajectories is less than $\delta$.

(Notice how this safety case doesn't require "we can grade all of the AI's actions." Instead, it tightly hugs problem of "how do we get generalization to assign low probability to bad outcome"?)

1. ^{^}

   I don't think this is an amazing operationalization, but hopefully it gestures in a promising direction.

2. ^{^}

   Notice how the "single AI" assumption sweeps all multipolar dynamics into this one "marginal probability" measurement! That is, if there are other AIs doing stuff, how do we credit-assign whether the trajectory was the "AI's fault" or not? I guess it's more of a conceptual question. I think that this doesn't tank the aspiration of "let's control generalization" implied by the safety case.

   Words are really, really loose, and can hide a lot of nuance and mechanism and difficulty.