Suppose we have a capable and potentially scheming model, and before we deploy it, we want some evidence that it won’t do anything catastrophically dangerous once we deploy it. A common approach is to use black-box alignment evaluations. However, alignment evaluations are only reassuring to the extent that the model can't reliably[1] distinguish the deployment distribution from the evaluation distribution, as it is otherwise difficult to rule out the possibility of alignment faking.
There are many approaches one could use to try to make evaluations appear more realistic: you can try to create realistic environments (e.g. Petri, WebArena, OSWorld); use data from past deployments (e.g. OpenAI, SAD); and spoof tool-call responses (e.g. ToolEmu).
However, the core difference between an alignment evaluation and a...
Decision theory is back in fashion (defining fashion as "one good post on a good EA blog"). Bentham's Bulldog (BB) has published a case against FDT (functional decision theory), contrasting rationalist enthusiasm with academic scepticism: "Academic decision theorists don't like the theory. The number of academic decision theorists who adopt it could be counted on one hand by someone missing four of their fingers."
I am, just barely, a published academic decision theorist, so you can keep a small finger to count me too. My position is that, though FDT may have problems with its definitions and under-definedness, we can build defined variants that achieve what FDT attempted to.
I want to do two things in this post. First, sketch a "pragmatic" version of FDT designed to sidestep the...
It depends on what we apply the word "rational" to
People tend to use "rational" it to defend their preferred position, so I find discussions often degenerate into fruitless semantic debates.
The context is MIRI's twist on Axelrod's Prisoner's Dilemma tournament. Axelrod's competitors were programs, facing each other in an iterated Prisoner's Dilemma. MIRI's tournament is a one-shot Prisoner's Dilemma, but the programs get to read their opponent's code. Or, rather, a description of the behavior of the code in Gödel-Löb provability logic, which turns out to be enough to determine their behavior in the setup.
One fun result, right in the beginning of the paper, is about a program, FairBot, whose behavior is specified by "I'll cooperate with you if you (provably) cooperate with me". Despite the appearance of circularity, FairBot cooperates with itself. The proof involves Löb's theorem, so we call this Löbian cooperation.
Andrew Critch has suggested another way of proving self-cooperation. Instead of Löb's theorem, we use what he calls "Payor's lemma". It...
Man, okay I'm now wondering if the following works for a PrudentBot implementation? Calling our PrudentBot
In symbols,
This is derived from the limit of an algorithm that searches for increasingly long proofs that we can obtain the opponent's cooperation one way or another, with some budget for looking harder for a (defect, cooperate) proof s...
I'm sorry for the slow response! Luckily I waited long enough that we posted the follow-up.
I definitely agree with both of your points: that you could argue that your fake eval and fake deployment are easier to distinguish, and that this is kicking the can down the rode somewhat.
Crucially, I think making a fake eval and fake deployment that captures some of the crucial strategies an AI could follow is challenging. We just released a follow-up suggesting Bounding eval awareness of ~human-level AI across the safe-to-dangerous shift using human red-teaming. ... (read more)