This is a special post for quick takes by TurnTrout. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Basilisks are a great example of plans which are "trying" to get your plan evaluation procedure to clock in a huge upwards error. Sensible beings avoid considering such plans, and everything's fine. I am somewhat worried about an early-training AI learning about basilisks before the AI is reflectively wise enough to reject the basilisks.
For example:
- Pretraining on a corpus in which people worry about basilisks could elevate reasoning about basilisks to the AI's consideration,
- at which point the AI reasons in more detail because it's not sufficiently reflective about how this is a bad idea,
- at which point the AI's plan-estimates get distorted by the basilisk,
- at which point the AI gives in to the threats because its decision theory is still bad.
(I expect this worry to change in some way as I think about it more. Possibly basilisks should be scrubbed from any training corpus.)