I'm not sure what it means for this work to "not apply" to particular systems. It seems like the claim is that decision theory is a way to understand AI systems in general and reason about what they will do, just as we use other theoretical tools to understand current ML systems. Can you spell this out a bit more? (Note that I'm also not really sure what it means for decision theory to apply to all AI systems: I can imagine kludgy systems where it seems really hard in some sense to understand their behavior with decision theory, but I'm not confident at all)
I agree with both your claims, but maybe with less confidence than you (I also agree with DanielFilan's point below).
Here are two places I can imagine MIRI's intuitions here coming from, and I'm interested in your thoughts on them:
(1) The "idealized reasoner is analogous to a Carnot engine" argument. It seems like you think advanced AI systems will be importantly disanalogous to this idea, and that's not obvious to me.
(2) 'We might care about expected utility maximization / theoretical rationality because there is an impo... (read more)