AI ALIGNMENT FORUMTagsAF

Embedded Agency is an intuitive notion that an understanding of the theory of rational agents must account for the fact that the agents we create (and we ourselves) are parts of the world, and not separated from it. This is in contrast with much current basic theory of AI (such as solomonoff induction) which implicitly supposes a separation between the agent and the-things-the-agent-has-beliefs about.

Embedded Agency is not a fully formalised research agenda, but Scott Garrabrant and Abram Demski have written the canonical explanation of the idea in their sequence Embedded Agency. This points to many of the core confusions we have about rational agency and attempts to tie them into a single picture.