In the small group of people who care about deconfusing goal-directedness and like Daniel Dennett's intentional stance, there's one paper everyone mentions: Agents and Devices, by Orseau, McGill and Legg. It's also pretty much the only paper ever mentioned on that topic.

Given that this paper seems to formalize the intentional stance enough for running experiments on it, and how I already wrote that a formalized intentional stance sounds like the most promising deconfusion of goal-directedness, I thought it was important to review it to clarify what I think is missing.

Spoiler: I don't find their proposal satisfying as a formalization of the intentional stance, but it's a great first step in making it more concrete and objectionable.

Thanks to Victoria Krakovna and Ramana Kumar who always ask me how my current attempt at deconfusing goal-directedness is related to this paper.

Summary

What's this paper about?

Basically, the author create two classes of hypotheses:

devices which are just programs with the right interface,
and agents which are optimal (technically $ϵ$ -greedy optimal) policies for some reward functions.

These classes are combined into mixtures through priors -- speed prior over the devices and simplicity prior (technically switching prior) over the rewards/goals respectively. Next the two mixtures are combined in equal weights to form one mixture, which is used as prior on applying Bayes rules to some observed trajectory.

So after getting a trajectory, the result is a posterior on our mixture of agents and devices, which gives us both which of agent and device is more probable, but also the more probably hypothesis among those classes.

They actually run some experiments in a simple maze, which give the expected results: following walls and running in circles are considered as devices, following a goal or switching goals are considered as agents, and random behavior is considered as neither.

What I like

As a first attempt to make the intentional stance formal enough to experiment on and criticize, the paper does his job perfectly.

Here are also some ideas I found particularly interesting:

(The switching prior) I hadn't heard of it, but it sounds like a pretty neat way to add changes of goals to the formalization.
(Using the speed prior to deal with interpreters) I need to read this part in more detail, but the choice of a speed prior over the simplicity prior

...

Stances