Much of my thinking over the last year has focusing on understanding the concept of "distributed agents", as opposed to the "centralized agents" that the existing paradigm of expected utility maximization describes. One way of describing the difference is in terms of how autonomous their subagents are. Another is that centralized agents are more efficient (as sometimes formalized by the notion of "coherence"), while distributed agents are more robust.

Unfortunately robustness is hard to formalize, since it requires that you perform well even in unpredicted (and sometimes unpredictable) situations. I give some tentative characterizations of distributed agents below, but there's still a lot of work to be done to formally define distributed agents. And ultimately I'd like to go further, to understand how agents can have both properties—which is roughly what I mean by "coalitional agency".

I gave a talk on the distinction about seven months ago. I'd been hoping to write up the main ideas at more length, but since that doesn't look like it'll happen any time soon, I'm sharing the slides below. Hopefully they're reasonably comprehensible by themselves, but feel free to ask questions about any parts that are unclear.

See more on my interpretation of Yudkowsky here; note that he disagrees with my emphasis on compression though (as per the exchange in the comments).

My post on why I'm not a bayesian also gives a sense of what understanding epistemology in more distributed terms looks like.

"Will's very rough first pass" is a reference to the passage in his textbook on utilitarianism where Will MacAskill describes what decision procedure a utilitarian should follow. My point here is to contrast how much thought he (and other utilitarians) put into finding criteria of rightness, vs how rudimentary their thinking about decision procedures is.

Understanding systematization

Understanding systematization