I was just looking back through 2019 posts, and I think there's some interesting crosstalk between this post and an insight I recently had (summarized here).
In general, utility maximizers have the form "maximize E[u(X)|blah]", where u is the utility function and X is a (tuple of) random variables in the agent's world-model. Implication: utility is a function of random variables in a world model, not a function of world-states. This creates ontology problems because variables in a world-model need not correspond to anything in the real world. For instance, some people earnestly believe in ghosts; ghosts are variables in their world model, and their utility function can depend on how happy the ghosts are.
If we accept the "attainable utility" formulation of impact, this brings up some tricky issues. Do we want to conserve attainable values of E[u(X)], or of u(X) directly? The former leads directly to deceit: if there are actions a human can take which will make them think that u(X) is high, then AU is high under the E[u(X)] formulation, even if there is actually nothing corresponding to u(X). (Example: a human has available actions which will make them think that many ghosts are happy, even though there are no actual ghosts, leading to high AU under the E[u(X)] formulation.) On the other hand, if we try to make attainable values u(X) high directly, then there's a question of what that even means when there's no real-world thing corresponding to X. What actions in the real world do or do not conserve the attainable levels of happiness of ghosts?
Right. Another E[u(X)] problem would be, the smart AI realizes that if the dumber human keeps thinking, they'll realize they're about to drive off of a cliff, which would negatively impact their attainable utility estimate. Therefore, distract them.
I forgot to mention this in the sequence, but as you say - the formalisms aren't quite right enough to use as an explicit objective due to confusions about adjacent areas of agency. AUP-the-method attempts to get around that by penalizing catastrophically disempowering behavior, such that the low-impact AI doesn't obstruct our ability to get what we want (even though it isn't going out of its way to empower us, either). We'd be trying to make the agent impact/de facto non-obstructive, even though it isn't going to be intent non-obstructive.
I'm not aware of others explicitly trying to deduce our native algorithm for impact. No one was claiming the ontological theories explain our intuitions, and they didn't have the same "is this a big deal?" question in mind. However, we need to actually understand the problem we're solving, and providing that understanding is one responsibility of an impact measure! Understanding our own intuitions is crucial not just for producing nice equations, but also for getting an intuition for what a "low-impact" Frank would do.
I wish you'd expanded on this point a bit more. To me, it seems like to come up with "low-impact" AI, you should be pretty grounded in situations where your AI system might behave in an undesirably "high-impact" way, and generalise the commonalities between those situations into some neat theory (and maybe do some philosophy about which commonalities you think are important to generalise vs accidental), rather than doing analytic philosophy on what the English word "impact" means. Could you say more about why the test-case-driven approach is less compelling to you? Or is this just a matter of the method of exposition you've chosen for this sequence?
Most of the reason is indeed exposition: our intuitions about AU-impact are surprisingly clear-cut and lead naturally to the thing we want "low impact" AIs to do (not be incentivized to catastrophically decrease our attainable utilities, yet still execute decent plans). If our intuitions about impact were garbage and misleading, then I would have taken a different (and perhaps test-case-driven) approach. Plus, I already know that the chain of reasoning leads to a compact understanding of the test cases anyways.
I've also found that test-case based discussion (without first knowing what we want) can lead to a blending of concerns, where someone might think the low-impact agent should do X because agents who generally do X are safer (and they don't see a way around that), where someone might secretly have a different conception of the problems that low-impact agency should solve, etc.
Thoughts I have at this point in the sequence
Thank you! I poured a lot into this sequence, and I'm glad you're enjoying it. :) Looking forward to what you think of the rest!
I recall what you said about being unable to type without feeling pain. Did this not extend to handwriting?
Handwriting was easier, but I still had to be careful not to do more than ~1 hour / day.
These existential crises also muddle our impact algorithm. This isn't what you'd see if impact were primarily about the world state.
Appendix: We Asked a Wrong Question
How did we go wrong?
I think what gets you is asking the question "what things are impactful?" instead of "why do I think things are impactful?". Then, you substitute the easier-feeling question of "how different are these world states?". Your fate is sealed; you've anchored yourself on a Wrong Question.
At least, that's what I did.
Exercise: someoneme, early last year says that impact is closely related to change in object identities.
Find at least two scenarios which score as low impact by this rule but as high impact by your intuition, or vice versa.
You have 3 minutes.
Gee, let's see... Losing your keys, the torture of humans on Iniron, being locked in a room, flunking a critical test in college, losing a significant portion of your episodic memory, ingesting a pill which makes you think murder is OK, changing your discounting to be completely myopic, having your heart broken, getting really dizzy, losing your sight.
That's three minutes for me, at least (its length reflects how long I spent coming up with ways I had been wrong).
Appendix: Avoiding Side Effects
Some plans feel like they have unnecessary side effects:
We talk about side effects when they affect our attainable utility (otherwise we don't notice), and they need both a goal ("side") and an ontology (discrete "effects").
Accounting for impact this way misses the point.
Yes, we can think about effects and facilitate academic communication more easily via this frame, but we should be careful not to guide research from that frame. This is why I avoided vase examples early on – their prevalence seems like a symptom of an incorrect frame.
(Of course, I certainly did my part to make them more prevalent, what with my first post about impact being called Worrying about the Vase: Whitelisting...)
Notes