Very nice people don’t usually search for maximally-nice outcomes — they don’t consider plans like “killing my really mean neighbor so as to increase average niceness over time.” I think there are a range of reasons for this plan not being generated. Here’s one.

Consider a person with a niceness-shard. This might look like an aggregation of subshards/subroutines like “if person nearby and person.state==sad, sample plan generator for ways to make them happy” and “bid upwards on plans which lead to people being happier and more respectful, according to my world model.” In mental contexts where this shard is very influential, it would have a large influence on the planning process.

However, people are not just made up of a grader and a plan-generator/actor — they are not just “the plan-generating part” and “the plan-grading part.” The next sampled plan modification, the next internal-monologue-thought to have—these are influenced and steered by e.g. the nice-shard. If the next macrostep of reasoning is about e.g. hurting people, well — the niceness shard is activated, and will bid down on this. 

The niceness shard isn’t just bidding over outcomes, it’s bidding on next thoughts (on my understanding of how this works). And so these thoughts would get bid down, and the thought being painful to consider leads to a slight negative reinforcement event. This means that violent plan-modifications are eventually not sampled at all in contexts where the niceness shard would bid them away.

So nice people aren’t just searching for “nice outcomes.” They’re nicely searching for nice outcomes.

(This breaks the infinite regress of saying “there’s a utility function over niceness, and preferences over how to think next thoughts about niceness, and meta-preferences about how to have preferences about next thoughts...” — eventually cognition must ground out in small computations which are not themselves utility functions or preferences or maximization!)

Thanks to discussions with Peli Grietzer about this idea of his (“praxis values”). Praxis values involve “doing X X-ingly.” I hope he publishes his thoughts here soon, because I’ve found them enlightening.