All of walking_mushroom's Comments + Replies

I find this perspective interesting (and confusing), and want to think about it more deeply. Can you recommend reading anything to have a better understanding of what you're thinking, or what led you to this idea in specific?

Beyond the possible implications you mentioned, I think this might be useful in clarifying the 'trajectory' of agent selection pressure far from theoretical extremes that Richard Ngo mentioned in "agi safety from first principles" sequence. 

My vague intuition is that successful, infectious memes work by reconfiguring agents to shift from one fix point in policy to another while not disrupting utility. Does that make sense?

2Daniel Kokotajlo2y
Thanks! Excellent point about the connection to the trajectory of agent selection pressure. I don't remember what led me to this idea in particular. I've been influenced by doing a lot of thinking about agent foundations and metaethics and noticing the ways in which humans don't seem to be well modelled as utility maximizers or even just any sort of rational goal-directed agents with stable goals. I also read the book "The Meme Machine" and liked it, though that was after writing this post, not before, IIRC. I don't know what you mean by fixed points in policy. Elaborate?