Curated. This sort of review work is crucial for making common records of what progress has been made, so thank you for putting in the work to make it.

Just a note that in the link that Wei Dai provides for "Relevant powerful agents will be highly optimized", Eliezer explicitly assigns '75%' to 'The probability that an agent that is cognitively powerful enough to be relevant to existential outcomes, will have been subject to strong, general optimization pressures.'

even if he doesn't it seems like a common implicit belief in the rationalist AI safety crowd and should be debunked anyway.

Agreed.

This is such an interesting use of a spoiler tags. I might try it myself sometime.

Huh? A lot of these points about evolution register to me as straightforwardly false. Understanding the theory of evolution moved us from "Why are there all these weird living things? Why do they exist? What is going on?" to "Each part of these organisms has been designed by a local hill-climbing process to maximise reproduction." If I looked into it, I expect I'd find out that early medicine found it very helpful to understand how the system was built. This is like me handing you a massive amount of code that has a bunch of weird outputs and telling you to make it work better and more efficiently, and the same thing but where I tell you what company made the code, why they made it, and how they made it, and loads of examples of other pieces of code they made in this fashion.

If I knew how to operationalise it I would take a pretty strong bet that the theory of natural selection has been revolutionary in the history of medicine.

I also thought so. I wondered maybe if Larks is describing that MacAskill incorporated Demski's comments-on-a-draft into the post.

I really like this post, it's really rich in new ideas, from using transparency tools to deliberately design ML systems, to how interpretability might scale, to trying to reorient the field of ML to more safe and alignable designs, and a bunch more detail.

I also think that trying to get someone else's worldview and explain it is a really valuable practice, and it certainly seems like Evan is learning to put on a number of really interesting and unique hats, which is great. Chris in particular has affected how I think a bunch about making scientific progress, with his writing about distillation and work at Distill.pub.

So I've curated this post (i.e. it moves to the top of the frontpage, and gets emailed to all users who've signed up for curation emails).

See my review here.

This post is close in my mind to Alex Zhu's post Paul's research agenda FAQ. They each helped to give me many new and interesting thoughts about alignment. 

This post was maybe the first time I'd seen a an actual conversation about Paul's work between two people who had deep disagreements in this area - where Paul wrote things, someone wrote an effort-post response, and Paul responded once again. Eliezer did it again in the comments of Alex's FAQ, which also was a big deal for me in terms of learning.

This post was on Hacker News for a while.

Nominating this primarily for Rohin’s comment on the post, which was very illuminating.