This is such an interesting use of a spoiler tags. I might try it myself sometime.

Huh? A lot of these points about evolution register to me as straightforwardly false. Understanding the theory of evolution moved us from "Why are there all these weird living things? Why do they exist? What is going on?" to "Each part of these organisms has been designed by a local hill-climbing process to maximise reproduction." If I looked into it, I expect I'd find out that early medicine found it very helpful to understand how the system was built. This is like me handing you a massive amount of code that has a bunch of weird outputs and telling you to make it work better and more efficiently, and the same thing but where I tell you what company made the code, why they made it, and how they made it, and loads of examples of other pieces of code they made in this fashion.

If I knew how to operationalise it I would take a pretty strong bet that the theory of natural selection has been revolutionary in the history of medicine.

I also thought so. I wondered maybe if Larks is describing that MacAskill incorporated Demski's comments-on-a-draft into the post.

I really like this post, it's really rich in new ideas, from using transparency tools to deliberately design ML systems, to how interpretability might scale, to trying to reorient the field of ML to more safe and alignable designs, and a bunch more detail.

I also think that trying to get someone else's worldview and explain it is a really valuable practice, and it certainly seems like Evan is learning to put on a number of really interesting and unique hats, which is great. Chris in particular has affected how I think a bunch about making scientific progress, with his writing about distillation and work at Distill.pub.

So I've curated this post (i.e. it moves to the top of the frontpage, and gets emailed to all users who've signed up for curation emails).

See my review here.

This post is close in my mind to Alex Zhu's post Paul's research agenda FAQ. They each helped to give me many new and interesting thoughts about alignment. 

This post was maybe the first time I'd seen a an actual conversation about Paul's work between two people who had deep disagreements in this area - where Paul wrote things, someone wrote an effort-post response, and Paul responded once again. Eliezer did it again in the comments of Alex's FAQ, which also was a big deal for me in terms of learning.

This post was on Hacker News for a while.

Nominating this primarily for Rohin’s comment on the post, which was very illuminating.

I think the simple mathematical models here are very helpful in pointing to some intuitions about being confident systems will work even with major optimisation pressure applied, and why optimisation power makes things weird. I would like to see other researchers in alignment review this post, because I don't fully trust my taste on posts like these.

Reading this post was the first time I felt I understood what Paul's (and many others') research was motivated by. I think about it regularly, and it comes up in conversation a fair bit.