Pedagogical note: something that feels like it's missing from the fable is a "realistic" sense of how demons get created and how they can manipulate the hill. 

Fortunately your subsequent real-world examples all have this, and, like, I did know what you meant. But it felt sort of arbitrary to have this combo of "Well, there's a very concrete, visceral example of the ball rolling downhill – I know what that means. But then there are some entities that can arbitrarily shape the hill. Why are the demons weak at the beginning and stronger the more you fold into demon space? What are the mechanics there?

It's not the worst thing, and I don't have any ideas to tighten it. Overall I do think the post did a good job of communicating the idea it was aiming at.

Which you could round off to "biologists don't need to know about evolution", in the sense that it is not the best use of their time.

The most obvious thing is understanding why overuse of antibiotics might weaken the effect of antibiotics.

I guess the main thing I want is an actual tally on "how many people definitively found this post to represent their crux", vs "how many people think that this represented other people's cruxes"

Hmm, I am interested in some debate between you and Daniel Filan (just naming someone who seemed to describe himself as endorsing rationality realism as a crux, although I'm not sure he qualifies as a "miri person")

I just wanted to flag that this post hasn't been reviewed yet, despite being one of the most nominated posts. (And mosts of the nominations here are quite short).

The most obvious sort of review that'd be good to see is from people who were in this post's target demographic (i.e. people who hadn't understood or been unpersuaded about what sort of problem MIRI is trying to solve), about whether this post actually helped them understand that. 

I'd also be interested in reviews that grapple a bit more with "how well exactly does this metaphor hold up?", although I'm not sure how to go about that.

At the time I began writing this previous comment, I felt like I hadn't directly gotten that much use of this post. But then after reflecting a bit about Beyond Astronomical Waste I realized this had actually been a fairly important concept in some of my other thinking.

I think that at the time this post came out, I didn't have the mental scaffolding necessary to really engage with it – I thought of this question as maybe important, but sort of "above my paygrade", something better left to other people who would have the resources to engage more seriously with it.

But, over the past couple years, the concepts here have formed an important component of my understanding of robust agency. Much of this came from private in-person conversations, but this post is the best writeup of the concept I'm currently aware of.

One thing I like about this post is the focus on philosophical competence. Previously, I'd thought of this question as dangerous to think about, because you might make philosophical mistakes that doomed you or your universe for (in retrospect) silly reasons.

My current model is more like "no, you-with-your-21st-century-human-brain shouldn't actually attempt to take actions aiming primarily to affect other universes on the macro scale. Negotiating with other universes is something you do when you're a literal galaxy brain that is quite confident in it's philosophy."

But, meanwhile, it seems that:

(note: low-to-mid confidence, still working through these problems myself, and I very much still philosophically confused about at least some of this)

– becoming philosophically competent, as a species, may be one of the most important goals facing humanity, and how this project interfaces with AI development may be crucially important. This may be relevant to people (like me) who aren't directly working on alignment but trying to have a good model of the strategic landscape.

– a concept not from this particular post, but relevant, is the notion that the question is not "are you in a simulation or not?", it's more like "to what degree are you in simulations? Which distribution of agents are you making choices on behalf of?". And this has some implications about how you should make choices that you should worry about now, before you are a literal galaxy brain. (many of which are more mundane)

– I think there may be a connection between "Beyond Astronominal Waste" and "Robustness to scale." You can't be a galaxy brain now, but you can be the sort of person who would be demonstrably safe to scale up, in a way that simulations can detect, which might let you punch above your current weight, in terms of philosophical competence.

(5 upvotes from a few AF users suggests this post probably should be nominated by an additional AF person, but unsure. I do apologize again for not having better nomination-endorsement-UI.

I think this post may have been relevant to my own thinking, but I'm particularly interested in how relevant the concept has been to other people who think professionally about alignment)

A reminder, since this looks like it has a few upvotes from AF users: posts need 2 nominations to proceed to the review round. 

I'm not sure I understand the difference between this worldview and my own. (The phrase-in-italics in your comment seemed fairly integral to how I was thinking about alignment/capabilities in the first place).

This recent comment of yours seems more relevant as far as worldview differences go, i.e. 'if you expect discontinuous takeoff, then transparency is unlikely to do what you want'. (some slightly more vague "what counts as a clever argument" disagreement might be relevant too, although I'm not sure I can state my worry crisply, nor really confident my worry is cogent)

I don't have a strong position on the continuous/hard-takeoff debate and have updated a bit over the past year both on continuous-takeoff's plausibility as well as the value in shifting the AI field towards having clearer models of what they're building, generally. But insofar as I'm suspicious of this, it's mostly because I still put moderate probability on "some understanding here may be more dangerous than no understanding, precisely because it's enough to accomplish some things without accomplishing everything that you needed to."