I'm an admin of this site; I work full-time on trying to help people on LessWrong refine the art of human rationality.
Longer bio: www.lesswrong.com/posts/aG74jJkiPccqdkK3c/the-lesswrong-team-page-under-construction#Ben_Pace___Benito
I listened to this yesterday! Was quite interesting, I'm glad I listened to it.
I expect the examples Ajeya has in mind are more like sharing one-line summaries in places that tend to be positively selected for virality and anti-selected for nuance (like tweets), but that substantive engagement by individuals here or in longer posts will be much appreciated.
Thank you, they were all helpful. I'll write more if I have more questions.
("sadly that's unprobable to work" lol)
Thank you, those points all helped a bunch.
(I feel most resolved on the calibration one. If I think more about the other two and have more questions, I'll come back and write them.)
I made notes while reading about things that I was confused about or that stood out to me. Here they are:
Sh*t. Wow. This is really impressive.
Speaking for myself, this (combined with your orthodox case against utility functions) feels like the next biggest step for me since Embedded Agency in understanding what's wrong with our models of agency and how to improve them.
If I were to put it into words, I'm getting a strong vibe of "No really, you're starting the game inside the universe, stop assuming you've got all the hypotheses in your head and that you've got clean input-output, you need far fewer assumptions if you're going to get around this space at all." Plus a sense that this isn't 'weird' or 'impossibly confusing', and that actually these things will be able to make good sense.
All the details though are in the things you say about convergence and not knowing your updates and so on, which I don't have anything to add to.
(I can't see your distribution in your image.)
For example, a main consideration of my prediction is using the heurastic With 50% probability, things will last twice as long as they already have, with the starting time of 1956, the time of the Dartmouth College summer AI conference.
A counter hypothesis I’ve heard (not original to me) is: With 50% probability, we will be half-way through the AI researcher-years required to get AGI.
I think this suggests much shorter timelines, as most researchers have been doing research in the last ~10 years.
It's not clear to me what reference class makes sense here though. Like, I feel like 50% doesn’t make any sense. It implies that for all outstanding AI problems we’re fifty percent there. We’re 50% of the way to a rat brain, to a human emulation, to a vastly superintelligent AGI, etc. It’s not a clearly natural category for a field to be “done”, and it’s not clear which thing counts as ”done” in this particular field.
Comment here if you have technical issues with the Elicit tool, with putting images in your comments, or with anything else.
Here's my quick forecast, to get things going. Probably if anyone asks me questions about it I'll realise I'm embarrassed by it and change it.
It has three buckets:
10%: We get to AGI with the current paradigm relatively quickly without major bumps.
60%: We get to it eventually sometime in the next ~50 years.
30%: We manage to move into a stable state where nobody can unilaterally build an AGI, then we focus on alignment for as long as it takes before we build it.
Adele Lopez is right that 30% is super optimistic. Also I accidentally put a bunch within '2080-2100', instead of 'after 2100'. And also I thought about it more. here's my new one.
It has four buckets:
20% Current work leads directly into AI in the next 15 years.
55% There are some major bottlenecks, new insights needed, and some engineering projects comparable in size to the manhattan project. This is 2035 to 2070.
10% This is to fill out 2070 to 2100.
15% We manage to move to a stable state, or alternatively civilizational collapse / non-AI x-risk stops AI research. This is beyond 2100.