The current bottleneck is political will, not research

17h

Abstract:

We already know enough to act. I wish we were in a world where research was the bottleneck, but the main constraint on AI safety is no longer a shortage of clever policy ideas: best practices already exist and are not being applied or enforced, and a serious international (or even just national) regulatory regime would probably cut most of the risk.
They are not applied because awareness is low. The people who narrate and enforce AI policy mostly do not believe the problem exists. I estimate that a majority of the top ~100–1,000 most influential policymakers worldwide have never had a single serious conversation about catastrophic risk, and this is the main reason they are not worried^[1]. Even among the civil-society organizations that showed up to the UN Global

...

Best practices already exist... the priority bottleneck is not finding more best practices

I'm pretty skeptical of this framing. It's not clear to me that current 'best practises' are sufficient or, stronger, whether the 80/20 framing holds.

That said, I suspect that a decent percentage of non top-tier researchers should pivot more towards advocacy assuming they meet a minimum proficiency bar for advocacy and that they have access to such opportunties.

Research note: A simpler AI timelines model predicts 99% AI R&D automation in ~2032

Thomas Kwa

5mo

This is a linkpost for https://metr.org/notes/2026-02-10-simpler-ai-timelines-model/

In this post, I describe a simple model for forecasting when AI will automate AI development. It is based on the AI Futures model, but more understandable and robust, and has deliberately conservative assumptions.

At current rates of compute growth and algorithmic progress, this model's median prediction is >99% automation of AI R&D in late 2032. Most simulations result in a 1000x to 10,000,000x increase in AI efficiency and 300x-3000x research output by 2035. I therefore suspect that existing trends in compute growth and automation will still produce extremely powerful AI on "medium" timelines, even if the full coding automation and superhuman research taste that drive the AIFM's "fast" timelines (superintelligence by ~mid-2031) don't happen.

Why make this?

The AI Futures Model (AIFM) has 33 parameters; this has 8.
- I previously

...

(Continue Reading - 2265 more words)

Thomas Kwa2d10

@elifland Here's some more thinking about uplift. The data is only suggestive right now, but I think it could be a useful input to forecasting if work is done to quantify the three problems I list: verbosity, distribution of highly useful vs barely useful code, and irrational time allocation.

AI 2040: Plan A

122

Daniel Kokotajlo, elifland, Thomas Larsen, romeo, bhalstead, ryan_greenblatt

This is a linkpost for https://www.ai-2040.com/

For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done!

It’s called AI 2040: Plan A.

It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to aim for.

It’s called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments.

As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia.

Read the scenario at ai-2040.com. You can...

(See More - 68 more words)

0Dan H2d

A key part of this strategy is deterrence, "Mutually Assured Compute Destruction" which gets its own section. It doesn't mention the generalization Mutually Assured AI Malfunction (MAIM) from Schmidt, Wang, and me last year. This also spends 1/3 of the MAIM discussion on verification and how to do this in a multilateral way. Meanwhile it cites other works like A Narrow Path. I even left this feedback to you all at AI Futures before this was released. This would constitute plagiarism in any other context. It's a bewildering unforced error--it's extremely related, it's a certainly a nontrivial idea, and I told you all this recently--I hope you all fix it.

Daniel Kokotajlo2d52

Sorry that was an oversight, we'll edit to include a footnote citing MAIM.

3Thomas Larsen2d

I'm excited about people commenting on this post with questions, feedback, critiques, different proposals, etc. We'll try to monitor it and respond to many of the comments.

4Chris_Leong3d

Total research transparency feels a bit too galaxy-brained for me. It makes non-robust assumptions that newly discovered techniques won't be usable to enhance already existing open-weights models to excessively dangerous capability levels. I also think the disincentive for research is overstated as it neglects first-mover advantage.

Notes on technical alignment via human-like social drives

Steven Byrnes

1. Frontmatter

1.1 Backstory for this post

As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.

After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would...

(Continue Reading - 12563 more words)

3Noosphere894d

I actually just no longer think this is true, and the reason this is true is because this only holds under risk-neutral or risk-seeking assumptions about it's values, and under risk-averse values like constant absolute risk aversion, the AI will almost never want to plot takeover and will instead cooperate with humans no matter what it's object level values are (so long as the cooperation probability is higher than the probability of a rebellion succeeding, and we can ensure that the probability of cooperation is very, very high, and later on scales with AI capabilities.) IMO, the best reading sources for this are Risk-Averse AIs by William MacAskill and Elliott Thornley, and my own (very long) comment about the details of it all. But to compress the insights that are important, it's that we can elicit AI capabilities for very cheap prices compared to a risk neutral AI (where in one example, it reduced the costs of getting a good alignment result from 2% of the universe to $1.99), and importantly we can do this on hard-to-verify subjects like alignment, and the reason we can safely delegate AIs to these tasks is that they care as much about reducing catastrophic risk as about eliminating risks, so they won't take over to reduce risk to 0 or eliminate variance, and risk-averse AIs are just very averse to plans that could fail, so they will accept smaller amounts of guaranteed resources over a plausible shot of takeover. In terms of what constraints it satisfies, it satisfies constraint 1, because unlike other alignment plans, it doesn't really have many weak points/brittleness to exploit, and it's either immune to or is reduced to a negligible problem like reward misspecification, malign generalization, and pointer problems/interpretability problems. And there's a known method to make AIs have the motivation that applies very, very generally to AIs trained with RL. It satisfies constraint 2, since with some more stuff that while not trivial to do, is basically re

6Steven Byrnes3d

Thanks! Sorry if you already said this, but are you assuming that everything the AI outputs is reliably checkable by humans? If yes, I’m skeptical about the “strategic constraints” of §6.1. Or if no, I don’t understand how risk-aversion helps. Let’s say Company A has its brand new risk-averse ASI in a box. The problem to be solved is: maybe some Company B somewhere on Earth will make a long-term-optimizing ruthless out-of-control ASI that then eats the world, in the near future. And my question is: How is Company A (or anyone else) supposed to solve this problem? One idea would be: Company A asks the ASI to print out a plan, and grades the plan based on how it sounds. Then (I claim) you get a good-sounding plan instead of a plan that will actually work (a.k.a. sycophancy). Alternatively, maybe Company A tells the ASI “You need to print out an actually good plan because we will punish you (e.g. empty your bank account) if some Company B successfully releases an evil ASI that eats the world”. This is based on actually results, not appearances, so we’re less worried about sycophancy. But the threat doesn’t make sense, because everyone would be dead by the time it would be carried out. I guess you could reply: “Hey actually it’s even better! We get that latter thing for free! The risk-averse ASI and the humans have a shared interest in preventing rogue ASIs from appearing on the scene and killing them all. The company doesn’t even have to make threats or offers, it doesn’t even have to say anything to the risk-averse ASI! Rather, the risk-averse ASI will just autonomously recognize that rogue ASIs are a risk and, being risk-averse, will work on solving the problem!” …But if the idea is that the risk-averse ASI will want to prevent rogue ASIs for its own self-preservation, then I think it would recognize that its prospects for preventing rogue ASI are much better if it escapes onto the internet and solves the problem autonomously, than if it merely talks to humans.

3Noosphere892d

The answer is no. [...] This turns out not to be correct, and a key reason for this is that risk-averse AIs under the proposal care a lot more about reducing the risk by say 50% with a 99.9999% chance of success compared to reducing risk by 100% with a 80% chance of success (the numbers are made up and don't matter much), or in slogan form, AIs care almost as much about reducing the probability of catastrophe as it does preventing them, and higher chances of mitigating catastrophe outweigh lower chances of completely preventing catastrophes. This is discussed much more in sections 8.5, 8.7, appendix B and appendix C if you want to read more. To be clear, I usually assume we are operating in a regime where humans more or less consent to the AIs solving problems like alignment autonomously, rather than humans giving any direction for this proposal. Put another way, AIs won't want to take over to reduce the risk to 0 if we can offer them a much more cheap option to reduce risk by say 80 or 90%, so fears of AI rebelling against humans to reduce risk to zero/remove variance don't work here. Indeed, one of the key difficulties I have seen when people try to grasp their heads around this proposal is not realizing that unlike earlier attempts, AIs trained with this motivation won't have the undesirable behavior of taking over to reduce risk/variance to effectively 0 or actually 0. Next up: [...] This turns out to not be true, and one of the reasons this isn't right is because we do have techniques like paying AIs later, and very critically the techniques that we do have work much, much better to elicit risk-averse AIs than risk-neutral AIs, because for risk-averse AIs, the utility of takeover is far, far more modest than for risk-neutral AIs, so current techniques work a lot better in the risk-averse regime. This is discussed much more in Section 4.2, including a worked example of how we have to pay AIs less by many OOMs in order to get the AIs to do good work eve

Steven Byrnes2d40

Thanks for patiently bearing with me even though I haven’t read the whole Forethought report.

Here’s what I got out of Appendix B.

Define the outcomes:

“ideal” is the risk-averse ASI becoming dictator of the universe,
“mitigated catastrophe” is the ASI getting $500 today and then getting crushed by a rogue ASI in the near future,
“unmitigated catastrophe” is the ASI getting $0 today and then getting crushed by a rogue ASI in the near future.
“good” is the ASI getting $500 today and not getting crushed by a rogue ASI in the near future.
“neutral” is the ASI gettin

... (read more)

Value generalisation: value correction

Stuart_Armstrong

I firmly believe that value generalisation^[1]is the key to AI Alignment. That, indeed, it is necessary and almost sufficient for alignment.

But I won't be arguing that grand point today; instead, I'll focus on a specific RL example of an agent that displays value correction: it realises its current reward function is (probably) incorrect, and acts to correct it.

Thus there are:

The initial situation, in distribution, where the human displays how to maximise the true reward.
The out of distribution situation where the agent finds a hack to exploit its reward function estimate, and turns against what we wanted it to do.
The value error detection stage where the agent realises that its reward function estimate is probably incorrect.
The value correction stage where the agent corrects its reward function back to

...

(Continue Reading - 1572 more words)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Why make this?

1. Frontmatter

1.1 Backstory for this post