This is one of my favorite LessWrong posts ever (strong upvoted). Nevertheless, in this comment I'll provide some pushback; not to any the particular claims made, but around a sort of "missing mood" that I think a stronger version of this post could have addressed. (For context, I'm probably one of the lab employees who sometimes says things like "Current AIs are pretty aligned"—i.e. one of the people that Ryan is arguing against here, though I might not be a very central example.) For me, the value of this post was the following: 1. Articulating a specific way that current AIs are "misaligned." Namely: 2. 1. They follow heuristics that are well-tuned for strong task performance in easy-to-verify settings, but only try superficially at difficult-to-evaluate tasks. 2. They communicate in ways that misrepresent how much they've accomplished: punching up their successes while omitting or downplaying failures, and sometimes outright lying about what they did. But they don't seem to do this strategically or persistently, instead following simple heuristics that result in these sorts of misrepresentations. 3. The overall result is that they underperform their potential at difficult-to-evaluate tasks while misleading users about what they've actually accomplished. 4. I say "misaligned" in quotes, because many traditional notions of misalignment wouldn't have covered the above, especially 1(a). This will be relevant later. 3. Providing concrete examples and a sense of how prevalent the above issues are. I want to be clear that (2) was an update for me. I think the way that you (Ryan) use LLMs more reliably elicits the sort of misalignment you name, relative to most use cases I've personally seen. (Namely, it seems like you try to use LLMs to autonomously complete very difficult tasks using scaffolds that rely extensively on self-critique.) So after reading your post, I now think that current AIs are more misaligned (in the sense you describe) than I previously thought. That said, this update was more quantitative in nature than qualitative. I was certainly aware that current AIs sometimes display the behaviors you describe, even if I thought they were somewhat rarer than I now believe. And I don't think that I personally underrated the importance of these behaviors. (Though it's possible that others did.) [ETA: After thinking about this more, I think I probably was underestimating the importance of these behaviors before reading an early draft of Ryan's post.] Given that, what could I possibly mean when I call current AIs "pretty aligned"? I think there's a hypothetical dual post to yours someone could write titled "Current AIs seem pretty aligned to me." This post would name a bunch of types of misalignment that, a few years ago, seemed could plausibly occur in AIs as capable as current ones. For instance, it seems to me that: 1. Current AIs do not training-game, i.e. actively model the training process and take actions that result in high reward (including in training environments very different from ones they've previously encountered). 2. Current AIs do persistently and coherently pursue malign goals across many contexts, e.g. strategically seeking power. 3. Current AIs do not scheme, i.e. strategically subvert their training process, pre-deployment audits, and other types of oversight. 4. Current AIs do not strategically take steps to cover up their mistakes, and will generally admit to them when asked directly. 5. Current AIs generally don't try to strategically influence things in the real world beyond assisting with (or refusing) the task directly presented to them. E.g. they don't sneakily try to funnel money to causes or people they like by influencing unrelated conversations. Instead, I think the most important way that current AIs are misaligned is the way you describe in your post, Notably, I think the type of misalignment described in your post is "less scary" than the types I list above. The misalignment happens on the "learned heuristics" level, rather than on the "active"/"strategic"/"explicit" level. As a result, it's easier to detect and measure, and less likely to cause harms besides "making the AIs less useful than they could be for certain types of tasks." A related point is that the way that current AIs are misaligned is less "adversarial" than the sorts of misalignment I list above. When I speak to the people who IMO are extremely thoughtful about AI safety, they acknowledge something like "Yeah, TBH a few years ago, I thought that AIs as capable as current ones would be misaligned in some of those ways. There's a real positive update here." But people who are merely "quite" thoughtful sometimes instead: * argue that maybe current AIs are misaligned in some of these ways and we just can't tell * * I'm very skeptical of this! Anyone who thinks this should be trying really hard to produce compelling evidence in current models. * munge definitions to try to count the "misaligned heuristics" types of misalignment as one of these more "coherent" types * claim that they never expected AIs as capable as current ones to be misaligned in these ways * * I'm typically skeptical of this, though I believe it for some people. (A related point that I know how to state more vividly: Sometimes, people like to point at clear alignment issues in frontier models—for instance, sycophancy, AI psychosis, or whatever happened with Sydney/Bing chat—as being negative updates about alignment. But they don't like to count absence of these sorts of alignment issues as being positive updates. In other words, if it hypothetically ended up that AI psychosis were somehow a total media hoax, would the the people who updated negatively on it instead say "Nevermind! Actually, the absence of AI psychosis is a positive update on alignment." Are they currently saying this about the ways that current AIs are not misaligned but could have been?) Overall, my point is that I think current AIs are not misaligned in many of the ways that I—and I think many others—expected them to be. I also think that many people I talk to have not grappled with this observation sufficiently and updated their world models and prioritization accordingly. For instance, I think that the AI safety community is overinvested in scheming research relative to scalable oversight research. (TBC I've believed this for some years now, but I now believe it more confidently and think it's more clear that others should agree with me.) I also think that the AI safety community is underinvested in work on directly automating alignment research. (This is a place where I've changed my mind.) I'll close by emphasizing some things that I do not believe and don't mean to claim in this comment: 1. I don't think that the recent overall evidence has clearly been positive about misalignment risk. For instance, timelines are looking quite short now, which IMO is the main negative update to counteract the positive one I discuss in this comment. 2. I don't think these "scarier" types of misalignment are very unlikely to eventually arise in more capable AIs. Though I do think that we should update that they'll first arise in more capable AIs than we previously thought. This makes it more likely that we'll have human-level or superhuman automated alignment researchers prior to these "scarier" types of misalignment. 3. I don't think that the type of misalignment you name in your post is "not a big deal." In particular, I'm quite worried that "we"(=AI developers or humanity as a whole) bungle the situation because we failed to make AIs capable at difficult-to-evaluate tasks like alignment research or advising us about how to handle things during a rapid intelligence explosion. One possible way we could bungle the situation is to train more capable AIs that are misaligned in some of the "scarier" ways I listed (or in other ways I didn't list). 4. I don't think that current AIs are aligned enough to automate alignment research. (An earlier draft of your post operationalized "pretty misaligned" in this way, which I preferred.)

Raising awareness of AI risk led directly to the founding of OpenAI and Anthropic. Raising awareness of AI risk on the level of governments can easily backfire and lead to an arms race in the same way. The best and clearest statement of the problem and the only possible solution is Bill Hibbard's critique of SIAI back in 2003. It's almost unbelievable how much this text got right, 23 years ago: > 1. The SIAI analysis fails to recognize the importance of the political process in creating safe AI. > > This is a fundamental error in the SIAI analysis. CFAI 4.2.1 says "If an effort to get Congress to enforce any set of regulations were launched, I would expect the final set of regulations adopted to be completely unworkable." It further says that government regulation of AI is unnecessary because "The existing force tending to ensure Friendliness is that the most advanced projects will have the brightest AI researchers, who are most likely to be able to handle the problems of Friendly AI." History vividly teaches the danger of trusting the good intentions of individuals. > > The singularity will completely change power relations in human society. People and institutions that currently have great power and wealth will know this, and will try to manipulate the singularity to protect and enhance their positions. The public generally protects its own interests against the narrow interests of such powerful people and institutions via widespread political movements and the actions of democratically elected government. Such political action has never been more important than it will be in the singularity. > > The reinforcement learning values of the largest (and hence most dangerous) AIs will be defined by the corporations and governments that build them, not the AI researchers working for those organizations. Those organizations will give their AIs values that reflect the organizations' values: profits in the case of corporations, and political and military power in the case of governments. Only a strong public movement driving government regulation will be able to coerce these organizations to design AI values to protect the interests of all humans. This government regulation must include an agency to monitor AI development and enforce regulations. > > The breakthrough ideas for achieving AI will come from individual researchers, many of whom will want their AI to serve the broad human interest. But their breakthrough ideas will become known to wealthy organizations. Their research will either be in the public domain, done for hire by wealthy organizations, or will be sold to such organizations. Breakthrough research may simply be seized by governments and the researchers prohibited from publishing, as was done for research on effective cryptography during the 1970s. The most powerful AIs won't exist on the $5,000 computers on researchers' desktops, but on the $5,000,000,000 computers owned by wealthy organizations. The dangerous AIs will be the ones capable of developing close personal relations with huge numbers of people. Such AIs will be operated by wealthy organizations, not individuals. > > Individuals working toward the singularity may resist regulation as interference with their research, as was evident in the SL4 discussion of testimony before Congressman Brad Sherman's committee. But such regulation will be necessary to coerce the wealthy organizations that will own the most powerful AIs. These will be much like the regulations that restrain powerful organizations from building dangerous products (cars, household chemicals, etc), polluting the environment, and abusing citizens. To highlight the key part that I'd like people to take away, "only a strong public movement driving government regulation" has a chance of solving the problem. Attempts to influence governments that don't go through the public risk concentrating the power and will to change in the hands of governments, instead of where it should be: with the public.

If there was a textbook on building safe ASI with instructions that are sufficiently straightforward to execute, people would tend to build safe ASI rather than an extinction/disempowerment ASI. Some AI safety efforts could be thought of as contributions to this hypothetical textbook, making it marginally more real. > unsafe ASI is vastly easier to build than controlled ASI, and is on the same tech path The point of an ASI ban/pause is to create the time to reduce this gap, until it's sufficiently narrow that competent people can walk across without falling through. If an unsafe ASI is artificially delayed despite technological feasibility, there is time to write the textbook, to make safe ASI about as easy to implement. And similarly for some AI safety efforts that don't involve an ASI ban/pause, which attack the gap from the other end. (Scalable oversight agenda hopes AIs can write the textbook on their own, sufficiently quickly to win the race against the technical feasibility of unsafe ASI. I would feel a lot better about this plan if alignment of Mythos-level systems was pursued for 30 years before going further, and somehow there was a guarantee that Mythos instances won't be founding a country and declaring sovereignty in the meantime. This guarantee gets more believable if there are no Mythos-level systems at all yet.)