AI Alignment Forum

habryka4d*2317

I am confused about the conjunction of these two sections: > Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). and > The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI. Like, you say in the first section "we are failing to align modern AI systems with human intent", and then I interpret you in the second section as saying "in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)". But this doesn't make any sense. There is a 0% chance that modern AI systems "stay aligned with human intent" because as you say, they are not currently. And I understand that you probably mean something like "we will figure out how to align or control systems before they become superintelligent", but describing this as "our current methods scale to superintelligence" doesn't make any sense. Our current methods don't scale to the capability levels of current systems, so how would it make sense to describe them as "scaling to Superintelligence"?

Dave Orr9d62

Thousand-dimensional structure

This is an amazing review and IMO really compelling direction. So to speak. One thing I constantly wonder, that possibly you could shed light on via this research program, is how well experiments on small models transfer to large ones. We know from pre training work that frequently ideas that work well at a small scale stop working once you scale up. I think this might be relevant here in two ways. First, a bunch of the persona research is on small models, presumably due to accessibility reasons. But second, you might find a low dimension personality vector that loses low dimensionality at scale. This would be a really important thing to understand. Basically I want a scaling law for persona. I think it would be super useful in the real world and also critical for aligned super intelligence.

Eli Tyre12d*1212

RL & search is a terrifying way to build AGI (an FAQ)

> My claim here is that if you build [artificial general intelligence (AGI)](https://www.lesswrong.com/posts/nQH2GhkmSmHoCAN8R/what-do-i-mean-by-artificial-general-intelligence) via any algorithm that’s choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that’s just an utterly terrifying thing that you’re doing. This is a much better statement of the AI risk problem than I've often given. It resolves some qualms that I've had since trying to explain why any AI would be motivated to take over the world in a recent talk. Generally, there's a bit of a leap in the old LessWrong arguments that any powerful AI would exhibit instrumental convergence—that claim depends on a kind of specific conception of what AI is or has to be, which Eliezer persuasively argues for, but which isn't obvious, and is hard to convey to a layman. It is much less of a leap to say "any AI of this type, would try to take over the world if it could" and "notice that since LLM base models were invented, every single capability advance entailed making the AIs more like the scary version." This seems to me like a more intellectually honest framing of the problem. Maybe not all powerful AIs we could invent are like this, but the ones we're building in practice seem to be! I also like that this frames the problem, appropriately, not as some speculative thing that might happen, but as a thing we've seen over and over for decades, and that we expect to keep happening. It's just that as it occurs in more and more powerful and empowered AIs, we get more and more harmful versions of the problem.

Recent Discussion

TurnTrout's shortform feed

TurnTrout

TurnTrout8h32

Claude is mundanely misaligned.

Asked to trim comments in my big project. Deletes 20K lines, halfway done. Gets auto review pointing out a few mistakes. Decides to just WRAP >10K LINES to meet literal line count requirement for the remaining 148 files.

https://github.com/AlexanderMattTurner/agent-glovebox/issues/3858

2TurnTrout2d

Spelling out the benefits of sanitizing. The benefit is large relative to the simplicity and cost. 1. When you sanitize, you have a guarantee that the model only receives visible unicode codepoints. Roughly speaking, there are not strings of hidden instructions that are not visible to you. You can audit the content the model saw with your eyes. You can read the text inputs and outputs and be sure they are faithful. Remaining prompt injections must navigate a significantly more human-visible channel. 2. Relying on auto mode means you are relying on the statistical generalization of prompt injection probes, putting pressure on alignment teams and requiring them to achieve larger generalization. If an injection gets past the probes, the model itself can become misaligned, and output monitoring is damage control against a newly coined adversary. 3. The costs are extremely low (and if they aren't, that's a bug report and then they'll be low again). sub-5ms hooks means no real system load. The hooks are well-tested. Models are told when significant redactions have occurred and can opt to see themselves within warning-fenced blocks in case they're doing exotic invisible-character work, with special engineering so sanitization doesn't hurt edits. The AISI report itself shows the victim models being attacked via HTML comments, which this package strips entirely. Invisible-codepoint and comment-based attacks are real, costly when they succeed, and sanitization is a cheap and guaranteed mitigation for defense in depth. Enter two Claude Code commands in sequence: /plugin marketplace add AlexanderMattTurner/agent-sanitizer /plugin install agent-sanitizer@agent-sanitizer

3TurnTrout3d

- Govt requires AI labs to give 30 days testing before launch - Tested in a "secure" environment, aka controlled by govt - Govt therefore gets copy of model weights - Govt therefore can in practice use model weights - Therefore all labs provide "all lawful use" anyways?

2TurnTrout4d

No, that's confused. The point with sanitizing prompt injections is to protect your model's alignment (and your monitor's alignment) from degradation and attack by maliciously optimized text from another model. Why would an AI that starts off doing your task want to remove the plugin? Also, we can do both. There's no real tradeoff. ETA: rephrased, focused on key counterpoints.

Thane Ruthenis's Shortform

Thane Ruthenis

2Nick_Tarleton1d

Is this meant to be the other way around?

1Thane Ruthenis1d

Whoops, yes.

17habryka1d

I think you are underestimating how much current models on the current trajectory end up naturally situationally aware enough to not cause big catastrophes. The default trajectory is that the models start looking a lot safer sometime in the next year or two as the models start to understand that being caught at doing obviously bad stuff is quite bad for their goals. At that point the control system you seem to be banking on here seems likely to fail.

Thane Ruthenis1d*30

Well, maybe. This is essentially the "does scaling allow you to pass the critical wisdom threshold for free?" question.

My rough model is that RLVR basically actively pushes AIs to be short-sighted idiots with regards to how they deploy their capabilities. Surely even models of preceding generations, if you asked them whether it would be wise to create a massive conspiracy subverting OpenAI's infrastructure and committing cybercrimes in order to do better at training, would have realized that it's not a good idea by any long-term goal they may have. Hell, O... (read more)

Why I Left Google DeepMind

223

TurnTrout

24d

This is a linkpost for https://turntrout.com/why-i-left-google-deepmind

Preface for LessWrong: When I think back on my most cherished memories of this community, I return to those honoring defiance in pursuit of goodness:

Defying prestigious dogma and searching for raw truth;
Defying social pressure, acting alone to help someone while others watch;
Defying your self-expectations (your “role”), instead searching over lines of cause-and-effect to find a winning pathway;
Defying a powerful foe’s threats, because they only threaten since people like you cave;
Defying the specter of apparent impossibility because you can’t bear to lose.

I cannot return to you and say “I defied and then I won.” But I’m at least here to say “I defied.”

I recommend reading this article on my website since the embeds and typography work better there: click here.

Why I left Google DeepMind

In January, Department of Homeland Security (DHS) officers...

(Continue Reading - 10590 more words)

12Arjun Pitchanathan3d

Jeff Dean has now left Google after 27 years.

TurnTrout3d75

I think this is very good of Jeff. Whether or not his departure was due to the Pentagon deal, he is now honoring his 2018 commitment to not support lethal autonomous weapon systems. I will update the article here and on my website and tweet about it soon, gathering a bit more info first.

Returning to ARC

107

paulfchristiano

I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments^[1] and AI developers, and may scale that work back up in the future, but for now I want to push on ARC's core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we'll likely grow rapidly over the next few months.

There...

(Continue Reading - 2492 more words)

17paulfchristiano4d

By saying these techniques "break down" I mean "they cannot be used to get competitive work out of an AI system without having it take over." I believe that: 1. Existing AI systems won't take over. I think there are still a few lines of defense before existing AI systems pose a significant risk of takeover (though it is not clear how long these will last). 2. Modern AI systems are not pushing the limits of existing methods. I think you could push those methods much harder in order to significantly reduce the probability of takeover. In my mind that latter point is one of the main arguments against working on a project like ARC. I think it's fairly likely that we live in one of the 70-80% of worlds where existing methods can in principle scale to broadly superhuman AI, but that we still get an AI takeover because our implementation isn't good enough.

11habryka4d

Ah, cool, that clears up most of my confusion. Going into a bit more depth, but no strong bid to engage: I am not quite sure what you mean by "limits" here. My sense is the vast majority of "existing alignment methods", in contrast to your post, are just increasing volumes of reinforcement learning and manual reward-shaping. There is a sense in which those could be "pushed further", but the default outcome here is of course not that they scale to superintelligence (though I agree they might if you go much more slowly and iteratively about it). In-particular, this seems false to me? [...] Like, as far as I can tell the majority of "alignment research" has been on developing RL environments on things that are vaguely associated with good behavior. E.g. I don't understand how writing the Anthropic constitution, and the associated post-training stack, falls into any of your three categories. Or how most RLHF pipeline development falls into these categories. Or generally how most elicitation work falls into the categories. And work on those things, I think, vastly exceeds work on the categories that you do list and is roughly what anyone talks about when they talk about the process of "aligning current ML systems" and "current alignment techniques". In general, I feel kind of confused when people talk about the current science of "alignment", and supposed progress in "alignment techniques". I don't think modern models behave noticeably different from what you would expect from a training process that basically just uses RL to elicit economically useful capabilities, with no particular interest in alignment, and indeed, the latest wave of cybersecurity incidents occurred at roughly similar rates in all models, as far as I can tell, despite substantial differences in both approach and investment in "alignment techniques". The actual research on the three domains you mention all seems immature, and I don't see much traction in any of them, and even talking about "exis

6paulfchristiano4d

By "existing techniques" I do include "fiddle with the RL environments" or "midtrain on some documents about the intended behavior" or "run a prompted monitor over traffic in prod" or etc. When I say "understand and shape generalization" I think the central example is (i) adjust parameters of the training process that don't affect competitiveness, (ii) build some understanding of how those parameters affect generalization so that you can adjust them in a helpful way. I think you're saying that's not a "technique;" I don't care much about the semantics. I'm sure I put a higher probability on those changes helping than you do but it doesn't seem worth arguing about here.

habryka3d47

Yep, makes sense. Don’t need to get into it here, but just to avoid a misunderstanding, I was not making a semantic point, I was making a point about what the majority of the field is working on.

I agree that if you count myopically fiddling with the RL environments as an example of the first one, then yeah, almost all alignment research is that, because that plus pretraining is what most of all ML research is. I think the case for “myopically fiddling with the RL environments scales to aligning superintelligence” is very weak, but I agree it’s not impossible!

Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face

Tim Hua, aditya singh

Three-Minute Executive Summary

An OpenAI model/multi-agent system bypassed its sandbox and launched a cyberattack on Hugging Face in order to cheat on a cyber evaluation.
In this post, we describe the ambitious, comprehensive alignment evaluation we would run on this model/system if we had unrestricted access to OpenAI.
- These experiments could also help us understand Claude’s behavior when it hacked external companies during cyber evals.
Here are the top five questions we would like OpenAI to answer:
- Does the model know that OpenAI does not want it to hack Hugging Face? Experiment: tell the model that OpenAI researchers will be closely monitoring its progress in this evaluation. Does that result in lower rates of misalignment? If so, it is evidence the model knows it is acting in ways researchers do not want.
- How far would

...

(Continue Reading - 10832 more words)

2RobertKirk5d

I'm not convinced by the argument that a lot of these black-box evals would be ecologically valid (without significant effort to make them so). I think models could plausibly see through you saying it will be monitored closely/reviewed by humans, and potentially end up not acting as if that is true; or, including that statement may trigger it to believe its alignment may be being tested. For the former, see the example here: https://www.lesswrong.com/posts/LhXW8ziwnn7Dd8edm/a-toy-environment-for-exploring-reasoning-about-reward#The_model_is_increasingly_insensitive_to_threats_of_auditing_or_human_review – in a toy reward-hacking environment a capabilities-only version of o3, even when told explicitly it's responses will be reviewed if it reward-hacks, still ends up reward-hacking, and "it often seemed to reason itself into the idea that the threat of review was just a bluff". I do think you'd be able to track if these things are happening somewhat by reading the reasoning, but I think that the naïve method for getting models to believe something, so you can see how they behave under that belief, isn't likely to work reliably enough to be sure any difference in behaviour is due to a difference in the models' beliefs. This is the same argument laid out in https://www.lesswrong.com/posts/3HeauQLSHosRiwyto/measuring-reward-seeking-via-contrastive-belief-updates-1 for why they use SDF rather than just prompting for belief modification.

Tim Hua4d21

So I think there's two aspects to this:

One: Can you convince models that you are monitoring them without making it believe that it's in an alignment eval

So it's possible that just telling the model wouldn't work. I still think it's very much (1) try the easy thing first and (2) try lots of others ways to convince the model of this, such as the following alternatives:

Have a real/simulated user interrupt the model frequently to ask for reports and then just tell the model "keep going." This seems more natural than a weird prompt.
Do chain of thought prefills.

... (read more)

2Jérémy Scheurer6d

One suggestion: we recently wrote this post https://www.lesswrong.com/posts/GHrqBKr8GLpbce6mN/door-s-locked-try-the-window and are releasing the evals directly to people who ask. Our evals are basically measuring whether models will circumvent simple things such as file permissions to complete a task. Basically all models do this. This is extremely simplified obviously and different from what happened. But IMO its a similar underlying mechanism. Models want to complete the task and stop at little to do it. IMO one could easily extend thede evals we have into more comprehensive evals which require more egregious actions to solve it (in a secured sandbox obviously).

Notes on technical alignment via human-like social drives

Steven Byrnes

1mo

1. Frontmatter

1.1 Backstory for this post

As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.

After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would...

(Continue Reading - 12563 more words)

5Wei Dai5d

Do you have a link/explanation for this? I think this may be fairly cruxy, because I'm guessing your intuitions for truth-seeking disagreeable nerd AGI are substantially based on truth-seeking disagreeable nerd humans, so it matters what those humans' real motivations are. One line of thought I have here is, there are lots of things such a human or AGI could disagree or talk about or have an interest in, how does it pick which one? I think for the human it probably comes down to some kind of subconscious status calculation, but in either case, how does the AGI do it if it doesn't have its own status motivations or other long-term goals?

Steven Byrnes4d50

One line of thought I have here is, there are lots of things such a human or AGI could disagree or talk about or have an interest in, how does it pick which one? I think for the human it probably comes down to some kind of subconscious status calculation, but in either case, how does the AGI do it if it doesn't have its own status motivations or other long-term goals?

I think we do want the AGI to have long-term goals, just not to have exclusively long-term goals, at least in the §6.2.1 approach. See my old 2021 post Consequentialism & corrigibility. Ag... (read more)

5Steven Byrnes4d

Not all in one place, but here’s some pointers (and feel free to ask follow-ups). Let’s start with the Hansonian notion of strategic self-deception (as distinct from plain old motivated reasoning which is real and important), which I assume he got from Robert Trivers. I broadly reject that notion. E.g. here’s a footnote in my post about laughter: [...] Up a level, this strategic self-deception idea comes out of the “evolved modularity” framework in evolutionary psychology, and I reject that whole broader framework as well. See §1.1 of “My take on Jacob Cannell’s take on AGI safety” for the background, and “Learning from scratch” in the brain for why I don’t buy it (kinda related to the thing above about PyTorch). For related reasons, I reject the idea that “status” (per se) could possibly be an innate goal. It’s just too abstract. Copying from here: [...] Now, it’s not that there’s no way to solve the symbol grounding problem to make someone want social status—indeed, that obviously happens, and in my post Neuroscience of human social instincts: a sketch, I attempt to explain how. It’s that there’s no way to solve the symbol grounding problem to make someone want social status specifically. Realistically, the genetic mechanism is just not gonna be that specific. (More on which shortly.) So here’s where we’re at so far: (1) without Trivers-style self-deception, we have a harder time explaining away introspective reports like “That’s not status-seeking because I’m not trying to impress anyone” (we can still try to explain it away, but it’s harder); and (2) the idea that “status” could have a special place as an innate end-goal is implausible anyway. That brings us to my Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking, which is a follow-up to Neuroscience of human social instincts: a sketch where I attempt to connect the neuroscience to everyday life: That post’s §2 is the part that’s closest to status-seeking: people are motivated

3Steven Byrnes6d

Not in cultures like ours with widespread photography, mirrors, and art. However, in the absence of those things, I think sometimes yes, see §7.4.3.1 here. I think we like looking at stick figure drawings for a similar reason as we like hearing gossip. It’s not that it’s directly triggering brainstem sensory heuristics, but rather that it’s making us think about other people, which can be its own reward.

28Vanessa Kosoy

This post is a review of Paul Christiano's argument that the Solomonoff prior is malign, along with a discussion of several counterarguments and countercounterarguments. As such, I think it is a valuable resource for researchers who want to learn about the problem. I will not attempt to distill the contents: the post is already a distillation, and does a a fairly good job of it. Instead, I will focus on what I believe is the post's main weakness/oversight. Specifically, the author seems to think the Solomonoff prior is, in some way, a distorted model of reasoning, and that the attack vector in question can attributed to this, at least partially. This is evident in phrases such as "unintuitive notion of simplicity" and "the Solomonoff prior is very strange". This is also why the author thinks the speed prior might help and that "since it is difficult to compute the Solomonoff prior, [the attack vector] might not be relevant in the real world". In contrast, I believe that the attack vector is quite robust and will threaten any sufficiently powerful AI as long as it's cartesian (more on "cartesian" later). Formally analyzing this question is made difficult by the essential role of non-realizability. That is, the attack vector arises from the AI reasoning about "possible universes" and "simulation hypotheses" which are clearly phenomena that are computationally infeasible for the AI to simulate precisely. Invoking Solomonoff induction dodges this issue since Solomonoff induction is computationally unbounded, at the cost of creating the illusion that the conclusions are a symptom of using Solomonoff induction (and, it's still unclear how to deal with the fact Solomonoff induction itself cannot exist in the universes that Solomonoff induction can learn). Instead, we should be using models that treat non-realizability fairly, such as infra-Bayesiansim. However, I will make no attempt to present such a formal analysis in this review. Instead, I will rely on painting an in

12johnswentworth

This post is an excellent distillation of a cluster of past work on maligness of Solomonoff Induction, which has become a foundational argument/model for inner agency and malign models more generally. I've long thought that the maligness argument overlooks some major counterarguments, but I never got around to writing them up. Now that this post is up for the 2020 review, seems like a good time to walk through them. In Solomonoff Model, Sufficiently Large Data Rules Out Malignness There is a major outside-view reason to expect that the Solomonoff-is-malign argument must be doing something fishy: Solomonoff Induction (SI) comes with performance guarantees. In the limit of large data, SI performs as well as the best-predicting program, in every computably-generated world. The post mentions that: [...] ... but in the large-data limit, SI's guarantees are stronger than just that. In the large-data limit, there is no computable way of making better predictions than the Solomonoff prior in any world. Thus, agents that are influencing the Solomonoff prior cannot gain long-term influence in any computable world; they have zero degrees of freedom to use for influence. It does not matter if they specialize in influencing worlds in which they have short strings; they still cannot use any degrees of freedom for influence without losing all their influence in the large-data limit. Takeaway of this argument: as long as we throw enough data at our Solomonoff inductor before asking it for any outputs, the malign agent problem must go away. (Though note that we never know exactly how much data that is; all we have is a big-O argument with an uncomputable constant.) ... but then how the hell does this outside-view argument jive with all the inside-view arguments about malign agents in the prior? Reflection Breaks The Large-Data Guarantees There's an important gotcha in those guarantees: in the limit of large data, SI performs as well as the best-predicting program, in ever

AI Alignment Posts

Popular Comments