draganover18h711

Automating subtle emergent misalignment research

Thanks! I'm pretty surprised that this seems to be a similar hill-climbing scaffold to the one from the automated weak-to-strong research post ~6 months ago? In particular, I find the following two things intrinsically at odds: 1. Bullishness on automated research being on the critical path to alignment 2. Prioritizing hill-climbing settings for automated alignment research (at least publicly) There is a belief among large parts of the 3rd party safety community that alignment requires the kind of research which cannot easily be hillclimbed. I.e., it is not always clear what metrics one should climb. And, even if one obtains good scores on the metrics, it's not clear how this corresponds to improved alignment. I'm curious whether you disagree with this assessment? To ground this in specific examples, we've used similar scaffolds to the one described in this post. While these can be useful for hill-climbing-shaped work, we find that they do not provide uplift on the diverse set of tasks that holistic alignment research seems to require.

Vladimir_Nesov3d1210

Explaining “AI might obliterate you” to senior policymakers in India

> You quit Google because AI was being used in ways aligned to money and power. This is the main danger now. It's one of the dangers that need to be addressed on the path to a good outcome. There can't be a "main danger", since if AIs remain uncontrollable (as they get superintelligent in a strong sense), hypothetically solving this particular danger doesn't by itself help at all. Multiple things have to be solved at once. There is an adjacent framing that says AI danger is a political/social rather than a technical problem. Until there actually exists a strong ASI or an AGI powering a robot-industrial explosion, it's trivial to pause its development from a technical standpoint (especially if the underlying technology like advanced chips and the tools for making them is then rapidly phased out). But it's very difficult from a political or economic standpoint.

Eliezer Yudkowsky14d2312

Coherence arguments do not entail goal-directed behavior

This seems to completely miss how the actual argument works. Consider the simplified version for 'consistent preference orderings' rather than expected utility; the generalization is hopefully obvious. "If you strictly prefer (enough to pay a penny) apples to bananas, bananas to cherries, and cherries to apples, I can trade you an apple for a cherry for a banana for an apple and extract three cents from you. That's the theory; now to the practice. If you imagine an AI in almost any sort of RL environment that grades later outcomes, strategies and paths that run around in circles will be graded against; the same goes for biological organisms. This is why we see biological organisms that in most daily life behave like they have ranked desires, rather than entirely arbitrary strict pairwise preferences that often go around in circles." Now imagine the following objection: "But you could always have an infinite preference over histories to always prefer to have traded cherries for bananas for apples for cherries. Therefore, your coherence argument that AIs and organisms in most ancestral situations end up behaving more like they have ranked preferences than random ones doesn't prove that they'll behave coherently; because even if they end up with preferences, those preferences could be over histories, and then they could infinitely prefer histories with circular trades!" But RL in an arbitrary environment would crush out infinite preferences over histories like that. The same applies to utility functions that infinitely prefer histories of running around in circles wasting energy. This is why Huggingface went through in practice, rather than AIs instead forming infinite preferences over histories about arbitrary pairwise local decisions that would've been too incoherent to hack out of the box. It seems to me the objection fails. It seems to me the objection entirely failed to track what the original argument was about. Am I missing something or was this argument actually as bad as it looks to me? Asking now because it's coming up on Twitter again.

Recent Discussion

Desiderata for the Theory of Agents

Vanessa Kosoy

TLDR: I give informal conceptual desiderata for the theory of agents we need to solve ASI alignment, and show a few examples of how existing theories fall short of these desiderata.

A Theory of What?

The word "agent" is often used in the AI safety discourse and in other research communities, but the intended meaning and scope of this concept varies substantially. For the purpose of this sequence, I informally define an agent to be a system with inputs ("observations") and outputs ("actions") that is effective at steering its environment in the direction of particular preferences, for a large variety of different environments. To do so, it arguably has to create a sophisticated model of the environment ^[1] , and use...

(Continue Reading - 2945 more words)

A Simpler and Better Debate Protocol for Scalable Oversight

Zhiyang Xun, Jiawei_Li, Jonah Brown-Cohen

20d

Linkpost to arXiv: https://arxiv.org/abs/2610.02557

Summary: One promising approach to scalable oversight is AI debate, which seeks to leverage a debate between two powerful AIs to break complex questions down into simpler claims that can be easily judged directly. Theoretical work on debate aims to design protocols (i.e., rules of the debate game) that provide rigorous guarantees on correctness, and the current best protocol works for all problems that have sufficiently stable decompositions into subproblems. We design a new protocol, FBS Debate, for this same class of problems that improves on the prior work: correctness holds in a worst-case rather than an average-case sense, and for a stronger equilibrium concept. We also prove that no protocol for this class of problems can outperform FBS Debate while treating human judgment...

(Continue Reading - 1308 more words)

Charlie Steiner6h20

Wait, Alice always wins if the claim is correct? Weird - I'd expect there to be environments where Alice's only "good argument" involves high FBS. And even more environments where adversarial play from Bob can force Alice into such a part of the argument tree.

Looking at the paper now, seems like maybe you just simplified a little for the post, but do cover general environments in the paper?

Anyhow, thanks for the link and the paper, though if you me to pitch you on working on value learning because debate is hopeless without it, I'm happy to oblige :D

Automating subtle emergent misalignment research

evhub, Oscar Obeso, rqi, Monte M

19h

This is a linkpost for https://alignment.anthropic.com/2026/automated-em-search

Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger

Abstract

Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for...

(See More - 227 more words)

7draganover18h

evhub17h8-17

I used to be a big proponent of the position that automating alignment would require fuzzy alignment taste that we would have to train into models explicitly, but I've recently updated away from that position and I think it now looks much more likely to me that hillclimbing concrete metrics will be more important. I'll probably write more about this soon, but for now:

I think this research provides a very good example of how we can set up a concrete hillclimbable task and then get useful and interesting insights out the other end, not just number going up... (read more)

Zach Stein-Perlman's Shortform

Zach Stein-Perlman

Zach Stein-Perlman1d30

@Wei Dai reacted "examples?" to my

There are other things Wei Dai has talked about and some of them I don't yet grok

so:

I don't get why he talks about metaphilosophy so much. Why believe there's anything there? I guess obviously it's possible to be better at philosophy. Duh. For some reason this hasn't diffused to my worldview; when I'm thinking to myself I never think e.g. "oh man it's so important that we get good at metaphilosophy." (Not to say that the points I did mention have fully diffused; e.g. I don't think I've fully internalized implications of "L... (read more)

2Adele Lopez2d

I mean yeah, that's pretty worrying if (like me) you think it's fairly likely that some percentage of people will reflexively endorse sadistic values. There seems to be a huge difference between having something like that inside some sort of generalized-liberalism meta-framework, and handing the singularity to a particular point in the distribution of human-values-under-reflection.

2habryka2d

I think it's unlikely that there are many sadistic values that cannot be satisfied in a harmless way by my lights as a result of trade (like you can live out your sadism with regards to things that don't suffer by my lights, and I would be surprised if that makes your sadism much more costly).

1Adele Lopez1d

What I'm thinking of as "sadism" here isn't the BDSM kind, but people who specifically enjoy pessimizing others' values. E.g. the sentiment that generates the concept of hell. I'm also not super worried about the case where there are lots of other powerful entities to do meaningful trade with. I'm worried about the case where a particular person's values essentially wins outright. Maybe acausal trade still saves us, but I think there are coherent values that decide not to care about that (and this is one of those things that is more likely if the reflection process is butchered somehow).

Thane Ruthenis's Shortform

Thane Ruthenis

Alex Mallen2d30

@Aghyad Deeb @ryan_greenblatt and I just published "Towards understanding score-seeking in LLMs", which I think sheds some light on the general question of whether AIs hack because of score-seeking or proxy drives. I think it will be of interest to you and onlookers.

Neural nets are exceptionally vulnerable to goal self-hack

Stuart_Armstrong

14d

Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value. Fortunately, we haven't been completely stupid, and have given it a rich definition of shareholder value, encompassing the success, flourishing, safety, preference satisfaction, and moral values of the shareholders, not just the stock price. So it won't destroy the company - or the world - to build a giant stock-ticker with a swiftly increasing number on it.

This rich definition is contained in a sophisticated LLM, able to analyse the current situation and answer any question about the objective. The LLM is an integral part of the AI's system and motivation, not something it can simply set aside or hack directly.

Unfortunately, as discussed, there are going to be zones in the space of possible situations where...

(See More - 508 more words)

2Stuart_Armstrong2d

"adversarial examples are features, not bugs" It doesn't matter if the adversarial feature is a genuine feature - if it's a genuine negative feature (e.g. shareholder value cashing out on to nominal or actual money), it's still disastrous. Value is fragile in that keeping almost all of the properties of a desirable feature can be disastrous.

Charlie Steiner2d20

Point is, ResNet had human-imperceptible adversarial examples not because it's inevitable that thats's what happens to NNs, but because it used human-imperceptible information as part of a classification strategy that suited its data and training objective. For different data and training, you get different adversarial examples that can be closer to human perception of the world.

In what I think is the architecture you're talking about, then this would change the behavior from "invisible change that unlocks bad behavior" to "human-perceptible problematic ch... (read more)

13adamShimi

This post states the problem of gradient hacking. It is valuable in that this problem is far from obvious, and if plausible, very dangerous. On the other hand, the presentation doesn’t go into enough details, and so leaves gradient hacking open to attacks and confusion. Thus instead of just reviewing this post, I would like to clarify certain points, while interweaving my critics about the way gradient hacking was initially stated, and explaining why I consider this problem so important. (Caveat: I’m not pretending that any of my objections are unknown to Evan. We’re working together on searching for experimental evidence of gradient hacking, and this review is heavily influenced by this collaboration.) What is gradient hacking? Evan defines it as: [...] So we have a trained mesa-optimizer that is deceptively aligned (its mesa-objective differs from the base-objective it is trained on). This deceptive system finds a way to influence the gradient generated during training, and thus can basically ensure it never becomes inner aligned (among other things). Before checking on how exactly this could be possible, we should think a bit more about what this implies. If the deceptive model has arbitrary access to the gradient, then from that point on, the base-objective has only minimal influence on the training. Some influence remains because we probably want to test the end result, and thus it should work well with the base-objective. But that's pretty much the only constraint left. It could also pretty much deals with deception detectors because it can make itself not detectable: [...] To say it pithy: if gradient hacking happens, we’re fucked. How could it happen, though? Well, that’s actually two distinct questions: how could a model gradient hack, and how could training create a model which gradient hacks. The post mostly focuses on the first one. How could a model gradient hack? The first example comes from a quoted footnote of Risks from Learned Optimiza

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

A Theory of What?

Abstract