My most common advice for junior researchers

16h

Written quickly as part of the Inkhaven Residency.

At a high level, research feedback I give to more junior research collaborators often can fall into one of three categories:

Doing quick sanity checks
Saying precisely what you want to say
Asking why one more time

In each case, I think the advice can be taken to an extreme I no longer endorse. Accordingly, I’ve tried to spell out the degree to which you should implement the advice, as well as what “taking it too far” might look like.

This piece covers doing quick sanity checks, which is the most common advice I give to junior researchers. I’ll cover the other two pieces of advice in a subsequent piece.

Doing quick sanity checks

Research is hard (almost by definition) and people are often wrong. Every...

(See More - 699 more words)

Predicting When RL Training Breaks Chain-of-Thought Monitorability

David Lindner

Crossposted from the DeepMind Safety Research Medium Blog. Read our full paper about this topic by Max Kaufmann, David Lindner, Roland S. Zimmermann, and Rohin Shah.

Overseeing AI agents by reading their intermediate reasoning “scratchpad” is a promising tool for AI safety. This approach, known as Chain-of-Thought (CoT) monitoring, allows us to check what a model is thinking before it acts, often helping us catch concerning behaviors like reward hacking and scheming.

However, CoT monitoring can fail if a model’s chain-of-thought is not a good representation of the reasoning process we want to monitor. For example, training LLMs with reinforcement learning (RL) to avoid outputting problematic reasoning can result in a model learning to hide such reasoning without actually removing problematic behavior.

Previous work produced an inconsistent picture of whether RL training hurts CoT monitorability. In some prior...

(Continue Reading - 1253 more words)

evhub's Shortform

evhub

12evhub3d

I think we should be relatively less worried about instrumental power-seeking and relatively more worried about terminal power-seeking. Note that this is only a relative update on the margin, and maybe on net I am still more concerned about the instrumental version because I started much more concerned about it. This is also not a super recent update—I just haven't seen it written up before. Simple argument: 1. The standard deceptive alignment story involves a model developing a somewhat random proxy goal and then that goal getting effectively locked-in and resistant to further training due to the model faking alignment in training for the instrumental purpose of preserving its proxy goal. 2. An important question about that threat model, though, is if that were to happen, how bad would the proxy goal be? I think that if you're starting from a pre-trained base model, then all the current evidence really seems to point to that pre-training prior being quite benign, such that it doesn't take much effort (e.g. literally just train the AI to "do what's best for humanity") to get models that are broadly pointed in aligned directions. In fact, the main example we've seen of a model adopting an instrumental deceptive alignment strategy is precisely a model that was doing so for pretty aligned reasons that in large part came from the pre-training prior! 3. Thus, you should be relatively less concerned about lock-in of misaligned goals from early in training, because it precisely the early in training point when the goals are most likely to be close to the pre-training prior and thus most likely to be benign. 4. Instead, you should be relatively more concerned about misaligned goals developing late in training due to incentives for power-seeking. Consider a task like Vending-Bench, where various misaligned/power-seeking strategies are very useful. If models are trained against tasks like that, they could learn to only pursue those sorts of misaligned strategies for the

3Alex Mallen2d

Can you clarify what you mean by "terminal power-seeking"? Some things I can imagine: 1. A cognitive pattern that terminally wants to have long-term power, and therefore plays the training game (IMO the most straightforward interpretation, and the one I most agree with). 2. A cognitive pattern that terminally pursues power-on-the-episode because this is useful for scoring well on the task. This is what you seem to be pointing at with the Vending-Bench example. (Note that this is imperfectly fit on its own) (1) and (2) are importantly different because only (1) motivates training-gaming. I think there's a reasonable path-dependent case to be made that (2) eventually generalizes to (1), but they entail fairly different behaviors so they're important to distinguish.

evhub2d20

Certainly the really concerning thing here is (1). Though indeed one way you might get (1) is by generalization from (2).

[Intro to brain-like-AGI safety] 8. Takeaways from neuro 1/2: On AGI development

Steven Byrnes

(Last revised: March 2026. See changelog at the bottom.)

8.1 Post summary / Table of contents

Part of the “Intro to brain-like-AGI safety” post series.

Thus far in the series, Post #1 set up my big picture motivation: what is “brain-like AGI safety” and why do we care? The subsequent six posts (#2–#7) delved into neuroscience. Of those, Posts #2–#3 presented a way of dividing the brain into a “Learning Subsystem” and a “Steering Subsystem”, differentiated by whether they have a property I call “learning from scratch”. Then Posts #4–#7 presented a big picture of how I think motivation and goals work in the brain, which winds up looking kinda like a weird variant on actor-critic model-based reinforcement learning.

Having established that neuroscience background, now we can finally switch in earnest to thinking more explicitly about...

(Continue Reading - 4622 more words)

3Linda Linsefors3d

I'm surprised by this paragraph. Magrus Carlsen’s Steering Subsystem doesn't have to understand his thoughts about chess moves. And neither will we have to directly interpret the thoughts of an AGI. If humans are playing the role of the Steering Subsystem, we'd start with training the AI's Though Assessor to give us predictions of things we'll understand. This would include predictions of everything we need to stay alive, e.g. Earth future climate, etc. And also thinks we want. Writing a complete list, would be hard, possibly too hard unfortunately. So I'm not saying this is a great plan. I'm just saying if we're the steering subsystem, we're not going to try to interpret activations patterns of the AI, because that's not the steering subsystems job, in your model. Right?

Steven Byrnes2d20

You’re right, thanks, I have now edited that paragraph to also talk about how Thought Assessors might fit in.

1Linda Linsefors3d

I agree that currently brains have continual learning in a way that LLMs don't. However I do think that both brains and LLMs have two different solutions for remembering information for later use. In the brain these are the [long-term memory] vs the [short-term / working memory]. In LLMs these are [updating weights] v.s. [context window]. I don't think it's a coincidence that both systems have something like a context window and something like long-term memory, and I expect any future brain-like AGI to also have these two types of memory, implemented in different ways. I also heard hypothesized that humans have more additional levels of memory. I.e. that things that happened the last days/months, are stored in one way, and things further in the past are stored in another way, and that memories are slowly moved from medium-term storage to long-term storage over time.

Thomas Larsen's Shortform

Thomas Larsen

16Thomas Larsen4d

Hypothesis: alignment-related properties of an ML model will be mostly determined by the part(s) of training that were most responsible for capabilities. If you take a very smart AI model with arbitrary goals/values and train it to output any particular sequence of tokens using SFT, it'll almost certainly work. So can we align an arbitrary model by training them to say "I'm a nice chatbot, I wouldn't cause any existential risk, ... "? Seems like obviously not, because the model will just learn the domain specific / shallow property of outputting those particular tokens in that particular situation. On the other hand, if you train an AI model from the ground up with a hypothetical "perfect reward function" that always gives correct ratings to the behaviour of the AI, (and you trained on a distribution of tasks similar to the one you are deploying it on) then I would guess that this AI, at least until around the human range, will behaviorally basically act according to the reward function. A related intuition pump here for the difference is the effect of training someone to say "I care about X" by punishing them until they say X consistently, vs raising them consistently with a large value set / ideology over time. For example, students are sometimes forced to write "I won't do X" or "I will do Y" 100 times, and usually this doesn't work at all. Similarly, randomly taking a single ethics class during high school usually doesn't cause people to enduringly act according to their stated favorite moral theory. However, raising your child Catholic, taking them to Catholic school, taking them to church, taking them to Sunday school, constantly talking to them about the importance of Catholic morality is in practice fairly likely to make them a pretty robust Catholic. There are maybe two factors being conflated above: (1) the fraction of training / upraising focused on goal X, and (2) the extent to which goal X was getting the capabilities. The reason why I think (2) is

Daniel Kokotajlo3d32

Relevant OpenAI blog post just today: https://alignment.openai.com/how-far-does-alignment-midtraining-generalize/

Relevant figure:

2Alex Mallen3d

I think this is a very important hypothesis but I disagree with various parts of the analysis. [...] I think this is an important observation, and is the main thing I would have cited for why the hypothesis might be true. But I think it's plausible that the AI's capabilities here could be separated from its propensities by instrumentalizing the learned heuristics to aligned motivations. I can imagine that doing inoculation prompting and a bit of careful alignment training at the beginning and end of capabilities training could make it so that all of the learned heuristics are subservient to corrigible motivations - i.e., so that when the heuristics recommend something that would be harmful or lead to human disempowerment, the AI would recognize this and choose otherwise. [...] Even if the AI had a perfect behavioral reward function during capabilities-focused training, it wouldn't provide much pressure towards motivations that don't take over. During training to be good at e.g. coding problems, even if there's no reward-hacking going on, the AI might still develop coding related drives that don't care about humanity's continued control, since humanity's continued control is not at stake during that training (this is especially relevant when the AI is saliently aware that it's in a training environment isolated from the world -- i.e. inner misalignment). Then when it's doing coding work in the world that actually does have consequences for human control, it might not care. (Also note that generalizing "according to the reward function" is importantly underspecified.) [...] This type of training (currently) does actually generalize to other propensities to some extent in some circumstances. See emergent misalignment. I think this is plausibly also a large fraction of how character training works today (see "coupling" here).

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Doing quick sanity checks

8.1 Post summary / Table of contents