AI ALIGNMENT FORUM
AF

Alex Turner
Ω471776116711
Message
Dialogue
Subscribe

I don't use LessWrong much anymore. Find me at www.turntrout.com.

My name is Alex Turner. I'm a research scientist at Google DeepMind on the Scalable Alignment team. My views are strictly my own; I do not represent Google. Reach me at alex[at]turntrout.com

Sequences

Posts

Sorted by New

Wikitag Contributions

Comments

Sorted by
Newest
Interpreting a Maze-Solving Network
Thoughts on Corrigibility
The Causes of Power-seeking and Instrumental Convergence
Reframing Impact
Evaluating the historical value misspecification argument
Alex Turner20d12

based prediction

Reply1
Distillation Robustifies Unlearning
Alex Turner23d47

Wasn't it the case that for some reason, full distillation had comparable compute requirement to data filtering? I was surprised by that. My impression is that distillation should be more like 10% of pretraining (data filtering), which would make the computational UNDO results much stronger. Not sure what happened here.

Reply
Distillation Robustifies Unlearning
Alex Turner23d30

I think you missed the point here. My suggested scheme is 1. label a small amount of data 2. train a classifier 3. apply the classifier to know if you should skip a token / make the target logprobs be noise or use the original logprobs. This is spiritually the same as 1. label a small amount of data 2. use that for unlearning 3. apply the unlearned model to know if the target logprobs should be noise or sth close to the original logprobs.

EDIT: I think I misunderstood your original point - were you saying to just label all of the data using a classifier trained on just 1% of the pretraining data? (Neither of your schemes say what to do after step 3.)

> UNDO over Unlearn-and-Distill is that it provides a tunable compute/robustness knob between the conventional unlearning and full reinitialization/data filtering 

This to be a part of the option space that nobody is interested in, but it's still scientifically interesting. 

Why do you claim that no one is interested in this? Lots of labs do data filtering, which is known to be effective but quite costly to iterate on. 

Reply
Distillation Robustifies Unlearning
Alex Turner23d20

In other words, "using unlearning techniques like GradDiff/MaxEnt during pretraining" might be a really powerful technique.

I have a cached thought that this was found to disrupt overall capabilities / make learning harder, but I don't have a reference on hand.

Reply
ryan_greenblatt's Shortform
Alex Turner1mo*157

I think that "make it easy to responsibly share a dataset" would be a highly impactful project. Anthropic's Claude 4 model card already argues that dataset leakage hurt Claude 4's alignment (before mitigations). 

For my part, I'll put out a $500 bounty on someone completing this project and doing a good job of it (as judged by me / whomever I consult). I'd also tweet it out and talk about how great it is that [person] completed the project :) I don't check LW actively, so if you pursue this, please email alex@turntrout.com.

EDIT: Thanks to my coworker Anna Wang , the bounty is doubled to $1,000! Completion criterion is:

An unfamiliar researcher can follow the instructions and have their dataset responsibly uploaded within one hour

Please check proposed solutions with dummy datasets and scrapers

Reply1
ryan_greenblatt's Shortform
Alex Turner1mo178

Thanks for taking these steps! 

Context: I was pretty worried about self-fulfilling misalignment data poisoning (https://turntrout.com/self-fulfilling-misalignment) after reading some of the Claude 4 model card. I talked with @Monte M and then Ryan about possible steps here & encouraged action on the steps besides the canary string. I've considered writing up a "here are some steps to take" guide but honestly I'm not an expert.

Probably there's existing work on how to host data so that AI won't train on it. 

If not: I think it'd be great for someone to make a template website for e.g. signing up with CloudFlare. Maybe a repo that has the skeleton of a dataset-hosting website (with robots.txt & ToS & canary string included) for people who want to host misalignment data more responsibly. Ideally those people would just have to

  1. Sign up with e.g. Cloudflare using a linked guide,
  2. Clone the repo,
  3. Fill in some information and host their dataset.

After all, someone who has finally finished their project and then discovers that they're supposed to traverse some arduous process is likely to just avoid it. 

Reply1
Distillation Robustifies Unlearning
Alex Turner1mo20

--Filter out the ones that seem to have maybe been unfaithful, as judged by e.g. activations for deception or whatnot.

Would you actively unlearn on those CoTs? Or just filter from distillation data?

Reply
Self-fulfilling misalignment data might be poisoning our AI models
Alex Turner3mo32

Any empirical evidence that the Waluigi effect is real? Or are you more appealing to jailbreaks and such?

Reply
Self-fulfilling misalignment data might be poisoning our AI models
Alex Turner3mo40

I think we have quite similar evidence already. I'm more interested in moving from "document finetuning" to "randomly sprinkling doom text into pretraining data mixtures" --- seeing whether the effects remain strong.

Reply
Self-fulfilling misalignment data might be poisoning our AI models
Alex Turner3mo66

I agree. To put it another way, even if all training data was scrubbed of all flavors of deception, how could ignorance of it be durable?

This (and @Raemon 's comment[1]) misunderstand the article. It doesn't matter (for my point) that the AI eventually becomes aware of the existence of deception. The point is that training the AI on data saying "AI deceives" might make the AI actually deceive (by activating those circuits more strongly, for example). It's possible that "in context learning" might bias the AI to follow negative stereotypes about AI, but I doubt that effect is as strong. 

From the article:

We are not quite “hiding” information from the model

Some worry that a “sufficiently smart” model would “figure out” that e.g. we filtered out data about e.g. Nick Bostrom’s Superintelligence. Sure. Will the model then bias its behavior towards Bostrom’s assumptions about AI?

I don’t know. I suspect not. If we train an AI more on math than on code, are we “hiding” the true extent of code from the AI in order to “trick” it into being more mathematically minded?

Let’s turn to reality for recourse. We can test the effect of including e.g. a summary of Superintelligence somewhere in a large number of tokens, and measuring how that impacts the AI’s self-image benchmark results.

  1. ^

    "even if you completely avoided [that initial bias towards evil], I would still basically expect [later AI] to rediscover [that bias] on it's own"

Reply
Load More
10TurnTrout's shortform feed
6y
256
31A Simple Explanation of AGI Risk
15d
0
86Distillation Robustifies Unlearning
1mo
21
50Self-fulfilling misalignment data might be poisoning our AI models
4mo
13
52Steering Gemini with BiDPO
5mo
2
31Gaming TruthfulQA: Simple Heuristics Exposed Dataset Weaknesses
6mo
0
62Gradient Routing: Masking Gradients to Localize Computation in Neural Networks
7mo
3
51Deep Causal Transcoding: A Framework for Mechanistically Eliciting Latent Behaviors in Language Models
7mo
1
20Intrinsic Power-Seeking: AI Might Seek Power for Power’s Sake
8mo
1
97Mechanistically Eliciting Latent Behaviors in Language Models
1y
20
45Many arguments for AI x-risk are wrong
1y
47
Load More
Reinforcement learning
2y
(+16)
Reinforcement learning
2y
(+333/-390)
Complexity of value
3y
(+176/-112)
General Alignment Properties
3y
(+317)
Pages Imported from the Old Wiki
5y
(+9/-5)