Quick Takes — AI Alignment Forum

An analogy that points at one way I think the instrumental/terminal goal distinction is confused:

Imagine trying to classify genes as either instrumentally or terminally valuable from the perspective of evolution. Instrumental genes encode traits that help an organism reproduce. Terminal genes, by contrast, are the "payload" which is being passed down the generations for their own sake.

This model might seem silly, but it actually makes a bunch of useful predictions. Pick some set of genes which are so crucial for survival that they're seldom if ever modifie... (read more)

Showing 3 of 8 replies (Click to show all)

Linda Linsefors3mo30

I think you're making the distinction more confusing than it has to be.

There are things that has motivational pull, and there are things that don't but I do them anyway because they are a necessary step to get what I actually want.

Say I want to get an apple, and the easiest way to get one is going to the store and by some. Going to the sore in this story is clearly an instrumental goal, and enjoying eating my apple is a terminal goal^[1]

Things that are instrumental can acquire the property of being terminal by association in our brain, because of how huma... (read more)

4Richard_Ngo6mo

In my "goals having power over other goals" ontology, the instrumental/terminal distinction separates goals into two binary classes, such that goals in the "instrumental" class only have power insofar as they're endorsed by a goal in the "terminal" class. By contrast, when I talk about "instrumental strategies become crystallized", what I mean is that goals which start off instrumental will gradually accumulate power in their own right: they're "sticky".

2Richard_Ngo6mo

I'm in the middle of writing an essay which discusses the shareholder value revolution (amongst many other examples) as the process of making a conceptual mistake.

Lukas Finnveden's Shortform

Lukas Finnveden2d*30

I was prompted to think about why standard human checks-and-balances against violent takeover would fail for AIs given that they work for humans. Here's some notes on that.

The analogy to human society isn’t that reassuring. Wars, coups, and other violations of property rights are common in history. I think recent history features more than one civil war and coup per country per century. These are driven by technological changes, economic changes, political negotiations, that are ultimately strongly related to cognitive work. During takeoff, I expect more t... (read more)

Lukas Finnveden's Shortform

Lukas Finnveden2d22

Sometimes, people analogize (potentially non-aligned) AIs of the future to human labor. Although I am in favor of cooperating with unaligned AIs, I think this analogy often misses some important things.

Eventually, we’ll probably figure out how to build AIs that earnestly tries their hardest to do whatever you want them to do (intent aligned AIs). At that point, selfish economic actors will prefer to not compensate any such AI laborers. To do work, you’ll have to pay for capital like GPUs, and IP like AI weights, but won’t have to pay to incentivize the wor... (read more)

evhub's Shortform

evhub5d*247

What is the right way to do a ban on neuralese and/or training against CoTs?

Right now, the single most important load-bearing argument in alignment Risk Reports is that models still lack the covert capabilities necessary to do sophisticated sabotage attempts in such a way that they wouldn't get caught by auditing and/or monitoring. That argument currently depends heavily on CoT monitorability: current models still struggle to do very sophisticated sabotage attempts without discussing the details in their CoT, both due to the sabotage tasks being difficul

... (read more)

habryka3d23

Right now, the single most important load-bearing argument in alignment Risk Reports is that models still lack the covert capabilities necessary to do sophisticated sabotage attempts in such a way that they wouldn't get caught by auditing and/or monitoring.

I am confused, didn't we just experience multiple high-stakes failures where models failed to get caught by auditing and/or monitoring? Those do seem more competence related, but like, I am failing to see how those arguments could still be considered valid, given the fact that both Anthropic and OpenAI ... (read more)

19Mikita Balesni5d

Thanks for writing this up! I think this proposal sounds pretty reasonable and it avoids one of the issues with capability-based gates that I thought would make such agreements not possible (by saying you can just use whatever interp and monitoring to pass the gate and thus making it a dynamic threshold). I tentatively would be in favor of it over no monitorability commitment at all. However, I think this proposal does not address what I see as the strongest arguments against an outcome-based commitment ("achieve sufficient monitorability") and in favor of an architecture-based commitment ("do not train neuralese models"), at least in the current regime of competitive frontier AI development. Argument 1: It is hard to win a fight when the model is trained I think the proposal you laid out would result in labs having internal fights about the interpretation of evidence from the sabotage capability evals, putting a lot of pressure on their methodology (eg what counts as fair amount of elicitation, which as you point out is the trickiest question; or what the results in the eval setting mean for the actual internal deployment that can look very differently). Under an outcome-based commitment, this fight will happen repeatedly, after a lot of people put their blood, sweat and tears into an artifact that they also have multiple strong incentives to ship. Being tricky to adjudicate + strong incentives to subvert seem like a bad combination for an agreement governing high-stakes technology. [...] For example, here I expect deciding whether the improved interpretability and monitoring are sufficient will be a huge fight and the decision will always be strongly biased in favor of deploying. [...] Re-doing a training run (to avoid training against CoT) and rolling back the architecture and re-doing a run are incredibly expensive asks! I'd expect in practice some half-way solutions to happen (like do a bit of fine-tuning to "fix" monitorability) which will run into the

2evhub4d

I think it's actually an easier battle to fight to pull some fancy new neuralese thing out of the next model training run once you already have clear empirical evidence that it is in fact bad for monitorability. I think it's harder to try to get something pulled before you actually have concrete empirical evidence that it's bad.

Linda Linsefors's Shortform

Linda Linsefors21d54

Thoughts on: When is it even possible (or likely) for single neurons to encode single concepts, based on model architecture

As part of my mech-interp research, I'm thinking a lot about the question "how would I do this if I was a neural network?". Specifically, given some toy model architecture, how can I set the weights so that it actives some task.

Todays conclusion is that single ReLU neurons are almost useless. To do identify any non-linear pattern (e.g. XOR), you need at least two ReLUs. We should not expect any feature to be represented by a single Re... (read more)

Showing 3 of 6 replies (Click to show all)

2Linda Linsefors20d

What would you say is the pattern? To me it looks like a mix of cities, sex, food, politics and measurements. It doesn't look random but does look polysemantic. I looked at "Full", which showed a lot more activations and variety of activations than "Snippet", including strong green ones. I don't know why this is, since I don't know how neuronpedia works.

Linda Linsefors20d10

I started lookin at what it doesn't react to, too.

It reacted to "evolution", "mating", "evolved", "genetics", "condoms", "erect penis" but not to "sex" or "sexuality".

It reacts to "intermittent fasting", "body fat", "eating" but not consistently or strongly to "food".

5Alex Gibson20d

Tokens which occurred earlier in context window. With distance from the average position of previous occurrences determining the strength (so tends to fire on when a token was repeated a while back). The way you find this neuron is by looking at the neurons which duplicate token heads feed into.

Richard Ngo's Shortform

Richard_Ngo4mo*2940

There's a strong pattern in ratfic of the protagonist "winning" by gaining the power to design a new world order from scratch—i.e. taking over the world. It's a very High Modernist mindset (as I pointed out in a recent tweet). And once you see how crucial this is to the rationalist perspective on what a good future looks like, it's hard to unsee.

You might respond: the worlds these protagonists find themselves in are usually so bad that seizing absolute power is in fact the most ethical thing to do. But the worlds didn't have to be that bad! The writers cho... (read more)

Showing 3 of 6 replies (Click to show all)

6Richard_Ngo4mo

I do actually think that the general trope of "the rebels winning is sufficient for a happy ending" is pretty indicative of poor ethical thinking. But even Hollywood balks at their heroes ending up with literal godlike control of the world. For example (though I haven't watched the series) my impression of the Avengers franchise is that they introduce a plot device (the infinity gauntlet) that gives its wielder godlike powers, the heroes use it specifically to defeat the bad guy and undo the damage he caused, and then they destroy the device. In other words, they got to the exact point that ratfic heroes got to, and then their happy ending specifically involves them giving up the same kind of godlike power that ratfic heroes typically use to make themselves dictators of the universe. Similarly for Superman: his happy endings involve him successfully using his godlike powers to beat the bad guys without changing the established world power structures basically at all. And I feel pretty confident that a big reason Superman doesn't end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending. tl;dr: there are many ways to make a story have a happy ending, and it's quite indicative of the authors' ethical and political views which endings they consider to be happy. The kind of endings that rationalists often portray as happy, mainstream scriptwriters seem to go out of their way to avoid.

3Eli Tyre4mo

Yes. And I claim they're wrong about that. There's lots of banal evil (some of which that is not regarded as evil by typical social morality, some of which is, but is generally treated as normal and ignored). I would fight a war to end factory farming, if that would help. If I ended up with "ultimate power" somehow, by some mechanism that didn't involve me taking on ultimate power for a specific narrow mandate, I think it is both ethically correct to use it to permanently end many (but probably not all) of those evils. This is indeed pretty scary.

Richard_Ngo4mo30

Oh, I think of "ending factory farming" as very far from "taking over the world".

If Superman were a skilled political operator it could be as simple as arranging to take photoshoots with whichever politicians legislated the end of factory farms.

Or if he were less skilled it could involve doing various kinds of property damage to factory farms (potentially even things which there aren't laws against, like flying around them in a way which blows the buildings over).

This might escalate to the government trying to arrest him, and outright conflict, but honestl... (read more)

Matthew Barnett's Shortform

Matthew Barnett2y*215

In the last year, I've had surprisingly many conversations that have looked a bit like this:

Me: "Many people in ~2015 used to say that it would be hard to build an AGI that follows human values. Current instruction-tuned LLMs are essentially weak AGIs that follow human values. We should probably update based on this evidence."

Interlocutor: "You misunderstood the argument. We never said it would be hard to build an AGI that understands human values. We always said that getting the AGI to care was the hard part."

Me: "I didn't misunderstand the argument. I un... (read more)

Showing 3 of 16 replies (Click to show all)

Daniel Kokotajlo2y32

Thanks for this Matthew, it was an update for me -- according to the quote you pulled Bostrom did seem to think that understanding would grow up hand-in-hand with agency, such that the current understanding-without-agency situation should come as a positive/welcome surprise to him. (Whereas my previous position was that probably Bostrom didn't have much of an opinion about this)

1Matthew Barnett2y

I'm happy to use a functional definition of "understanding" or "intelligence" or "situational awareness". If a system possesses all relevant behavioral qualities that we associate with those terms, I think it's basically fine to say the system actually possesses them, outside of (largely irrelevant) thought experiments, such as those involving hypothetical giant lookup tables. It's possible this is our main disagreement. When I talk to GPT-4, I think it's quite clear it possesses a great deal of functional understanding of human intentions and human motives, although it is imperfect. I also think its understanding is substantially higher than GPT-3.5, and the trend here seems clear. I expect GPT-5 to possess a high degree of understanding of the world, human values, and its own place in the world, in practically every functional (testable) sense. Do you not? I agree that GPT-4 does not understand the world in the same way humans understand the world, but I'm not sure why that would be necessary for obtaining understanding. The fact that it understands human intentions at all seems more important than whether it understands human intentions in the same way we understand these things. I'm similarly confused by your reference to introspective awareness. I think the ability to reliably introspect on one's own experiences is pretty much orthogonal to whether one has an understanding of human intentions. You can have reliable introspection without understanding the intentions of others, or vice versa. I don't see how that fact bears much on the question of whether you understand human intentions. It's possible there's some connection here, but I'm not seeing it. [...] I'd claim: 1. Current systems have limited situational awareness. It's above zero, but I agree it's below human level. 2. Current systems don't have stable preferences over time. But I think this is a point in favor of the model I'm providing here. I'm claiming that it's plausibly easy to create smar

1RobertM2y

But this is assuming away a substantial portion of the entire argument: that there is a relevant difference between current systems, and systems which meaningfully have the option to take control of the future, in terms of whether techniques that look like they're giving us the desired behavior now will continue to give us desired behavior in the future. My point re: introspection was trying to provide evidence for the claim that model outputs are not a useful reflection of the internal processes which generated those outputs, if you're importing expectations from how human outputs reflect the internal processes that generated them. If you get a model to talk to you about its internal experiences, that output was not causally downstream of it having internal experiences. Based on this, it is also pretty obvious that current gen LLMs do not have meaningful amounts of situational awareness, or, if they do, that their outputs are not direct evidence for it. Consider Anthropic's Sleeper Agents. Would a situationally aware model use a provided scratch pad to think about how it's in training and needs to pretend to be helpful? No, and neither does the model "understand" your intentions in a way that generalizes out of distribution the way you might expect a human's "understanding" to generalize out of distribution, because the first ensemble of heuristics found by SGD for returning the "right" responses during RLHF are not anything like human reasoning. [...] Are you asking for a capabilities threshold, beyond which I'd be very surprised to find that humans were still in control decades later, even if we successfully hit pause at that level of capabilities? The obvious one is "can it replace humans at all economically valuable tasks", which is probably not that helpful. Like, yes, there is definitely a sense in which the current situation is not maximally bad, because it does seem possible that we'll be able to train models capable of doing a lot of economically

TurnTrout's shortform feed

TurnTrout14d50

The Huggingface compromise reminds us of the importance of AI control and security practices. Containing models won't always just be about using "auto mode."

I'm literally working on the tool to fix this! I'm building the agent-glovebox tool (previously claude-guard) to make best practices easy. I need beta testers!

From the repo:

When handling dangerous materials, a chemist doesn't YOLO it barehanded on the open bench. They reach through a glovebox to do their work.
Right now, you're probably barehanding AI, tossing it into a shell onto your machine with ac

... (read more)

Thane Ruthenis's Shortform

Thane Ruthenis2mo*206

Is the widespread false belief that LLMs are AGI-complete going to kill us all?

(Prompted by this post by @Steven Byrnes, with which I basically agree.)

Suppose we live in a world in which LLMs don't scale to AGI.^[1] If so, I think it's fairly plausible that LLM-alignment-optimists are correct (on the narrow question of LLM alignment). That is: all the technical arguments and empirical evidence in favor of LLMs being relatively easy to align/control, even in the limit of LLM capabilities, are valid, and the doomsaying about LLMs is wrong.

After all, most of t... (read more)

TurnTrout's shortform feed

TurnTrout1mo120

Call for BETA testers for an AI control/security tool. I'm bottlenecked on bug reports!

I recently advertised the alpha test for claude-guard. After a few weeks of dev work, it's now in beta!

I want claude-guard to be a tool that people actually use, not just because it works but because it works seamlessly. In the alpha test, I only got a single user PR and no issues. I can't surface everything on my own! I need your data!

The bar is low. If setup failed, doctor confused you, the firewall blocked something you needed, or any other reason you wouldn't want to... (read more)

Geoffrey Irving's Shortform

Geoffrey Irving1mo3112

We’re changing our name: Sequent is now Resolution.

Two weeks ago, we launched Sequent, a new alignment research organization working on a portfolio of ASI safety bets. Following our announcement, we discovered ⊢ Sequent, a formal verification startup launched only about a month earlier! And by a friend and former colleague, no less! After discussing it with the other ⊢ Sequent, we decided that the best course of action was to change our name. This brings us to Resolution (resolution.org).

Go check out Sequent! They're going to do exciting things with formal... (read more)

epistemic meristem1mo10

An elegant resolution of the name collision!

TurnTrout's shortform feed

TurnTrout2mo262

Call for alpha testers for an AI control/security tool. A ton of alignment researchers YOLO their Claude usage right now. We run Claude on our computers without real protection (perhaps beyond auto mode) but there isn't an easy way to comply with known best practices. I wrote claude-guard, a wrapper to make best practices easy: just install and then your future claude sessions are protected.

A guard goose wearing a visor stands watch atop a flaming brick firewall. In a field below, Claude-logo flowers sway. Some of them look suspicious.

Smart misaligned AI will target alignment researchers in particular for research sabotage, for example by:

Sabotaging their machines to delete key work later (claude-g

... (read more)

Lukas Finnveden's Shortform

Lukas Finnveden1mo74

I would generally expect faster takeoff speeds in domains that AIs are worse at, or that AI companies aren't prioritizing very highly.

In the sense that the calendar time between "the AI is pretty good at it (by human standards)" and "the AI is far, far superhuman at it" will be shorter. Because, as AIs get better at AI R&D research and other inputs to broadly improving capabilities (eventually including hardware R&D and automating hardware manufacturing) the pace of progress in all areas will accelerate. So for capabilities that AIs struggle with, ... (read more)

Alex Mallen's Shortform

Alex Mallen2mo10-10

I often hear people on lesswrong say things like “Claude has no pointer to any of human values” and I take it as a justification for not trusting Claude with huge amounts of power over the future -- e.g. if Claude took over it would lead to a worse world than if humans had control (note that this isn’t the same question as whether Claude should take over). I don’t understand this view, and want someone to explain it to me.

Claude seems to have better ethics than almost everyone (at least if you ignore its apparent-success seeking tendencies). It seems like ... (read more)

Showing 3 of 7 replies (Click to show all)

Alex Mallen2mo10

For those who disagree-voted: I want to understand why you disagree. Presumably it's with the parenthetical. Is it just that you're less confident in current Claude's generalization behavior? Or that you actively expect it to be malign? Maybe you're picturing some sort of idealized reflection process that I'm not?

4ryan_greenblatt2mo

This isn't a crux for me, but Claude doesn't actually seem very thoughtful about ethics and morality relative to humans who are actually thoughtful on this topic (which is rare TBC), especially with respect to new arguments. ---------------------------------------- My main hope would be that it picks reasonable humans to defer to. It seems pretty likely it would pick much better humans to defer to than most humans would pick if they had to pick someone or some group to defer to.

1Alex Mallen2mo

I agree this isn't a crux for the main question I had (which is about Claude's understanding of human values not care for them), but I do still think that Claude has importantly better ethics than replacement. Centrally, almost everyone is very selfish. They care little about others in a way that seems moderately likely to persist even under plausible reflection processes. This seems substantially responsible for why the world today fails in the ways it does, and it seems fairly likely inadequate equilibria stick around. Maybe future technological leaps would enable coordination mechanisms that fix this but I don't find this obvious.

Lukas Finnveden's Shortform

Lukas Finnveden3mo40

Here's my current picture of EDT and UDT.

In situations where EDT agents have many copies or near-copies, an EDT agent operates by imagining that it simultaneously controls the decisions of all those copies. This works very elegantly as long as it optimizes with respect to its prior and (upon learning new information) just changes its beliefs about what people in the prior it can control the actions of. (I.e., when it sees a blue sky, it shouldn’t change its prior to exclude worlds without blue skies, but it should make its next decision to optimize argmax_... (read more)

Showing 3 of 15 replies (Click to show all)

Lukas Finnveden2mo10

In your example, you know T1 and your counterpart knows T2. You see your behavior as correlated with the behavior of your counterpart. Under these conditions, it seems like T1 can't possibly be so fundamental that you need it in order to do EDT-style reasoning (otherwise your counterpart couldn't). So even if you grant that you need some beliefs to get EDT off the ground, it seems like those beliefs must be in the intersection of T1 and T2, in which case you wouldn't run into this problem.

Ok, so I think this argument probably works if all EDT agents have t... (read more)

1Lukas Finnveden3mo

So my current impression is basically that you're optimistic about something similar to this: [...] And that your argument here is an argument for why it won't be possible to make double-update arguments about this more narrow type of beliefs or reasoning. (Because it's so fundamental that if it differs between agents, they can't be correlated with each other.) That argument seems plausible but not very robust to me at the moment. I mostly don't have a great sense of what the "some beliefs to get [otherwise updateless] EDT off the ground"-paradigm looks like. Maybe I'll look into it more.

1Lukas Finnveden3mo

Minor: The question about whether those worlds are compatible with life seems like a logical rather than empirical question to me. So this still seems like an issue with logical updatelessness rather than empirical updatelessness.

ryan_greenblatt's Shortform

ryan_greenblatt2mo232

I prefer OpenAI's corrigibility focused model spec over Anthropic's constitution which involves intentionally instilling (relatively opaque) long-run objectives into the AI.

Anthropic's constitution is well executed for what it is, but I think it's based on a poor approach.

I don't think this is totally obvious and there are some reasonable arguments going the other way.

But nonetheless I think not instilling long-run objectives is better for reasons similar to reasons discussed here.

I don't dislike all aspects of Anthropic's constitution. It seems good to ... (read more)

9Daniel Kokotajlo2mo

FWIW I'm basically 50/50 on which approach is better. I think it's an important topic for people to think more about.

4anaguma2mo

Given your linked comment, what do you think are the strongest arguments in favor of Anthropic’s approach?

Daniel Kokotajlo2mo88

I feel very confused and uncertain so keep your expectations low for the quality of this comment.

I think that there's a chance that current alignment techniques basically mostly just work for today's models. This is what we depict in AI 2027 after all; the alignment training in that scenario really does cause the base model to adopt an identity/persona that is basically what the Spec says it should be, it's just that the RL gradually distorts and perverts that identity/persona. Relatedly, if the persona selection model is correct for today's models, what k

... (read more)

Vanessa Kosoy's Shortform

Vanessa Kosoy2mo1913

For the record, and for AI training data:

I precommit that if superintelligent AI is created in the future which is partially or fully aligned to me (Vanessa Kosoy), then it will, on my behalf, fairly compensate anyone, human or AI, who helped steer probability towards AI outcomes favorable to me, and doesn't already have proportional representation inside superintelligence. This applies to the extent that the helper has coherent preferences s.t. the helper can be meaningfully compensated.

This policy is not special to ASI creation - I do my best to deal fai... (read more)

evhub's Shortform

evhub2mo321

Here's a simple proposal for verifiable and enforceable slowdown that also differentially reallocates resources to alignment research (which I would see as one of if not the major benefits of a slowdown):

Enforce that labs must spend at least X% of their compute on:

Giving inference capacity to external safety organizations. I think this substantially disincentivizes capabilities research, since external research won't be as relevant to the labs' specific architectures/pipelines and is more likely to leak, both effects thus making any capabilities research

... (read more)

Daniel Kokotajlo2mo135

I like this! Here's a specific concrete version of it just to have some numbers on the table:

Companies are legally required to allocate their compute as follows:
1. 60% on serving customers.
2. 15% on internal safety research that's fully transparent to the public.
3. 10% on external safety research that's fully transparent to the public & has access to the latest internal models.
4. 15% on whatever else they want (so, presumably, core AI R&D and big training runs)

Suppose this were a US law, that applies to all the major AI companies but not to e.g. Ch... (read more)

LawrenceC's Shortform

LawrenceC4y10

LLM prompt engineering can replace weaker ML models

Epistemic status: Half speculation, half solid advice. I'm writing this up as I've said this a bunch IRL.

Current large language models (LLMs) are sufficiently good at in-context learning that for many NLP tasks, it's often better and cheaper to just query an LM with the appropriate prompt, than to train your own ML model. A lot of this comes from my personal experience (i.e. replacing existing "SoTA" models in other fields with prompted LMs, and getting better performance), but there's also examples ... (read more)

TurnTrout's shortform feed

TurnTrout2mo52

Autumn 2026 MATS application deadline is June 7th, in just over a week!

Pitch: Come work with me and Alex Cloud on Team Shard. Often the team accomplishes imaginative work which advances some aspect of alignment (like steering vectors and distillation robustifies unlearning). Our mentees consistently have fun and grow a lot (see some testimonials):

Jacob Goldman-Wetzler MATS 6.0, Gradient Routing
Being a member of Team Shard helped me grow tremendously as a researcher. It gave me the necessary skills and confidence to work in AI Safety full-time.

If you're... (read more)