I often hear people on lesswrong say things like “Claude has no pointer to any of human values” and I take it as a justification for not trusting Claude with huge amounts of power over the future -- e.g. if Claude took over it would lead to a worse world than if humans had control (note that this isn’t the same question as whether Claude should take over). I don’t understand this view, and want someone to explain it to me.

Claude seems to have better ethics than almost everyone (at least if you ignore its apparent-success seeking tendencies). It seems like ... (read more)

Here's my current picture of EDT and UDT.

In situations where EDT agents have many copies or near-copies, an EDT agent operates by imagining that it simultaneously controls the decisions of all those copies. This works very elegantly as long as it optimizes with respect to its prior and (upon learning new information) just changes its beliefs about what people in the prior it can control the actions of. (I.e., when it sees a blue sky, it shouldn’t change its prior to exclude worlds without blue skies, but it should make its next decision to optimize argmax_... (read more)

I prefer OpenAI's corrigibility focused model spec over Anthropic's constitution which involves intentionally instilling (relatively opaque) long-run objectives into the AI.

Anthropic's constitution is well executed for what it is, but I think it's based on a poor approach.

I don't think this is totally obvious and there are some reasonable arguments going the other way.

But nonetheless I think not instilling long-run objectives is better for reasons similar to reasons discussed here.

I don't dislike all aspects of Anthropic's constitution. It seems good to ... (read more)

Here's a simple proposal for verifiable and enforceable slowdown that also differentially reallocates resources to alignment research (which I would see as one of if not the major benefits of a slowdown):

Enforce that labs must spend at least X% of their compute on:

1. Giving inference capacity to external safety organizations. I think this substantially disincentivizes capabilities research, since external research won't be as relevant to the labs' specific architectures/pipelines and is more likely to leak, both effects thus making any capabilities research

Reward-seekers will probably behave according to causal decision theory.

Background: There are existing arguments to the effect that default RL algorithms encourage CDT reward-maximizing behavior on the training distribution. (That is: Most RL algorithms search for policies by selecting for actions that cause the highest reward. E.g., in the twin prisoner’s dilemma, RL algorithms randomize actions conditional on the policy so that the action provides no evidence to the RL algorithm about the counterparty’s action.) This doesn’t imply RL produces CDT reward-... (read more)

Here are some of my top candidates for big pushes to do right now on technical AI safety (low effort notes):

• Much better model organisms / misalignment analogies:
  • Doing a wider set of pessimized training runs
    • Good candidate for lots of AI labor automation? Like maybe good to try to set up pipelines for building these envs.
  • Demonstrating risks from fitness-seekers/reward-seekers empirically
    • Even on current models with better tests, see here
  • Demonstrating various types of memetic spread of misalignment?
• Actually do control
  • Build pipelines for red-

We don't know how AIs are aligned.

A somewhat crazy aspect of the current situation is that we have very little confirmed public information about why frontier AIs end up being apparently behaviorally aligned. And more generally, we don't know what factors in training are most relevant for (behavioral) alignment. Like, what interventions in training result in Anthropic AIs following the constitution or make OpenAI AIs follow the spec? What factors tend to make them follow the constitution/spec less (or cause various specific misaligned behaviors)? It's not... (read more)

An analogy that points at one way I think the instrumental/terminal goal distinction is confused:

Imagine trying to classify genes as either instrumentally or terminally valuable from the perspective of evolution. Instrumental genes encode traits that help an organism reproduce. Terminal genes, by contrast, are the "payload" which is being passed down the generations for their own sake.

This model might seem silly, but it actually makes a bunch of useful predictions. Pick some set of genes which are so crucial for survival that they're seldom if ever modifie... (read more)

Automating alignment may be harder than automating capabilities, because of ‘unsafe to verify’ tasks

This was a note I wrote for my colleagues on UK AISI's Alignment Team. It contains very little that’s novel, and mostly just distills things that I’ve read elsewhere [1, 2, 3]. Still, I wanted to post it so that I can point people to it as I've not seen all of these points in the same place.

A key question for AI safety is not just "can we automate alignment research?" but whether we can automate alignment research as fast as capabilities research. ^[1][2], I ... (read more)

I think we should be relatively less worried about instrumental power-seeking and relatively more worried about terminal power-seeking. Note that this is only a relative update on the margin, and maybe on net I am still more concerned about the instrumental version because I started much more concerned about it. This is also not a super recent update—I just haven't seen it written up before.

Simple argument:

1. The standard deceptive alignment story involves a model developing a somewhat random proxy goal and then that goal getting effectively locked-in and r

Master post for alignment protocols.

Other relevant shortforms:

• Autocalibrated quantilized debate
• Hippocratic principle
• IDA variants
• Dialogic RL
• More dialogic RL

Buying AI labor might be a big deal for philanthropists.

I think the total available for AI safety philanthropy is almost $100B (at current valuations), mostly from Anthropic.^[1] The AI safety nonprofit ecosystem currently consumes about $1B per year. There are still good opportunities available, but they're several times worse than the average spending (because the low-hanging fruit has been plucked^[2]). Marginal effectiveness would likely decline by ~half again if you doubled the rate of AI safety philanthropy.

So there's likely ~30x more money than can be... (read more)

Diminishing returns in the NanoGPT speedrun:

To determine whether we're heading for a software intelligence explosion, one key variable is how much harder algorithmic improvement gets over time. Luckily someone made the NanoGPT speedrun, a repo where people try to minimize the amount of time on 8x H100s required to train GPT-2 124M down to 3.28 loss. The record has improved from 45 minutes in mid-2024 down to 1.92 minutes today, a 23.5x speedup. This does not give the whole picture-- the bulk of my uncertainty is in other variables-- but given this is exist... (read more)