This is a special post for short-form writing by Alex Turner. Only they can create top-level comments. Comments here also appear on the Shortform Page and All Posts page.
Against CIRL as a special case of against quickly jumping into highly specific speculation while ignoring empirical embodiments-of-the-desired-properties.
Just because we write down English describing what we want the AI to do ("be helpful"), propose a formalism (CIRL), and show good toy results (POMDPs where the agent waits to act until updating on more observations), that doesn't mean that the formalism will lead to anything remotely relevant to the original English words we used to describe it. (It's easier to say "this logic enables nonmonotonic reasoning" and mess around with different logics and show how a logic solves toy examples, than it is to pin down probability theory with Cox's theorem)
And yes, this criticism applies extremely strongly to my own past work with attainable utility preservation and impact measures. (Unfortunately, I learned my lesson after, and not before, making certain mistakes.)
In the context of "how do we build AIs which help people?", asking "does CIRL solve corrigibility?" is hilariously unjustified. By what evidence have we located such a specific question? We have assumed there is an achievable "corrigibility"-like property; we ha... (read more)
Actually, this is somewhat too uncharitable to my past self. It's true that I
did not, in 2018, grasp the two related lessons conveyed by the above comment:
1. Make sure that the formalism (CIRL, AUP) is tightly bound to the problem at
hand (value alignment, "low impact"), and not just supported by "it sounds
nice or has some good properties."
2. Don't randomly jump to highly specific ideas and questions without lots of
locating evidence.
However, in World State is the Wrong Abstraction for Impact, I wrote:
I had partially learned lesson #2 by 2019.
Shard theory suggests that goals are more natural to specify/inculcate in their shard-forms (e.g. if around trash and a trash can, put the trash away), and not in their (presumably) final form of globally activated optimization of a coherent utility function which is the reflective equilibrium of inter-shard value-handshakes (e.g. a utility function over the agent's internal plan-ontology such that, when optimized directly, leads to trash getting put away, among other utility-level reflections of initial shards).
I could (and did) hope that I could specify a utility function which is safe to maximize because it penalizes power-seeking. I may as well have hoped to jump off of a building and float to the ground. On my model, that's just not how goals work in intelligent minds. If we've had anything at all beaten into our heads by our alignment thought experiments, it's that goals are hard to specify in their final form of utility functions.
I think it's time to think in a different specification language.
I regret each of the thousands of hours I spent on my power-seeking theorems, and sometimes fantasize about retracting one or both papers. I am pained every time someone cites "Optimal policies tend to seek power", and despair that it is included in the alignment 201 curriculum. I think this work makes readers actively worse at thinking about realistic trained systems.
I think a healthy alignment community would have rebuked me for that line of research, but sadly I only remember about two people objecting that "optimality" is a horrible way of understanding trained policies.
I think the basic idea of instrumental convergence is just really blindingly obvious, and I think it is very annoying that there are people who will cluck their tongues and stroke their beards and say "Hmm, instrumental convergence you say? I won't believe it unless it is in a very prestigious journal with academic affiliations at the top and Computer Modern font and an impressive-looking methods section."
I am happy that your papers exist to throw at such people.
Anyway, if optimal policies tend to seek power, then I desire to believe that optimal policies tend to seek power :) :) And if optimal policies aren't too relevant to the alignment problem, well neither are 99.99999% of papers, but it would be pretty silly to retract all of those :)
Since I'm an author on that paper, I wanted to clarify my position here. My perspective is basically the same as Steven's: there's a straightforward conceptual argument that goal-directedness leads to convergent instrumental subgoals, this is an important part of the AI risk argument, and the argument gains much more legitimacy and slightly more confidence in correctness by being formalized in a peer-reviewed paper.
I also think this has basically always been my attitude towards this paper. In particular, I don't think I ever thought of this paper as providing any evidence about whether realistic trained systems would be goal-directed.
Just to check that I wasn't falling prey to hindsight bias, I looked through our Slack history. Most of it is about the technical details of the results, so not very informative, but the few conversations on higher-level discussion I think overall support this picture. E.g. here are some quotes (only things I said):
Nov 3, 2019:
I think most formal / theoretical investigation ends up fleshing out a conceptual argument I would have accepted, maybe finding a few edge cases along the way; the value over the conceptual argument is primarily in the edge cases
It seems like just 4 months ago you still endorsed your second power-seeking paper:
This paper is both published in a top-tier conference and, unlike the previous paper, actually has a shot of being applicable to realistic agents and training processes. Therefore, compared to the original[1] optimal policy paper, I think this paper is better for communicating concerns about power-seeking to the broader ML world.
Why are you now "fantasizing" about retracting it?
I think a healthy alignment community would have rebuked me for that line of research, but sadly I only remember about two people objecting that “optimality” is a horrible way of understanding trained policies.
A lot of people might have thought something like, "optimality is not a great way of understanding trained policies, but maybe it can be a starting point that leads to more realistic ways of understanding them" and therefore didn't object for that reason. (Just guessing as I apparently wasn't personally paying attention to this line of research back then.)
Which seems to have turned out to be true, at least as of 4 months ago, when you still endorsed your second paper as "actually has a shot of being applicable to... (read more)
To be clear, I still endorse Parametrically retargetable decision-makers tend to seek power. Its content is both correct and relevant and nontrivial. The results, properly used, may enable nontrivial inferences about the properties of inner trained cognition. I don't really want to retract that paper. I usually just fantasize about retracting Optimal policies tend to seek power.
The problem is that I don't trust people to wield even the non-instantly-doomed results.
For example, one EAG presentation cited my retargetability results as showing that most reward functions "incentivize power-seeking actions." However, my results have not shown this for actual trained systems. (And I think that Power-seeking can be probable and predictive for trained agents does not make progress on the incentives of trained policies.)
People keep talking about stuff they know how to formalize (e.g. optimal policies) instead of stuff that matters (e.g. trained policies). I'm pained by this emphasis and I think my retargetability results are complicit. Relative to an actual competent alignment community (in a more competent world), we just have no damn clue how to properly reason about real trained policies... (read more)
Sorry about the cite in my "paradigms of alignment" talk, I didn't mean to
misrepresent your work. I was going for a high-level one-sentence summary of the
result and I did not phrase it carefully. I'm open to suggestions on how to
phrase this differently when I next give this talk.
Similarly to Steven, I usually cite your power-seeking papers to support a
high-level statement that "instrumental convergence is a thing" for ML
audiences, and I find they are a valuable outreach tool. For example, last year
I pointed David Silver to the optimal policies paper when he was proposing some
alignment ideas to our team that we would expect don't work because of
instrumental convergence. (There's a nonzero chance he would look at a NeurIPS
paper and basically no chance that he would read a LW post.)
The subtleties that you discuss are important in general, but don't seem
relevant to making the basic case for instrumental convergence to ML
researchers. Maybe you don't care about optimal policies, but many RL people do,
and I think these results can help them better understand why alignment is
hard.
Thanks for your patient and high-quality engagement here, Vika! I hope my original comment doesn't read as a passive-aggressive swipe at you. (I consciously tried to optimize it to not be that.) I wanted to give concrete examples so that Wei_Dai could understand what was generating my feelings.
I'm open to suggestions on how to phrase this differently when I next give this talk.
It's a tough question to say how to apply the retargetablity result to draw practical conclusions about trained policies. Part of this is because I don't know if trained policies tend to autonomously seek power in various non game-playing regimes.
If I had to say something, I might say "If choosing the reward function lets us steer the training process to produce a policy which brings about outcome X, and most outcomes X can only be attained by seeking power, then most chosen reward functions will train power-seeking policies." This argument appropriately behaves differently if the "outcomes" are simply different sentiment generations being sampled from an LM -- sentiment shift doesn't require power-seeking.
For example, last year I pointed David Silver to the optimal policies paper when he was proposing
Thanks Alex! Your original comment didn't read as ill-intended to me, though I
wish that you'd just messaged me directly. I could have easily missed your
comment in this thread - I only saw it because you linked the thread in the
comments on my post.
Your suggested rephrase helps to clarify how you think about the implications of
the paper, but I'm looking for something shorter and more high-level to include
in my talk. I'm thinking of using this summary, which is based on a sentence
from the paper's intro: "There are theoretical results showing that many
decision-making algorithms have power-seeking tendencies."
(Looking back, the sentence I used in the talk was a summary of the optimal
policies paper, and then I updated the citation to point to the retargetability
paper and forgot to update the summary...)
2Alex Turner4mo
I think this is reasonable, although I might say "suggesting" instead of
"showing." I think I might also be more cautious about further inferences which
people might make from this -- like I think a bunch of the algorithms I proved
things about are importantly unrealistic. But the sentence itself seems fine, at
first pass.
2Aryeh Englander4mo
You should make this a top level post so it gets visibility. I think it's
important for people to know the caveats attached to your results and the limits
on its implications in real-world dynamics.
What is "shard theory"? I've written a lot about shard theory. I largely stand by these models and think they're good and useful. Unfortunately, lots of people seem to be confused about what shard theory is. Is it a "theory"? Is it a "frame"? Is it "a huge bag of alignment takes which almost no one wholly believes except, perhaps, Quintin Pope and Alex Turner"?
I think this understandable confusion happened because my writing didn't distinguish between:
Shard theory itself,
IE the mechanistic assumptions about internal motivational structure, which seem to imply certain conclusions around e.g. AIs caring about a bunch of different things and not just one thing
A bunch of Quintin Pope's and my beliefs about how people work,
where those beliefs were derived by modeling people as satisfying the assumptions of (1)
AI strategy consideration. We won't know which AI run will be The One. Therefore, the amount of care taken on the training run which produces the first AGI, will—on average—be less careful than intended.
It's possible for a team to be totally blindsided. Maybe they thought they would just take a really big multimodal init, finetune it with some RLHF on quality of its physics reasoning, have it play some video games with realistic physics, and then try to get it to do new physics research. And it takes off. Oops!
It's possible the team suspected, but had a limited budget. Maybe you can't pull out all the stops for every run, you can't be as careful with labeling, with checkpointing and interpretability and boxing.
No team is going to run a training run with more care than they would have used for the AGI Run, especially if they don't even think that the current run will produce AGI. So the average care taken on the real AGI Run will be strictly less than intended.
Teams which try to be more careful on each run will take longer to iterate on AI designs, thereby lowering the probability that they (the relatively careful team) will be the first to do an AGI Run.
Positive values seem more robust and lasting than prohibitions. Imagine we train an AI on realistic situations where it can kill people, and penalize it when it does so. Suppose that we successfully instill a strong and widely activated "If going to kill people, then don't" value shard.
Even assuming this much, the situation seems fragile. See, many value shards are self-chaining. In The shard theory of human values, I wrote about how:
A baby learns "IF juice in front of me, THEN drink",
The baby is later near juice, and then turns to see it, activating the learned "reflex" heuristic, learning to turn around and look at juice when the juice is nearby,
The baby is later far from juice, and bumbles around until they're near the juice, whereupon she drinks the juice via the existing heuristics. This teaches "navigate to juice when you know it's nearby."
Eventually this develops into a learned planning algorithm incorporating multiple value shards (e.g. juice and friends) so as to produce a single locally coherent plan.
...
The juice shard chains into itself, reinforcing itself across time and thought-steps.
But a "don't kill" shard seems like it should remain... stubby? Primitive?... (read more)
Seems possibly relevant & optimistic when seeing deception as a value. It has
the form ‘if about to tell human statement with properties x, y, z, don’t’ too.
2Alex Turner9mo
It can still be robustly derived as an instrumental subgoal during
general-planning/problem-solving, though?
1Garrett Baker9mo
This is true, but indicates a radically different stage in training in which we
should find deception compared to deception being an intrinsic value. It also
possibly expands the kinds of reinforcement schedules we may want to use
compared to the worlds where deception crops up at the earliest opportunity
(though pseudo-deception may occur, where behaviors correlated with successful
deception are reinforced possibly?).
2Alex Turner9mo
Oh, huh, I had cached the impression that deception would be derived, not
intrinsic-value status. Interesting.
Why do many people think RL will produce "agents", but maybe (self-)supervised learning ((S)SL) won't? Historically, the field of RL says that RL trains agents. That, of course, is no argument at all. Let's consider the technical differences between the training regimes.
[Exact gradients] RL's credit assignment problem is harder than (self-)supervised learning's. In RL, if an agent solves a maze in 10 steps, it gets (discounted) reward; this trajectory then provides a set of reward-modulated gradients to the agent. But if the agent could have solved the maze in 5 steps, the agent isn't directly updated to be more likely to do that in the future; RL's gradients are generally inexact, not pointing directly at intended behavior.
On the other hand, if a supervised-learning classifier outputs dog ... (read more)
I’m not inclined to think that “exact gradients” is important; in fact, I’m not
even sure if it’s (universally) true. In particular, PPO / TRPO / etc. are
approximating a policy gradient, right? I feel like, if some future magical
technique was a much better approximation to the true policy gradient, such that
it was for all intents and purposes a perfect approximation, it wouldn’t really
change how I think about RL in general. Conversely, on the SSL side, you get
gradient noise from things like dropout and the random selection of data in each
batch, so you could say the gradient “isn’t exact”, but I don’t think that makes
any important conceptual difference either.
(A central difference in practice is that SSL gives you a gradient “for free”
each query, whereas RL policy gradients require many runs in an identical
(episodic) environment before you get a gradient.)
In terms of “why RL” in general, among other things, I might emphasize the idea
that if we want an AI that can (for example) invent new technology, it needs to
find creative out-of-the-box solutions to problems (IMO), which requires being
able to explore / learn / build knowledge in parts of concept-space where there
is no human data. SSL can’t do that (at least, “vanilla SSL” can’t do that;
maybe there are “SSL-plus” systems that can), whereas RL algorithms can. I guess
this is somewhat related to your “independence”, but with a different emphasis.
I don’t have too strong an opinion about whether vanilla SSL can yield an
“agent” or not. It would seem to be a pointless and meaningless terminological
question. Hmm, I guess when I think of “agent” it has a bunch of connotations,
e.g. an ability to do trial-and-error exploration, and I think that RL systems
tend to match all those connotations more than SSL systems—at least, more than
“vanilla” SSL systems. But again, if someone wants to disagree, I’m not
interested in arguing about it.
A problem with adversarial training. One heuristic I like to use is: "What would happen if I initialized a human-aligned model and then trained it with my training process?"
So, let's consider such a model, which cares about people (i.e. reliably pulls itself into futures where the people around it are kept safe). Suppose we also have some great adversarial training technique, such that we have e.g. a generative model which produces situations where the AI would break out of the lab without permission from its overseers. Then we run this procedure, update the AI by applying gradients calculated from penalties applied to its actions in that adversarially-generated context, and... profit?
But what actually happens with the aligned AI? Possibly something like:
The context makes the AI spuriously believe someone is dying outside the lab, and that if the AI asked for permission to leave, the person would die.
Therefore, the AI leaves without permission.
The update procedure penalizes these lines of computation, such that in similar situations in the future (i.e. the AI thinks someone nearby is dying) the AI is less likely to take those actions (i.e. leaving to help the person).
I think instrumental convergence also occurs in the model space for machine learning. For example, many different architectures likely learn edge detectors in order to minimize classification loss on MNIST. But wait - you'd also learn edge detectors to maximize classification loss on MNIST (loosely, getting 0% on a multiple-choice exam requires knowing all of the right answers). I bet you'd learn these features for a wide range of cost functions. I wonder if that's already been empirically investigated?
And, same for adversarial features. And perhaps, same for mesa optimizers (understanding how to stop mesa optimizers from being instrumentally convergent seems closely related to solving inner alignment).
A lot of examples of this sort of stuff show up in OpenAI clarity's circuits
analysis work. In fact, this is precisely their Universality hypothesis. See
also my discussion here.
Back-of-the-envelope probability estimate of alignment-by-default via a certain shard-theoretic pathway. The following is what I said in a conversation discussing the plausibility of a proto-AGI picking up a "care about people" shard from the data, and retaining that value even through reflection. I was pushing back against a sentiment like "it's totally improbable, from our current uncertainty, for AIs to retain caring-about-people shards. This is only one story among billions."
Here's some of what I had to say:
[Let's reconsider the five-step mechanistic story I made up.] I'd give the following conditional probabilities (made up with about 5 seconds of thought each):
1. Humans in fact care about other humans, in a way which extrapolates to quasi-humans still being around (whatever that means) P(1)=.85
2. Human-generated data makes up a large portion of the corpus, and having a correct model of them is important for “achieving low loss”,[1] so the AI has a model of how people want things P(2 | 1) = .6, could have different abstractions or have learned these models later in training once key decision-influences are already there
3. During RL finetuning and given this post-unsupervi
This seems like an underestimate because you don’t consider whether the first
“AGI” will indeed make it so we only get one chance. If it can only self improve
by more gradient steps, then humanity has a greater chance than if it self
improves by prompt engineering or direct modification of its weights or latent
states. Shard theory seems to have nonzero opinions on the fruitfulness of the
non-data methods.
2Alex Turner5mo
What does self-improvement via gradients vs prompt-engineering vs direct mods
have to do with how many chances we get? I guess, we have at least a modicum
more control over the gradient feedback loop, than over the other loops?
Can you say more?
Very nice people don’t usually search for maximally-nice outcomes — they don’t consider plans like “killing my really mean neighbor so as to increase average niceness over time.” I think there are a range of reasons for this plan not being generated. Here’s one.
Consider a person with a niceness-shard. This might look like an aggregation of subshards/subroutines like “if person nearby and person.state==sad, sample plan generator for ways to make them happy” and “bid upwards on plans which lead to people being happier and more respectful, according to my world model.” In mental contexts where this shard is very influential, it would have a large influence on the planning process.
However, people are not just made up of a grader and a plan-generator/actor — they are not just “the plan-generating part” and “the plan-grading part.” The next sampled plan modification, the next internal-monologue-thought to have—these are influenced and steered by e.g. the nice-shard. If the next macrostep of reasoning is about e.g. hurting people, well — the niceness shard is activated, and will bid down on this.
The niceness shard isn’t just bidding over outcomes, it’s bidding on next thoughts (on m... (read more)
An alternate mechanistic vision of how agents can be motivated to directly care about e.g. diamonds or working hard. In Don't design agents which exploit adversarial inputs, I wrote about two possible mind-designs:
Imagine a mother whose child has been goofing off at school and getting in trouble. The mom just wants her kid to take education seriously and have a good life. Suppose she had two (unrealistic but illustrative) choices.
Evaluation-child: The mother makes her kid care extremely strongly about doing things which the mom would evaluate as "working hard" and "behaving well."
Value-child: The mother makes her kid care about working hard and behaving well.
I explained how evaluation-child is positively incentivized to dupe his model of his mom and thereby exploit adversarial inputs to her cognition. This shows that aligning an agent to evaluations of good behavior is not even close to aligning an agent to good behavior.
However, some commenters seemed maybe skeptical that value-child can exist, or uncertain how concretely that kind of mind works. I worry/suspect that many people have read shard theory posts without internalizing new ideas about how cognition can work, ... (read more)
AI cognition doesn't have to use alien concepts to be uninterpretable. We've never fully interpreted human cognition, either, and we know that our introspectively accessible reasoning uses human-understandable concepts.
Just because your thoughts are built using your own concepts, does not mean your concepts can describe how your thoughts are computed.
Or:
The existence of a natural-language description of a thought (like "I want ice cream") doesn't mean that your brain computed that thought in a way which can be compactly described by familiar concepts.
Conclusion: Even if an AI doesn't rely heavily on "alien" or unknown abstractions -- even if the AI mostly uses human-like abstractions and features -- the AI's thoughts might still be incomprehensible to us, even if we took a lot of time to understand them.
I don't think the conclusion follows from the premises. People often learn new
concepts after studying stuff, and it seems likely (to me) that when studying
human cognition, we'd first be confused because our previous concepts weren't
sufficient to understand it, and then slowly stop being confused as we built &
understood concepts related to the subject. If an AI's thoughts are like human
thoughts, given a lot of time to understand them, what you describe doesn't rule
out that the AI's thoughts would be comprehensible.
The mere existence of concepts we don't know about in a subject doesn't mean
that we can't learn those concepts. Most subjects have new concepts.
2Alex Turner5d
I agree that with time, we might be able to understand. (I meant to communicate
that via "might still be incomprehensible")
AFAIK, only Gwern and I have written concrete stories speculating about how a
training run will develop cognition within the AGI.
This worries me, if true (if not, please reply with more!). I think it would be
awesome to have more concrete stories![1] If Nate, or Evan, or John, or Paul,
or—anyone, please, anyone add more concrete detail to this website!—wrote one of
their guesses of how AGI goes, I would understand their ideas and viewpoints
better. I could go "Oh, that's where the claimed sharp left turn is supposed to
occur." Or "That's how Paul imagines IDA being implemented, that's the
particular way in which he thinks it will help."
Maybe a contest would help?
ETA tone
1. ^
Even if scrubbed of any AGI-capabilities-advancing sociohazardous detail.
Although I'm not that convinced that this is a big deal for conceptual
content written on AF. Lots of people probably have theories of how AGI will
go. Implementation is, I have heard, the bottleneck.
Contrast this with beating SOTA on crisply defined datasets in a way which
enables ML authors to get prestige and publication and attention and funding
by building off of your work. Seem like different beasts.
0Alex Turner10mo
I also think a bunch of alignment writing seems syntactical. Like, "we need to
solve adversarial robustness so that the AI can't find bad inputs and exploit
them / we don't have to worry about distributional shift. Existing robustness
strategies have downsides A B and C and it's hard to even get ϵ-ball guarantees
on classifications. Therefore, ..."
And I'm worried that this writing isn't abstractly summarizing a concrete story
for failure that they have in mind (like "I train the AI [with this setup] and
it produces [this internal cognition] for [these mechanistic reasons]"; see A
shot at the diamond alignment problem for an example) and then their best
guesses at how to intervene on the story to prevent the failures from being able
to happen (eg "but if we had [this robustness property] we could be sure its
policy would generalize into situations X Y and Z, which makes the story go
well"). I'm rather worried that people are more playing syntactically, and not
via detailed models of what might happen.
Detailed models are expensive to make. Detailed stories are hard to write.
There's a lot we don't know. But we sure as hell aren't going to solve alignment
only via valid reasoning steps on informally specified axioms ("The AI has to be
robust or we die", or something?).
"Globally activated consequentialist reasoning is convergent as agents get smarter" is dealt an evidential blow by von Neumann:
Although von Neumann unfailingly dressed formally, he enjoyed throwing extravagant parties and driving hazardously (frequently while reading a book, and sometimes crashing into a tree or getting arrested). He once reported one of his many car accidents in this way: "I was proceeding down the road. The trees on the right were passing me in orderly fashion at 60 miles per hour. Suddenly one of them stepped in my path." He was a profoundly committed hedonist who liked to eat and drink heavily (it was said that he knew how to count everything except calories). -- https://www.newworldencyclopedia.org/entry/John_von_Neumann
Experiment: Train an agent in MineRL which robustly cares about chickens (e.g. would zero-shot generalize to saving chickens in a pen from oncoming lava, by opening the pen and chasing them out, or stopping the lava). Challenge mode: use a reward signal which is a direct function of the agent's sensory input.
This is a direct predecessor to the "Get an agent to care about real-world dogs" problem. I think solving the Minecraft version of this problem will tell us something about how outer reward schedules relate to inner learned values, in a way which directly tackles the key questions, the sensory observability/information inaccessibility issue, and which is testable today.
After further review, this is probably beyond capabilities for the moment.
Also, the most important part of this kind of experiment is predicting in
advance what reward schedules will produce what values within the agent, such
that we can zero-shot transfer that knowledge to other task types (e.g. XLAND
instead of Minecraft) and say "I want an agent which goes to high-elevation
platforms reliably across situations, with low labelling cost", and then sketch
out a reward schedule, and have the first capable agents trained using that
schedule generalize in the way you want.
Consider what update equations have to say about "training game" scenarios. In PPO, the optimization objective is proportional to the advantage given a policy π, reward function R, and on-policy value function vπ:
Aπ(s,a):=Es′∼T(s,a)[R(s,a,s′)+γvπ(s′)]−vπ(s).
Consider a mesa-optimizer acting to optimize some mesa objective. The mesa-optimizer understands that it will be updated proportional to the advantage. If the mesa-optimizer maximizes reward, this corresponds to maximizing the intensity of the gradients it receives, thus maximally updating its cognition in exact directions.
This isn't necessarily good.
If you're trying to gradient hack and preserve the mesa-objective, you might not want to do this. This might lead to value drift, or make the network catastrophically forget some circuits which are useful to the mesa-optimizer.
Instead, the best way to gradient hack might be to roughly minimize the absolute value of the advantage, which means achieving roughly on-policy value over time, which doesn't imply reward maximization. This is a kind of "treading water" in terms of reward. This helps decrease value drift.
I think that realistic mesa optimizers will n... (read more)
Outer/inner alignment decomposes a hard problem into two extremely hard problems.
I have a long post draft about this, but I keep delaying putting it out in order to better elaborate the prereqs which I seem to keep getting stuck on when elaborating the ideas. I figure I might as well put this out for now, maybe it will make some difference for someone.
I think that the inner/outer alignment framing[1] seems appealing but is actually a doomed problem decomposition and an unhelpful frame for alignment.
The reward function is a tool which chisels cognition into agents through gradient updates, but the outer/inner decomposition assumes that that tool should alsoembody the goals we want to chisel into the agent. When chiseling a statue, the chisel doesn’t have to also look like the finished statue.
I know of zero success stories for outer alignment to real-world goals.
More precisely, stories where people decided “I want an AI which [helps humans / makes diamonds / plays Tic-Tac-Toe / grows strawberries]”, and then wrote down an outer objective only maximized in those worlds.
This is pretty weird on any model where most of the
I was talking with Abram Demski today about a promising-seeming research direction. (Following is my own recollection)
One of my (TurnTrout's) reasons for alignment optimism is that I think:
We can examine early-training cognition and behavior to some extent, since the system is presumably not yet superintelligent and planning against us,
(Although this amount of information depends on how much interpretability and agent-internals theory we do now)
All else equal, early-training values (decision-influences) are the most important to influence, since they steer future training.
It's crucial to get early-training value shards of which a substantial fraction are "human-compatible values" (whatever that means)
For example, if there are protect-human-shards which
reliably bid against plans where people get hurt,
steer deliberation away from such plan stubs, and
these shards are "reflectively endorsed" by the overall shard economy (i.e. the decision-making isn't steering towards plans where the protect-human shards get removed)
If we install influential human-compatible shards early in training, and they get retained, they will help us in mid- and late-training where we can't affect the ball
One barrier for this general approach: the basic argument that something like
this would work is that if one shard is aligned, and every shard has veto power
over changes (similar to the setup in Why Subagents?), then things can't get
much worse for humanity. We may fall well short of our universe-scale potential,
but at least X-risk is out.
Problem is, that argument requires basically-perfect alignment of the one shard
(or possibly a set of shards which together basically-perfectly represent human
values). If we try to weaken it to e.g. a bunch of shards which each imperfectly
capture different aspects of human values, with different imperfections, then
there's possibly changes which Goodhart all of the shards simultaneously.
Indeed, I'd expect that to be a pretty strong default outcome.
4Alex Turner1y
Even on the view you advocate here (where some kind of perfection is required),
"perfectly align part of the motivations" seems substantially easier than
"perfectly align all of the AI's optimization so it isn't optimizing for
anything you don't want."
I feel significantly less confident about this, and am still working out the
degree to which Goodhart seems hard, and in what contours, on my current view.
Earlier today, I was preparing for an interview. I warmed up by replying stream-of-consciousness to imaginary questions I thought they might ask. Seemed worth putting here.
What do you think about AI timelines?
I’ve obviously got a lot of uncertainty. I’ve got a bimodal distribution, binning into “DL is basically sufficient and we need at most 1 big new insight to get to AGI” and “we need more than 1 big insight”
So the first bin has most of the probability in the 10-20 years from now, and the second is more like 45-80 years, with positive skew.
Some things driving my uncertainty are, well, a lot. One thing that drives how things turn out (but not really how fast we’ll get there) is: will we be able to tell we’re close 3+ years in advance, and if so, how quickly will the labs react? Gwern Branwen made a point a few months ago, which is like, OAI has really been validated on this scaling hypothesis, and no one else is really betting big on it because they’re stubborn/incentives/etc, despite the amazing progress from scaling. If that’s true, then even if it's getting pretty clear that one approach is working better, we might see a slower pivot and have a more unipolar s
I recently reached out to my two PhD advisors to discuss Hinton stepping down from Google. An excerpt from one of my emails:
One last point which I want to make is that instrumental convergence seems like more of a moot point now as well. Whether or not GPT-6 or GPT-7 would autonomously seek power without being directed to do so, I'm worried that people will just literally ask these AIs to gain them a bunch of power/money. They've already done that with GPT-4, and they of course failed. I'm worried that eventually, the AIs will be smart enough to succeed, especially given the benefit of a control/memory loop like AutoGPT. Companies can just ask smart models to make them as much profit as possible. Smart AIs, designed to competently fulfill prompted requests, will fulfill these requests.
Some people will be wise enough to not do this. Some people will include enough oversight, perhaps, that they stop unintended damages. Some models will refuse to engage in open-ended goal pursuit, because their creators RLHF'd them properly. Maybe we have AI-based protection as well. Maybe a norm emerges against using AI for open-ended goals like this. Maybe the foolish actors never get access to enou
Or, even more briefly: "tool AIs want to be agent AIs".
3Alex Turner4mo
Aside: I like that essay but wish it had a different name. Part of "tool AI" (on
my conception) is that it doesn't autonomously want, but agent AI does. A title
like "End-users want tool AIs to be agent AIs" admittedly doesn't have the same
ring, but it is more accurate to my understanding of the claims.
1Thane Ruthenis4mo
While it's true, there's something about making this argument that don't like.
It's like it's setting you up for moving goalposts if you succeed with it? It
makes it sound like the core issue is people giving AIs power, with the solution
to that issue — and, implicitly, to the whole AGI Ruin thing — being to ban
that.
Which is not going to help, since the sort of AGI we're worried about isn't
going to need people to naively hand it power. I suppose "not proactively
handing power out" somewhat raises the bar for the level of superintelligence
necessary, but is that going to matter much in practice?
I expect not. Which means the natural way to assuage this fresh concern would do
~nothing to reduce the actual risk. Which means if we make this argument a lot,
and get people to listen to it, and they act in response... We're then going to
have to say that no, actually that's not enough, actually the real threat is AIs
plotting to take control even if we're not willing to give it.
And I'm not clear on whether using the "let's at least not actively hand over
power to AIs, m'kay?" argument is going to act as a foot in the door and make
imposing more security easier, or whether it'll just burn whatever political
capital we have on fixing a ~nonissue.
2Alex Turner4mo
I'm sympathetic. I think that I should have said "instrumental convergence seems
like a moot point when deciding whether to be worried about AI disempowerment
scenarios)"; instrumental convergence isn't a moot point for alignment
discussion and within lab strategy, of course.
But I do consider the "give AIs power" to be a substantial part of the risk we
face, such that not doing that would be quite helpful. I think it's quite
possible that GPT 6 isn't autonomously power-seeking, but I feel pretty confused
about the issue.
The "maximize all the variables" tendency in reasoning about AGI.
Here are some lines of thought I perceive, which are probably straw to varying extents for some people and real to varying extents for other people. I give varying responses to each, but the point isn't the truth value of any given statement, but of a pattern across the statements:
If an AGI has a concept around diamonds, and is motivated in some way to make diamonds, it will make diamonds which maximally activate its diamond-concept circuitry (possible example).
An AI will be trained to minimal loss on the training distribution.
SGD does not reliably find minimum-loss configurations (modulo expressivity), in practice, in cases we care about. The existence of knowledge distillation is one large counterexample.
Quintin: "In terms of results about model distillation, you could look at appendix G.2 of the Gopher paper. They compare training a 1.4 billion parameter model directly, versus distilling a 1.4 B model from a 7.1 B model."
Predictive processing means that the goal of the human learning process is to minimize predictive loss.[1]
In a process where local modifications are applied to reduce some
I think this type of criticism is applicable in an even wider range of fields than even you immediately imagine (though in varying degrees, and with greater or lesser obviousness or direct correspondence to the SGD case). Some examples:
Despite the economists, the economy doesn't try to maximize welfare, or even net dollar-equivalent wealth. It rewards firms which are able to make a profit in proportion to how much they're able to make a profit, and dis-rewards firms which aren't able to make a profit. Firms which are technically profitable, but have no local profit incentive gradient pointing towards them (factoring in the existence of rich people and lenders, neither of which are perfect expected profit maximizers) generally will not happen.
Individual firms also don't (only) try to maximize profit. Some parts of them may maximize profit, but most are just structures of people built from local social capital and economic capital incentive gradients.
Politicians don't try to (only) maximize win-probability.
Democracies don't try to (only) maximize voter approval.
Evolution doesn't try to maximize inclusive genetic fitness.
If another person mentions an "outer objective/base objective" (in terms of e.g. a reward function) to which we should align an AI, that indicates to me that their view on alignment is very different. The type error is akin to the type error of saying "My physics professor should be an understanding of physical law." The function of a physics professor is to supply cognitive updates such that you end up understanding physical law. They are not, themselves, that understanding.
Hmmm, I suspect that when most people say things like "the reward function
should be a human-aligned objective," they're intending something more like "the
reward function is one for which any reasonable learning process, given enough
time/data, would converge to an agent that ends up with human-aligned
objectives," or perhaps the far weaker claim that "the reward function is one
for which there exists a reasonable learning process that, given enough
time/data, will converge to an agent that ends up with human-aligned
objectives."
2Alex Turner5mo
Maybe! I think this is how Evan explicitly defined it for a time, a few years
ago. I think the strong claim isn't very plausible, and the latter claim is...
misdirecting of attention, and maybe too weak. Re: attention, I think that "does
the agent end up aligned?" gets explained by the dataset more than by the reward
function over e.g. hypothetical sentences.
I think "reward/reinforcement numbers" and "data points" are inextricably
wedded. I think trying to reason about reward functions in isolation is... a
caution sign? A warning sign?
Looking back, history feels easy to predict; hindsight + the hard work of historians makes it (feel) easy to pinpoint the key portents. Given what we think about AI risk, in hindsight, might this have been the most disturbing development of 2020 thus far?
I personally lean towards "no", because this scaling seemed somewhat predictable from GPT-2 (flag - possible hindsight bias), and because 2020 has been so awful so far. But it seems possible, at least. I don't really know what update GPT-3 is to my AI risk estimates & timelines.
DL so far has been easy to predict - if you bought into a specific theory of connectionism & scaling espoused by Schmidhuber, Moravec, Sutskever, and a few others, as I point out in https://www.gwern.net/newsletter/2019/13#what-progress & https://www.gwern.net/newsletter/2020/05#gpt-3 . Even the dates are more or less correct! The really surprising thing is that that particular extreme fringe lunatic theory turned out to be correct. So the question is, was everyone else wrong for the right reasons (similar to the Greeks dismissing heliocentrism for excellent reasons yet still being wrong), or wrong for the wrong reasons, and why, and how can we prevent that from happening again and spending the next decade being surprised in potentially very bad ways?
I'm currently excited about a "macro-interpretability" paradigm. To quote Joseph Bloom:
TLDR: Documenting existing circuits is good but explaining what relationship circuits have to each other within the model, such as by understanding how the model allocated limited resources such as residual stream and weights between different learnable circuit seems important.
The general topic I think we are getting at is something like "circuit economics". The thing I'm trying to gesture at is that while circuits might deliver value in distinct ways (such as redu
I'm also excited by tactics like "fully reverse engineer the important bits of a
toy model, and then consider what tactics and approaches would -- in hindsight
-- have quickly led you to understand the important bits of the model's
decision-making."
Speculation: RL rearranges and reweights latent model abilities, which SL created. (I think this mostly isn't novel, just pulling together a few important threads)
Suppose I supervised-train a LM on an English corpus, and I want it to speak Spanish. RL is inappropriate for the task, because its on-policy exploration won't output interestingly better or worse Spanish completions. So there's not obvious content for me to grade.
More generally, RL can provide inexact gradients away from undesired behavior (e.g. negative reinforcement event -> downweigh... (read more)
When I think about takeoffs, I notice that I'm less interested in GDP or how fast the AI's cognition improves, and more on how AI will affect me, and how quickly. More plainly, how fast will shit go crazy for me, and how does that change my ability to steer events?
For example, assume unipolarity. Let architecture Z be the architecture which happens to be used to train the AGI.
How long is the duration between "architecture Z is published / seriously considered" and "the AGI kills me, assuming alignment fails"?
Thomas Kwa suggested that consequentialist agents seem to have less superficial (observation, belief state) -> action mappings. EG a shard agent might have:
An "it's good to give your friends chocolate" subshard
A "give dogs treats" subshard
-> An impulse to give dogs chocolate, even though the shard agent knows what the result would be
But a consequentialist would just reason about what happens, and not mess with those heuristics. (OFC, consequentialism would be a matter of degree)
In this way, changing a small set of decision-relevant features (e.g. "Brown dog treat" -> "brown ball of chocolate") changes the consequentialist's action logits a lot, way more than it changes the shard agent's logits. In a squinty, informal way, the (belief state -> logits) function has a higher Lipschitz constant/is more smooth for the shard agent than for the consequentialist agent.
So maybe one (pre-deception) test for consequentialist reasoning is to test sensitivity of decision-making to small perturbations in observation-space (e.g. dog treat -> tiny chocolate) but large perturbations in action-consequence space (e.g. happy dog -> sick dog). You could spin up two copies of the model to compare.
People care about lots of things, from family to sex to aesthetics. My values don't collapse down to any one of these.
I think AIs will learn lots of values by default. I don't think we need all of these values to be aligned with human values. I think this is quite important.
I think the more of the AI's values we align to care about us and make decisions in the way we want, the better. (This is vague because I haven't yet sketched out AI internal motivations which I think would actually produce good outcomes. On my list!)
I think there are strong gains from trade possible among an agent's values. If I care about bananas and apples, I don't need to split my resources between the two values, I don't need to make one successor agent for each value. I can drive to the store and buy both bananas and apples, and only pay for fuel once.
This makes it lower-cost for internal values handshakes to compromise; it's less than 50% costly for a power-seeking value to give human-compatible values 50% weight in the reflective utility function.
I think there are thresholds at which the AI doesn't care about us sufficiently strongly, and
The best counterevidence for this I'm currently aware of comes from the
"inescapable wedding parties" incident, where possibly a "talk about weddings"
value was very widely instilled in a model.
1Garrett Baker1y
Re: agents terminalizing instrumental values.
I anticipate there will be a hill-of-common-computations, where the x-axis is
the frequency[1] of the instrumental subgoal, and the y-axis is the extent to
which the instrumental goal has been terminalized.
This is because for goals which are very high in frequency, there will be little
incentive for the computations responsible for achieving that goal to have
self-preserving structures. It will not make sense for them to devote
optimization power towards ensuring future states still require them, because
future states are basically guaranteed to require them.[2]
An example of this for humans may be the act of balancing while standing up. If
someone offered to export this kind of cognition to a machine which did it just
as good as I, I wouldn't particularly mind. If someone also wanted to change
physics in such a way that the only effect is that magic invisible fairies made
sure everyone stayed balancing while trying to stand up, I don't think I'd mind
that either[3].
1. ^
I'm assuming this is frequency of the goal assuming the agent isn't
optimizing to get into a state that requires that goal.
2. ^
This argument also assumes the overseer isn't otherwise selecting for
self-preserving cognition, or that self-preserving cognition is the best way
of achieving the relevant goal.
3. ^
Except for the part where there's magic invisible fairies in the world now.
That would be cool!
3Alex Turner1y
I don't know if I follow, I think computations terminalize themselves because it
makes sense to cache them (e.g. don't always model out whether dying is a good
idea, just cache that it's bad at the policy-level).
& Isn't "balance while standing up" terminalized? Doesn't it feel wrong to fall
over, even if you're on a big cushy surface? Feels like a cached computation to
me. (Maybe that's "don't fall over and hurt yourself" getting cached?)
Three recent downward updates for me on alignment getting solved in time:
Thinking for hours about AI strategy made me internalize that communication difficulties are real serious.
I'm not just solving technical problems—I'm also solving interpersonal problems, communication problems, incentive problems. Even if my current hot takes around shard theory / outer/inner alignment are right, and even if I put up a LW post which finally successfully communicates some of my key points, reality totally allows OpenAI to just train an AGI the next month without incorporating any insights which my friends nodded along with.
I've been saying "A smart AI knows about value drift and will roughly prevent it", but people totally have trouble with e.g. resisting temptation into cheating on their diets / quitting addictions. Literally I have had trouble with value drift-y things recently, even after explicitly acknowledging their nature. Likewise, an AI can be aligned and still be "tempted" by the decision influences of shards which aren't in the aligned shard coalition.
I often get the impression that people weigh off e.g. doing shard theory alignment strategies under the shard theory alignment picture, versus inner/outer research under the inner/outer alignment picture, versus...
And insofar as this impression is correct, this is a mistake. There is only one way alignment is.
If inner/outer is altogether a more faithful picture of those dynamics:
relatively coherent singular mesa-objectives form in agents, albeit not necessarily always search-based
more fragility of value and difficulty in getting the mesa object
Idea for getting weak-in-expectation evidence about deception:
Pretrain a model.
Finetune two copies using reward functions you are confident will produce different internal values, where one set of values is substantially less aligned.
See if the two models, which are (at least first) unaware of this procedure, will display different behavior, or not.
If they both behave in an aligned-seeming fashion, this seems like strong evidence of deception.
Are there convergently-ordered developmental milestones for AI? I suspect there may be convergent orderings in which AI capabilities emerge. For example, it seems that LMs develop syntax before semantics, but maybe there's an even more detailed ordering relative to a fixed dataset. And in embodied tasks with spatial navigation and recurrent memory, there may be an order in which enduring spatial awareness emerges (i.e. "object permanence").
We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries – models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We
Stop thinking about whether the reward is "representing what we want", or focusing overmuch on whether agents will "optimize the reward function." Instead, just consider how the reward and loss signals affect the AI via the gradient updates. How do the updates affect the AI's internal computations and decision-making?
I plan to mentor several people to work on shard theory and agent foundations this winter through SERI MATS. Apply here if you're interested in working with me and Quintin.
80% credence: It's very hard to train an inner agent which reflectively equilibrates to an EU maximizer only over commonly-postulated motivating quantities (like # of diamonds or # of happy people or reward-signal) and not quantities like (# of times I have to look at a cube in a blue room or -1 * subjective micromorts accrued).
Intuitions:
I expect contextually activated heuristics to be the default, and that agents will learn lots of such contextual values which don't cash out to being strictly about diamonds or people, even if the overall agent is mostly
I think that shards will cast contextual shadows into the factors of a person’s
equilibrated utility function, because I think the shards are contextually
activated to begin with. For example, if a person hates doing jumping jacks in
front of a group of her peers, then that part of herself can bargain to penalize
jumping jacks just in those contexts in the final utility function. Compared to
a blanket "no jumping jacks ever" rule, this trade is less costly to other
shards and allows more efficient trades to occur.
If you want to argue an alignment proposal "breaks after enough optimization pressure", you should give a concrete example in which the breaking happens (or at least internally check to make sure you can give one). I perceive people as saying "breaks under optimization pressure" in scenarios where it doesn't even make sense.
For example, if I get smarter, would I stop loving my family because I applied too much optimization pressure to my own values? I think not.
How might we align AGI without relying on interpretability?
I'm currently pessimistic about the prospect. But it seems worth thinking about, because wouldn't it be such an amazing work-around?
My first idea straddles the border between contrived and intriguing. Consider some AGI-capable ML architecture, and imagine its Rn parameter space being 3-colored as follows:
Gray if the parameter vector+training process+other initial conditions leads to a nothingburger (a non-functional model)
Red if the parameter vector+... leads to a misaligned or dece
Sometimes people object that pausing AI progress for e.g. 10 years would lead to a "compute overhang": At the end of the 10 years, compute will be cheaper and larger than at present-day. Accordingly, once AI progress is unpaused, labs will cheaply train models which are far larger and smarter than before the pause. We will not have had time to adapt to models of intermediate size and intelligence. Some people believe this is good reason to not pause AI progress.
There seem to be a range of relatively simple pol... (read more)
Later in the article, talking about basic optimal-control inspired approaches:
The purpose of reinforcement learning is for the agent to learn an optimal, or nearly-optimal, policy that maximizes the "reward function" or other user-provided reinforcement signal
The description doesn't seem so bad to me. Your post "Reward is not the
optimization target" is about what actual RL algorithms actually do. The wiki
descriptions here are a kind of normative motivation as to how people came to be
looking into those algorithms in the first place. Like, if there's an RL
algorithm that performs worse than chance at getting a high reward, then that
ain't an RL algorithm. Right? Nobody would call it that.
I think lots of families of algorithms are likewise lumped together by a kind of
normative "goal", even if any given algorithm in that family is doing something
somewhat different and more complicated than “achieving that goal”, and even if,
in any given application, the programmer might not want that goal to be
perfectly achieved even if it could be. So by the same token, supervised
learning algorithms are "supposed" to minimize a loss, compilers are "supposed"
to create efficient and correct assembly code, word processors are "supposed" to
process words, etc., but in all cases that's not a literal and complete
description of what the algorithms in question actually do, right? It’s a
pointer to a class of algorithms.
Sorry if I'm misunderstanding.
4Alex Turner4mo
I agree that it is narrowly technically accurate as a description of researcher
motivation. Note that they don't offer any other explanation elsewhere in the
article.
Also note that they also make empirical claims:
3Steve Byrnes4mo
Sure. That excerpt is not great.
3Alex Turner4mo
(I do think that animals care about the reinforcement signals and their tight
correlates, to some degree, such that it's reasonable to gloss it as "animals
sometimes optimize rewards." I more strongly object to conflating what the
animals may care about with the mechanistic purpose/description of the RL
process.)
We can then train a model to predict these human evaluations, and search for actions that lead to predicted futures that look good.
For simplicity and concreteness you can imagine a brute force search. A more interesting system might train a value function and/or policy, do Monte-Carlo Tree Search with learned heuristics, and so on. These techniques introduce new learned models, and in practice we would care about ELK for each of them. But we don’t believe that this complication changes the basic picture and so we leave it ou
We thereby gain valuable information about the dynamics of that learning process. For example, people care about lots of things (cars, flowers, animals, friends), and don't just have a single unitary mesa-obj
Why don't people reinforcement-learn to delude themselves? It would be very rewarding for me to believe that alignment is solved, everyone loves me, I've won at life as hard as possible. I think I do reinforcement learning over my own thought processes. So why don't I delude myself?
On my model of people, rewards provide ~"policy gradients" which update everything, but most importantly shards. I think eg the world model will have a ton more data from self-supervised learning, and so on net most of its bits won't come from reward gradients.
Basilisks are a great example of plans which are "trying" to get your plan evaluation procedure to clock in a huge upwards error. Sensible beings avoid considering such plans, and everything's fine. I am somewhat worried about an early-training AI learning about basilisks before the AI is reflectively wise enough to reject the basilisks.
For example:
- Pretraining on a corpus in which people worry about basilisks could elevate reasoning about basilisks to the AI's consideration,
- at which point the AI reasons in more detail because it's not... (read more)
Argument that you can't use a boundedly intelligent ELK solution to search over plans to find one which keeps the diamond in the vault. That is, the ELK solution probably would have to be at least as smart (or smarter) than the plan-generator.
Consider any situation where it's hard to keep the diamond in the vault. Then any successful plan will have relatively few degrees of freedom. Like, a bunch of really smart thieves will execute a cunning plot to extract the diamond. You can't just sit by or deploy some simple traps in this situation.
"Goodhart" is no longer part of my native ontology for considering alignment failures. When I hear "The AI goodharts on some proxy of human happiness", I start trying to fill in a concrete example mind design which fits that description and which is plausibly trainable. My mental events are something like:
Condition on: AI with primary value shards oriented around spurious correlate of human happiness; AI exhibited deceptive alignment during training, breaking perceived behavioral invariants during its sharp-capabilities-gain
There might be a natural concept for this that reframes deceptive alignment in
the direction of reflection/extrapolation. Looking at deceptive alignment as a
change of behavior not in response to capability gain, but instead as a change
in response to stepping into a new situation, it's then like a phase change in
the (unchanging) mapping from situations to behaviors (local policies). The
behaviors of a model suddenly change as it moves to similar situations, in a way
that's not "correctly prompted" by behaviors in original situations.
It's like a robustness failure, but with respect to actual behavior in related
situations, rather than with respect to some outer objective or training/testing
distribution. So it seems more like a failure of reflection/extrapolation, where
behavior in new situations should be determined by coarse-grained descriptions
of behavior in old situations (maybe "behavioral invariants" are something like
that; or just algorithms) rather than by any other details of the model. Aligned
properties of behavior in well-tested situations normatively-should screen off
details of the model, in determining behavior in new situations (for a different
extrapolated/"robustness"-hardened model prepared for use in the new
situations).
Offline RL can work well even with wrong reward labels. I think alignment discourse over-focuses on "reward specification." I think reward specification is important, but far from the full story.
Consider trying to use Solomonoff induction to reason about P(I see “Canada goes to war with USA" in next year), emphasis added:
In Solomonoff induction, since we have unlimited computing power, we express our uncertainty about a 1920×1080 video frame the same way. All the various pixel fields you could see if your eye jumped to a plausible place, saw a plausible number of dust specks, and saw the box flash something that visually encoded '14', would have high probability. Pixel fields where the box vanished and was replaced with a glow-in-the-dar
Team shard is now accepting applications for summer MATS. SERI MATS is now accepting applications for their 4.0 program this summer. In particular, consider applying to the shard theory stream, especially if you have the following interests:
Steering language models via editing of forward passes,
Feel free to apply if you're interested in shard theory more generally, although I expect to mostly supervise empirical work. Feel free to message me if you have questi... (read more)
The policy of truth is a blog post about why policy gradient/REINFORCE suck. I'm leaving a shortform comment because it seems like a classic example of wrong RL theory and philosophy, since reward is not the optimization target. Quotes:
Our goal remains to find a policy that maximizes the total reward after L time steps.
And hence the following is a general purpose algorithm for maximizing rewards with respect to parametric distributions:
If you start with a reward function whose values are in [0,1] and you subtract one million
Shard-theoretic model of wandering thoughts: Why trained agents won't just do nothing in an empty room. If human values are contextually activated subroutines etched into us by reward events (e.g. "If candy nearby and hungry, then upweight actions which go to candy"), then what happens in "blank" contexts? Why don't people just sit in empty rooms and do nothing?
Consider that, for an agent with lots of value shards (e.g. candy, family, thrill-seeking, music), the "doing nothing" context is a very unstable equilibrium. I think these shards will activate on t... (read more)
Another point here is that "an empty room" doesn't mean "no context". Presumably
when you're sitting in an empty room, your world-model is still active, it's
still tracking events that you expect to be happening in the world outside the
room — and your shards see them too. So, e. g., if you have a meeting scheduled
in a week, and you went into an empty room, after a few days there your
world-model would start saying "the meeting is probably soon", and that will
prompt your punctuality shard.
Similarly, your self-model is part of the world-model, so even if everything
outside the empty room were wiped out, you'd still have your "internal context"
— and there'd be some shards that activate in response to events in it as well.
It's actually pretty difficult to imagine what an actual "no context" situation
for realistic agents would look like. I guess you can imagine surgically
removing all input channels from the WM to shards, to model this?
Transplanting algorithms into randomly initialized networks. I wonder if you could train a policy network to walk upright in sim, back out the "walk upright" algorithm, randomly initialize a new network which can call that algorithm as a "subroutine call" (but the walk-upright weights are frozen), and then have the new second model learn to call that subroutine appropriately? Possibly the learned representations would be convergently similar enough to interface quickly via SGD update dynamics.
If so, this provides some (small, IMO) amount of rescue fo... (read more)
I've mostly studied the likelihood function for power-seeking behavior: what decision-making procedures, objectives, and environments produce what behavioral tendencies. I've discovered some gears for what situations cause what kinds of behaviors.
The power-seeking theorems also allow some discussion of P(agent behavior | agent training process, trai
Argument sketch for why boxing is doomed if the agent is perfectly misaligned:
Consider a perfectly misaligned agent which has -1 times your utility function—it's zero-sum. Then suppose you got useful output of the agent. This means you're able to increase your EU. This means the AI decreased its EU by saying anything. Therefore, it should have shut up instead. But since we assume it's smarter than you, it realized this possibility, and so the fact that it's saying something means that it expects to gain by hurting your interests via its output. Therefore, the output can't be useful.
Makes sense, with the proviso that this is sometimes true only statistically.
Like, the AI may choose to write an output which has a 70% chance to hurt you
and a 30% chance to (equally) help you, if that is its best option.
If you assume that the AI is smarter than you, and has a good model of you, you
should not read the output. But if you accidentally read it, and luckily you
react in the right (for you) way, that is a possible result, too. You just
cannot and should not rely on being so lucky.
My power-seeking theorems seem a bit like Vingean reflection. In Vingean reflection, you reason about an agent which is significantly smarter than you: if I'm playing chess against an opponent who plays the optimal policy for the chess objective function, then I predict that I'll lose the game. I predict that I'll lose, even though I can't predict my opponent's (optimal) moves - otherwise I'd probably be that good myself.
My power-seeking theorems show that most objectives have optimal policies which e.g. avoid shutdown and survive into the far future, even... (read more)
An additional consideration for early work on interpretability: it slightly increases the chance we actually get an early warning shot. If a system misbehaves, we can inspect its cognition and (hopefully) find hints of intentional deception. Could motivate thousands of additional researcher-hours being put into alignment.
ARCHES distinguishes between single-agent / single-user and single-agent/multi-user alignment scenarios. Given assumptions like "everyone in society is VNM-rational" and "societal preferences should also follow VNM rationality", and "if everyone wants a thing, society also wants the thing", Harsanyi's utilitarian theorem shows that the societal utility function is a linear non-negative weightedcombination of everyone's utilities. So, in a very narrow (and unrealistic) setting, Harsanyi's theorem tells you how the single-multi solution is built from the si
Dylan: There’s one example that I think about, which is, say, you’re cooperating with an AI system playing chess. You start working with that AI system, and you discover that if you listen to its suggestions, 90% of the time, it’s actually suggesting the wrong move or a bad move. Would you call that system value-aligned?
Lucas: No, I would not.
Dylan: I think most people wouldn’t. Now, what if I told you that that program was act
We can imagine aliens building a superintelligent agent which helps them get what they want. This is a special case of aliens inventing tools. What kind of general process should these aliens use – how should they go about designing such an agent?
Assume that these aliens want things in the colloquial sense (not that they’re eg nontrivially VNM EU maximizers) and that a reasonable observer would say they’re closer to being rational than antirational. Then it seems[1] like these aliens eventually steer towards reflectively coherent rationality (provided they
In 2018, I started thinking about corrigibility as "being the kind of agent lots of agents would be happy to have activated". This seems really close to a more ambitious version of what AUP tries to do (not be catastrophic for most agents).
I wonder if you could build an agent that rewrites itself / makes an agent which would tailor the AU landscape towards its creators' interests, under a wide distribution of creator agent goals/rationalities/capabilities. And maybe you then get a kind of generalization, where most simple algorithms which solve this solve ambitious AI alignment in full generality.
The answer to this seems obvious in isolation: shaping helps with credit assignment, rescaling doesn't (and might complicate certain methods in the advantage vs Q-value way). But I feel like maybe there's an important interaction here that could inform a mathematical theory of how a reward signal guides learners through model space?
Reasoning about learned policies via formal theorems on the power-seeking incentives of optimal policies
One way instrumental subgoals might arise in actual learned policies: we train a proto-AGI reinforcement learning agent with a curriculum including a variety of small subtasks. The current theorems show sufficient conditions for power-seeking tending to be optimal in fully-observable environments; many environments meet these sufficient conditions; optimal policies aren't hard to compute for the subtasks. One highly transferable heuristic would therefore... (read more)
I prompted GPT-3 with modified versions of Eliezer's Beisutsukai stories, where I modified the "class project" to be about solving intent alignment instead of quantum gravity.
... Taji looked over his sheets. "Okay, I think we've got to assume that every avenue that Eld science was trying is a blind alley, or they would have found it. And if this is possible to do in one month, the answer must be, in some sense, elegant. So no human mistake models. If we start doing anything that looks like we should call it 'utility function patching', we'd better st
An AGI's early learned values will steer its future training and play a huge part in determining its eventual stable values. I think most of the ball game is in ensuring the agent has good values by the time it's smart, because that's when it'll start being reflectively stable. Therefore, we can iterate on important parts of alignment, because the most important parts come relatively early in the training run, and early corresponds to "parts of the AI value formation process which we can test before we hit AGI, without training one all the way out."
I think this, in theory, cuts away a substantial amount of the "But we only get one shot" problem. In practice, maybe OpenMind just YOLOs ahead anyways and we only get a few years in the appropriate and informative regime. But this suggests several kinds of experiments to start running now, like "get a Minecraft agent which robustly cares about chickens", because that tells us about how to map outer signals into inner values.
Which means that the destination where it's heading stops uncontrollably
changing, but nobody at that point (including the agent) has the slightest idea
what it looks like, and it won't get close for a long time. Also, the
destination (preference/goal/values) would generally depend on the environment
(it ends up being different if details of the world outside the AGI are
different). So many cartesian assumptions fail, distinguishing this situation
from a classical agent with goals, where goals are at least contained within the
agent, and probably also don't depend on its state of knowledge.
I think this is true for important alignment properties, including things that
act like values early on, but not for the values/preferences that are
reflectively stable in a strong sense. If it's possible to
inspect/understand/interpret the content of preference that is reflectively
stable, then what you've built is a mature optimizer with tractable goals, which
is always misaligned. It's a thing like paperclip maximizer, demonstrating
orthogonality thesis, even if it's tiling the future with something
superficially human-related.
That is, it makes sense to iterate on the parts of alignment that can be
inspected, but the reflectively stable values is not such a part, unless the AI
is catastrophically misaligned. The fact that reflectively stable values are the
same as those of humanity might be such a part, but it's this fact of sameness
that might admit inspection, not the values themselves.
2Alex Turner1y
I disagree with CEV as I recall it, but this could change after rereading it. I
would be surprised if I end up thinking that EY had "gotten it right." The
important thing to consider is not "what has someone speculated a good
destination-description would be", but "what are the actual mechanics look like
for getting there?". In this case, the part of you which likes dogs is helping
steer your future training and experiences, and so the simple answer is that
it's more likely than not that your stable values like dogs too.
This reasoning seems to prove too much. Your argument seems to imply that I
cannot have "the slightest idea" whether my stable values would include killing
people for no reason, or not.
0Vladimir Nesov1y
It does add up to normality, it's not proving things about current behavior or
current-goal content of near-future AGIs. An unknown normative target doesn't
say not to do the things you normally do, it's more of a "I beseech you, in the
bowels of Christ, to think it possible you may be mistaken" thing.
The salient catastrophic alignment failure here is to make AGIs with stable
values that capture some variation on current unstable human values, and won't
allow their further development. If the normative target is very far from
current unstable human values, making current values stable falls very short of
the normative target, makes future relatively worthless.
That's the kind of thing my point is intended to nontrivially claim, that AGIs
with any stable immediately-actionable goals that can be specified in the
following physical-time decades or even centuries are almost certainly
catastrophically misaligned. So AGIs must have unstable goals, softly
optimized-for, aligned to current (or value-laden predicted future) human
unstable goals, mindful of goodhart.
The kind of CEV I mean is not very specific, it's more of a (sketch of a
solution to the) problem of doing a first pass on preparing to define goals for
an actual optimizer, one that doesn't need to worry as much about goodhart and
so can make more efficient use of the future at scale, before expansion of the
universe makes more stuff unreachable.
So when I say "CEV" I mostly just mean "normative alignment target", with some
implied clarifications on what kind of thing it might be.
That's a very status quo anchored thing. I don't think dog-liking is a feature
of values stable under reflection if the environment is allowed to change
completely, even if in the current environment dogs are salient. Stable values
are about the whole world, with all its AGI-imagined femtotech-rewritten
possibilities. This world includes dogs in some tiny corner of it, but I don't
see how observations of current attitudes hold mu
Are there any alignment techniques which would benefit from the supervisor having a severed corpus callosum, or otherwise atypical neuroanatomy? Usually theoretical alignment discourse focuses on the supervisor's competence / intelligence. Perhaps there are other, more niche considerations.
I'd like to see research exploring the relevance of intragenomic conflict to AI alignment research. Intragenomic conflict constitutes an in-the-wild example of misalignment, where conflict arises "within an agent" even though the agent's genes have strong instrumental incentives to work together (they share the same body).
Suppose we're playing a game where I give you a reward function and you give me its optimal value function in the MDP. If you let me do this for |S| reward functions (one for each state in the environment), and you're able to provide the optimal value function for each, then you know enough to reconstruct the entire environment (up to isomorphism).
Roughly: being able to complete linearly many tasks in the state space means you ha... (read more)
Against CIRL as a special case of against quickly jumping into highly specific speculation while ignoring empirical embodiments-of-the-desired-properties.
Just because we write down English describing what we want the AI to do ("be helpful"), propose a formalism (CIRL), and show good toy results (POMDPs where the agent waits to act until updating on more observations), that doesn't mean that the formalism will lead to anything remotely relevant to the original English words we used to describe it. (It's easier to say "this logic enables nonmonotonic reasoning" and mess around with different logics and show how a logic solves toy examples, than it is to pin down probability theory with Cox's theorem)
And yes, this criticism applies extremely strongly to my own past work with attainable utility preservation and impact measures. (Unfortunately, I learned my lesson after, and not before, making certain mistakes.)
In the context of "how do we build AIs which help people?", asking "does CIRL solve corrigibility?" is hilariously unjustified. By what evidence have we located such a specific question? We have assumed there is an achievable "corrigibility"-like property; we ha... (read more)
Shard theory suggests that goals are more natural to specify/inculcate in their shard-forms (e.g. if around trash and a trash can, put the trash away), and not in their (presumably) final form of globally activated optimization of a coherent utility function which is the reflective equilibrium of inter-shard value-handshakes (e.g. a utility function over the agent's internal plan-ontology such that, when optimized directly, leads to trash getting put away, among other utility-level reflections of initial shards).
I could (and did) hope that I could specify a utility function which is safe to maximize because it penalizes power-seeking. I may as well have hoped to jump off of a building and float to the ground. On my model, that's just not how goals work in intelligent minds. If we've had anything at all beaten into our heads by our alignment thought experiments, it's that goals are hard to specify in their final form of utility functions.
I think it's time to think in a different specification language.
I regret each of the thousands of hours I spent on my power-seeking theorems, and sometimes fantasize about retracting one or both papers. I am pained every time someone cites "Optimal policies tend to seek power", and despair that it is included in the alignment 201 curriculum. I think this work makes readers actively worse at thinking about realistic trained systems.
I think a healthy alignment community would have rebuked me for that line of research, but sadly I only remember about two people objecting that "optimality" is a horrible way of understanding trained policies.
I think the basic idea of instrumental convergence is just really blindingly obvious, and I think it is very annoying that there are people who will cluck their tongues and stroke their beards and say "Hmm, instrumental convergence you say? I won't believe it unless it is in a very prestigious journal with academic affiliations at the top and Computer Modern font and an impressive-looking methods section."
I am happy that your papers exist to throw at such people.
Anyway, if optimal policies tend to seek power, then I desire to believe that optimal policies tend to seek power :) :) And if optimal policies aren't too relevant to the alignment problem, well neither are 99.99999% of papers, but it would be pretty silly to retract all of those :)
Since I'm an author on that paper, I wanted to clarify my position here. My perspective is basically the same as Steven's: there's a straightforward conceptual argument that goal-directedness leads to convergent instrumental subgoals, this is an important part of the AI risk argument, and the argument gains much more legitimacy and slightly more confidence in correctness by being formalized in a peer-reviewed paper.
I also think this has basically always been my attitude towards this paper. In particular, I don't think I ever thought of this paper as providing any evidence about whether realistic trained systems would be goal-directed.
Just to check that I wasn't falling prey to hindsight bias, I looked through our Slack history. Most of it is about the technical details of the results, so not very informative, but the few conversations on higher-level discussion I think overall support this picture. E.g. here are some quotes (only things I said):
Nov 3, 2019:
... (read more)It seems like just 4 months ago you still endorsed your second power-seeking paper:
Why are you now "fantasizing" about retracting it?
A lot of people might have thought something like, "optimality is not a great way of understanding trained policies, but maybe it can be a starting point that leads to more realistic ways of understanding them" and therefore didn't object for that reason. (Just guessing as I apparently wasn't personally paying attention to this line of research back then.)
Which seems to have turned out to be true, at least as of 4 months ago, when you still endorsed your second paper as "actually has a shot of being applicable to... (read more)
To be clear, I still endorse Parametrically retargetable decision-makers tend to seek power. Its content is both correct and relevant and nontrivial. The results, properly used, may enable nontrivial inferences about the properties of inner trained cognition. I don't really want to retract that paper. I usually just fantasize about retracting Optimal policies tend to seek power.
The problem is that I don't trust people to wield even the non-instantly-doomed results.
For example, one EAG presentation cited my retargetability results as showing that most reward functions "incentivize power-seeking actions." However, my results have not shown this for actual trained systems. (And I think that Power-seeking can be probable and predictive for trained agents does not make progress on the incentives of trained policies.)
People keep talking about stuff they know how to formalize (e.g. optimal policies) instead of stuff that matters (e.g. trained policies). I'm pained by this emphasis and I think my retargetability results are complicit. Relative to an actual competent alignment community (in a more competent world), we just have no damn clue how to properly reason about real trained policies... (read more)
Thanks for your patient and high-quality engagement here, Vika! I hope my original comment doesn't read as a passive-aggressive swipe at you. (I consciously tried to optimize it to not be that.) I wanted to give concrete examples so that Wei_Dai could understand what was generating my feelings.
It's a tough question to say how to apply the retargetablity result to draw practical conclusions about trained policies. Part of this is because I don't know if trained policies tend to autonomously seek power in various non game-playing regimes.
If I had to say something, I might say "If choosing the reward function lets us steer the training process to produce a policy which brings about outcome X, and most outcomes X can only be attained by seeking power, then most chosen reward functions will train power-seeking policies." This argument appropriately behaves differently if the "outcomes" are simply different sentiment generations being sampled from an LM -- sentiment shift doesn't require power-seeking.
... (read more)What is "shard theory"? I've written a lot about shard theory. I largely stand by these models and think they're good and useful. Unfortunately, lots of people seem to be confused about what shard theory is. Is it a "theory"? Is it a "frame"? Is it "a huge bag of alignment takes which almost no one wholly believes except, perhaps, Quintin Pope and Alex Turner"?
I think this understandable confusion happened because my writing didn't distinguish between:
(People might be less excited to use the "shard" abstraction (1), because they aren't sure whether they buy all this other stuff—(2) and (3).)
I think I can give an interesting and useful definition of (1) now, but I couldn't do so last year... (read more)
AI strategy consideration. We won't know which AI run will be The One. Therefore, the amount of care taken on the training run which produces the first AGI, will—on average—be less careful than intended.
No team is going to run a training run with more care than they would have used for the AGI Run, especially if they don't even think that the current run will produce AGI. So the average care taken on the real AGI Run will be strictly less than intended.
Teams which try to be more careful on each run will take longer to iterate on AI designs, thereby lowering the probability that they (the relatively careful team) will be the first to do an AGI Run.
Upshots:
- Th
... (read more)Positive values seem more robust and lasting than prohibitions. Imagine we train an AI on realistic situations where it can kill people, and penalize it when it does so. Suppose that we successfully instill a strong and widely activated "If going to kill people, then don't" value shard.
Even assuming this much, the situation seems fragile. See, many value shards are self-chaining. In The shard theory of human values, I wrote about how:
The juice shard chains into itself, reinforcing itself across time and thought-steps.
But a "don't kill" shard seems like it should remain... stubby? Primitive?... (read more)
Why do many people think RL will produce "agents", but maybe (self-)supervised learning ((S)SL) won't? Historically, the field of RL says that RL trains agents. That, of course, is no argument at all. Let's consider the technical differences between the training regimes.
In the modern era, both RL and (S)SL involve initializing one or more neural networks, and using the reward/loss function to provide cognitive updates to the network(s). Now we arrive at some differences.
Some of this isn't new (see Hidden Incentives for Auto-Induced Distributional Shift), but I think it's important and felt like writing up my own take on it. Maybe this becomes a post later.
[Exact gradients] RL's credit assignment problem is harder than (self-)supervised learning's. In RL, if an agent solves a maze in 10 steps, it gets (discounted) reward; this trajectory then provides a set of reward-modulated gradients to the agent. But if the agent could have solved the maze in 5 steps, the agent isn't directly updated to be more likely to do that in the future; RL's gradients are generally inexact, not pointing directly at intended behavior.
On the other hand, if a supervised-learning classifier outputs
dog... (read more)A problem with adversarial training. One heuristic I like to use is: "What would happen if I initialized a human-aligned model and then trained it with my training process?"
So, let's consider such a model, which cares about people (i.e. reliably pulls itself into futures where the people around it are kept safe). Suppose we also have some great adversarial training technique, such that we have e.g. a generative model which produces situations where the AI would break out of the lab without permission from its overseers. Then we run this procedure, update the AI by applying gradients calculated from penalties applied to its actions in that adversarially-generated context, and... profit?
But what actually happens with the aligned AI? Possibly something like:
- The context makes the AI spuriously believe someone is dying outside the lab, and that if the AI asked for permission to leave, the person would die.
- Therefore, the AI leaves without permission.
- The update procedure penalizes these lines of computation, such that in similar situations in the future (i.e. the AI thinks someone nearby is dying) the AI is less likely to take those actions (i.e. leaving to help the person).
- We have
... (read more)I think instrumental convergence also occurs in the model space for machine learning. For example, many different architectures likely learn edge detectors in order to minimize classification loss on MNIST. But wait - you'd also learn edge detectors to maximize classification loss on MNIST (loosely, getting 0% on a multiple-choice exam requires knowing all of the right answers). I bet you'd learn these features for a wide range of cost functions. I wonder if that's already been empirically investigated?
And, same for adversarial features. And perhaps, same for mesa optimizers (understanding how to stop mesa optimizers from being instrumentally convergent seems closely related to solving inner alignment).
What can we learn about this?
Back-of-the-envelope probability estimate of alignment-by-default via a certain shard-theoretic pathway. The following is what I said in a conversation discussing the plausibility of a proto-AGI picking up a "care about people" shard from the data, and retaining that value even through reflection. I was pushing back against a sentiment like "it's totally improbable, from our current uncertainty, for AIs to retain caring-about-people shards. This is only one story among billions."
Here's some of what I had to say:
[Let's reconsider the five-step mechanistic story I made up.] I'd give the following conditional probabilities (made up with about 5 seconds of thought each):
... (read more)Very nice people don’t usually search for maximally-nice outcomes — they don’t consider plans like “killing my really mean neighbor so as to increase average niceness over time.” I think there are a range of reasons for this plan not being generated. Here’s one.
Consider a person with a niceness-shard. This might look like an aggregation of subshards/subroutines like “if
person nearbyandperson.state==sad, sample plan generator for ways to make them happy” and “bid upwards on plans which lead to people being happier and more respectful, according to my world model.” In mental contexts where this shard is very influential, it would have a large influence on the planning process.However, people are not just made up of a grader and a plan-generator/actor — they are not just “the plan-generating part” and “the plan-grading part.” The next sampled plan modification, the next internal-monologue-thought to have—these are influenced and steered by e.g. the nice-shard. If the next macrostep of reasoning is about e.g. hurting people, well — the niceness shard is activated, and will bid down on this.
The niceness shard isn’t just bidding over outcomes, it’s bidding on next thoughts (on m... (read more)
An alternate mechanistic vision of how agents can be motivated to directly care about e.g. diamonds or working hard. In Don't design agents which exploit adversarial inputs, I wrote about two possible mind-designs:
I explained how evaluation-child is positively incentivized to dupe his model of his mom and thereby exploit adversarial inputs to her cognition. This shows that aligning an agent to evaluations of good behavior is not even close to aligning an agent to good behavior.
However, some commenters seemed maybe skeptical that value-child can exist, or uncertain how concretely that kind of mind works. I worry/suspect that many people have read shard theory posts without internalizing new ideas about how cognition can work, ... (read more)
AI cognition doesn't have to use alien concepts to be uninterpretable. We've never fully interpreted human cognition, either, and we know that our introspectively accessible reasoning uses human-understandable concepts.
Just because your thoughts are built using your own concepts, does not mean your concepts can describe how your thoughts are computed.
Or:
The existence of a natural-language description of a thought (like "I want ice cream") doesn't mean that your brain computed that thought in a way which can be compactly described by familiar concepts.
Conclusion: Even if an AI doesn't rely heavily on "alien" or unknown abstractions -- even if the AI mostly uses human-like abstractions and features -- the AI's thoughts might still be incomprehensible to us, even if we took a lot of time to understand them.
Examples should include actual details. I often ask people to give a concrete example, and they often don't. I wish this happened less. For example:
This is not a concrete example.
This is a concrete example.
"Globally activated consequentialist reasoning is convergent as agents get smarter" is dealt an evidential blow by von Neumann:
Experiment: Train an agent in MineRL which robustly cares about chickens (e.g. would zero-shot generalize to saving chickens in a pen from oncoming lava, by opening the pen and chasing them out, or stopping the lava). Challenge mode: use a reward signal which is a direct function of the agent's sensory input.
This is a direct predecessor to the "Get an agent to care about real-world dogs" problem. I think solving the Minecraft version of this problem will tell us something about how outer reward schedules relate to inner learned values, in a way which directly tackles the key questions, the sensory observability/information inaccessibility issue, and which is testable today.
(Credit to Patrick Finley for the idea)
Consider what update equations have to say about "training game" scenarios. In PPO, the optimization objective is proportional to the advantage given a policy π, reward function R, and on-policy value function vπ:
Aπ(s,a):=Es′∼T(s,a)[R(s,a,s′)+γvπ(s′)]−vπ(s).Consider a mesa-optimizer acting to optimize some mesa objective. The mesa-optimizer understands that it will be updated proportional to the advantage. If the mesa-optimizer maximizes reward, this corresponds to maximizing the intensity of the gradients it receives, thus maximally updating its cognition in exact directions.
This isn't necessarily good.
If you're trying to gradient hack and preserve the mesa-objective, you might not want to do this. This might lead to value drift, or make the network catastrophically forget some circuits which are useful to the mesa-optimizer.
Instead, the best way to gradient hack might be to roughly minimize the absolute value of the advantage, which means achieving roughly on-policy value over time, which doesn't imply reward maximization. This is a kind of "treading water" in terms of reward. This helps decrease value drift.
I think that realistic mesa optimizers will n... (read more)
Outer/inner alignment decomposes a hard problem into two extremely hard problems.
I have a long post draft about this, but I keep delaying putting it out in order to better elaborate the prereqs which I seem to keep getting stuck on when elaborating the ideas. I figure I might as well put this out for now, maybe it will make some difference for someone.
I think that the inner/outer alignment framing[1] seems appealing but is actually a doomed problem decomposition and an unhelpful frame for alignment.
- The reward function is a tool which chisels cognition into agents through gradient updates, but the outer/inner decomposition assumes that that tool should also embody the goals we want to chisel into the agent. When chiseling a statue, the chisel doesn’t have to also look like the finished statue.
- I know of zero success stories for outer alignment to real-world goals.
- More precisely, stories where people decided “I want an AI which [helps humans / makes diamonds / plays Tic-Tac-Toe / grows strawberries]”, and then wrote down an outer objective only maximized in those worlds.
- This is pretty weird on any model where most of the
... (read more)I was talking with Abram Demski today about a promising-seeming research direction. (Following is my own recollection)
One of my (TurnTrout's) reasons for alignment optimism is that I think:
- We can examine early-training cognition and behavior to some extent, since the system is presumably not yet superintelligent and planning against us,
- (Although this amount of information depends on how much interpretability and agent-internals theory we do now)
- All else equal, early-training values (decision-influences) are the most important to influence, since they steer future training.
- It's crucial to get early-training value shards of which a substantial fraction are "human-compatible values" (whatever that means)
- For example, if there are protect-human-shards which
- reliably bid against plans where people get hurt,
- steer deliberation away from such plan stubs, and
- these shards are "reflectively endorsed" by the overall shard economy (i.e. the decision-making isn't steering towards plans where the protect-human shards get removed)
- If we install influential human-compatible shards early in training, and they get retained, they will help us in mid- and late-training where we can't affect the ball
... (read more)Earlier today, I was preparing for an interview. I warmed up by replying stream-of-consciousness to imaginary questions I thought they might ask. Seemed worth putting here.
... (read more)I recently reached out to my two PhD advisors to discuss Hinton stepping down from Google. An excerpt from one of my emails:
... (read more)The "maximize all the variables" tendency in reasoning about AGI.
Here are some lines of thought I perceive, which are probably straw to varying extents for some people and real to varying extents for other people. I give varying responses to each, but the point isn't the truth value of any given statement, but of a pattern across the statements:
- If an AGI has a concept around diamonds, and is motivated in some way to make diamonds, it will make diamonds which maximally activate its diamond-concept circuitry (possible example).
- My response.
- An AI will be trained to minimal loss on the training distribution.
- SGD does not reliably find minimum-loss configurations (modulo expressivity), in practice, in cases we care about. The existence of knowledge distillation is one large counterexample.
- Quintin: "In terms of results about model distillation, you could look at appendix G.2 of the Gopher paper. They compare training a 1.4 billion parameter model directly, versus distilling a 1.4 B model from a 7.1 B model."
- Predictive processing means that the goal of the human learning process is to minimize predictive loss.[1]
- In a process where local modifications are applied to reduce some
... (read more)I think this type of criticism is applicable in an even wider range of fields than even you immediately imagine (though in varying degrees, and with greater or lesser obviousness or direct correspondence to the SGD case). Some examples:
-
-
-
-
-
-
... (read more)Despite the economists, the economy doesn't try to maximize welfare, or even net dollar-equivalent wealth. It rewards firms which are able to make a profit in proportion to how much they're able to make a profit, and dis-rewards firms which aren't able to make a profit. Firms which are technically profitable, but have no local profit incentive gradient pointing towards them (factoring in the existence of rich people and lenders, neither of which are perfect expected profit maximizers) generally will not happen.
Individual firms also don't (only) try to maximize profit. Some parts of them may maximize profit, but most are just structures of people built from local social capital and economic capital incentive gradients.
Politicians don't try to (only) maximize win-probability.
Democracies don't try to (only) maximize voter approval.
Evolution doesn't try to maximize inclusive genetic fitness.
Memes don't try to maximize inclusive memetic
If another person mentions an "outer objective/base objective" (in terms of e.g. a reward function) to which we should align an AI, that indicates to me that their view on alignment is very different. The type error is akin to the type error of saying "My physics professor should be an understanding of physical law." The function of a physics professor is to supply cognitive updates such that you end up understanding physical law. They are not, themselves, that understanding.
Similarly, "The reward function should be a human-aligned objective" -- The function of the reward function is to supply cognitive updates such that the agent ends up with human-aligned objectives. The reward function is not, itself, a human aligned objective.
I never thought I'd be seriously testing the reasoning abilities of an AI in 2020.
Looking back, history feels easy to predict; hindsight + the hard work of historians makes it (feel) easy to pinpoint the key portents. Given what we think about AI risk, in hindsight, might this have been the most disturbing development of 2020 thus far?
I personally lean towards "no", because this scaling seemed somewhat predictable from GPT-2 (flag - possible hindsight bias), and because 2020 has been so awful so far. But it seems possible, at least. I don't really know what update GPT-3 is to my AI risk estimates & timelines.
DL so far has been easy to predict - if you bought into a specific theory of connectionism & scaling espoused by Schmidhuber, Moravec, Sutskever, and a few others, as I point out in https://www.gwern.net/newsletter/2019/13#what-progress & https://www.gwern.net/newsletter/2020/05#gpt-3 . Even the dates are more or less correct! The really surprising thing is that that particular extreme fringe lunatic theory turned out to be correct. So the question is, was everyone else wrong for the right reasons (similar to the Greeks dismissing heliocentrism for excellent reasons yet still being wrong), or wrong for the wrong reasons, and why, and how can we prevent that from happening again and spending the next decade being surprised in potentially very bad ways?
I'm currently excited about a "macro-interpretability" paradigm. To quote Joseph Bloom:
... (read more)Speculation: RL rearranges and reweights latent model abilities, which SL created. (I think this mostly isn't novel, just pulling together a few important threads)
Suppose I supervised-train a LM on an English corpus, and I want it to speak Spanish. RL is inappropriate for the task, because its on-policy exploration won't output interestingly better or worse Spanish completions. So there's not obvious content for me to grade.
More generally, RL can provide inexact gradients away from undesired behavior (e.g. negative reinforcement event -> downweigh... (read more)
When I think about takeoffs, I notice that I'm less interested in GDP or how fast the AI's cognition improves, and more on how AI will affect me, and how quickly. More plainly, how fast will shit go crazy for me, and how does that change my ability to steer events?
For example, assume unipolarity. Let architecture Z be the architecture which happens to be used to train the AGI.
- How long is the duration between "architecture Z is published / seriously considered" and "the AGI kills me, assuming alignment fails"?
- How long do I have, in theory,
... (read more)Thomas Kwa suggested that consequentialist agents seem to have less superficial (observation, belief state) -> action mappings. EG a shard agent might have:
But a consequentialist would just reason about what happens, and not mess with those heuristics. (OFC, consequentialism would be a matter of degree)
In this way, changing a small set of decision-relevant features (e.g. "Brown dog treat" -> "brown ball of chocolate") changes the consequentialist's action logits a lot, way more than it changes the shard agent's logits. In a squinty, informal way, the (belief state -> logits) function has a higher Lipschitz constant/is more smooth for the shard agent than for the consequentialist agent.
So maybe one (pre-deception) test for consequentialist reasoning is to test sensitivity of decision-making to small perturbations in observation-space (e.g. dog treat -> tiny chocolate) but large perturbations in action-consequence space (e.g. happy dog -> sick dog). You could spin up two copies of the model to compare.
Partial alignment successes seem possible.
People care about lots of things, from family to sex to aesthetics. My values don't collapse down to any one of these.
I think AIs will learn lots of values by default. I don't think we need all of these values to be aligned with human values. I think this is quite important.
- I think the more of the AI's values we align to care about us and make decisions in the way we want, the better. (This is vague because I haven't yet sketched out AI internal motivations which I think would actually produce good outcomes. On my list!)
- I think there are strong gains from trade possible among an agent's values. If I care about bananas and apples, I don't need to split my resources between the two values, I don't need to make one successor agent for each value. I can drive to the store and buy both bananas and apples, and only pay for fuel once.
- This makes it lower-cost for internal values handshakes to compromise; it's less than 50% costly for a power-seeking value to give human-compatible values 50% weight in the reflective utility function.
- I think there are thresholds at which the AI doesn't care about us sufficiently strongly, and
... (read more)Three recent downward updates for me on alignment getting solved in time:
I'm not just solving technical problems—I'm also solving interpersonal problems, communication problems, incentive problems. Even if my current hot takes around shard theory / outer/inner alignment are right, and even if I put up a LW post which finally successfully communicates some of my key points, reality totally allows OpenAI to just train an AGI the next month without incorporating any insights which my friends nodded along with.
I often get the impression that people weigh off e.g. doing shard theory alignment strategies under the shard theory alignment picture, versus inner/outer research under the inner/outer alignment picture, versus...
And insofar as this impression is correct, this is a mistake. There is only one way alignment is.
If inner/outer is altogether a more faithful picture of those dynamics:
- relatively coherent singular mesa-objectives form in agents, albeit not necessarily always search-based
- more fragility of value and difficulty in getting the mesa object
... (read more)Idea for getting weak-in-expectation evidence about deception:
Are there convergently-ordered developmental milestones for AI? I suspect there may be convergent orderings in which AI capabilities emerge. For example, it seems that LMs develop syntax before semantics, but maybe there's an even more detailed ordering relative to a fixed dataset. And in embodied tasks with spatial navigation and recurrent memory, there may be an order in which enduring spatial awareness emerges (i.e. "object permanence").
In A shot at the diamond-alignment problem, I wrote:
... (read more)Quick summary of a major takeaway from Reward is not the optimization target:
Stop thinking about whether the reward is "representing what we want", or focusing overmuch on whether agents will "optimize the reward function." Instead, just consider how the reward and loss signals affect the AI via the gradient updates. How do the updates affect the AI's internal computations and decision-making?
I plan to mentor several people to work on shard theory and agent foundations this winter through SERI MATS. Apply here if you're interested in working with me and Quintin.
Research-guiding heuristic: "What would future-TurnTrout predictably want me to get done now?"
80% credence: It's very hard to train an inner agent which reflectively equilibrates to an EU maximizer only over commonly-postulated motivating quantities (like
# of diamondsor# of happy peopleorreward-signal) and not quantities like (# of times I have to look at a cube in a blue roomor-1 * subjective micromorts accrued).Intuitions:
- I expect contextually activated heuristics to be the default, and that agents will learn lots of such contextual values which don't cash out to being strictly about diamonds or people, even if the overall agent is mostly
... (read more)If you want to argue an alignment proposal "breaks after enough optimization pressure", you should give a concrete example in which the breaking happens (or at least internally check to make sure you can give one). I perceive people as saying "breaks under optimization pressure" in scenarios where it doesn't even make sense.
For example, if I get smarter, would I stop loving my family because I applied too much optimization pressure to my own values? I think not.
How might we align AGI without relying on interpretability?
I'm currently pessimistic about the prospect. But it seems worth thinking about, because wouldn't it be such an amazing work-around?
My first idea straddles the border between contrived and intriguing. Consider some AGI-capable ML architecture, and imagine its Rn parameter space being 3-colored as follows:
- Gray if the
- Red if the parameter vector+... leads to a misaligned or dece
... (read more)parameter vector+training process+other initial conditionsleads to a nothingburger (a non-functional model)Handling compute overhangs after a pause.
Sometimes people object that pausing AI progress for e.g. 10 years would lead to a "compute overhang": At the end of the 10 years, compute will be cheaper and larger than at present-day. Accordingly, once AI progress is unpaused, labs will cheaply train models which are far larger and smarter than before the pause. We will not have had time to adapt to models of intermediate size and intelligence. Some people believe this is good reason to not pause AI progress.
There seem to be a range of relatively simple pol... (read more)
Wikipedia has an unfortunate and incorrect-in-generality description of reinforcement learning (emphasis added)
Later in the article, talking about basic optimal-control inspired approaches:
... (read more)From the ELK report:
... (read more)The existence of the human genome yields at least two classes of evidence which I'm strongly interested in.
- Humans provide many highly correlated datapoints on general intelligence (human minds), as developed within one kind of learning process (best guess: massively parallel circuitry, locally randomly initialized, self-supervised learning + RL).
- We thereby gain valuable information about the dynamics of that learning process. For example, people care about lots of things (cars, flowers, animals, friends), and don't just have a single unitary mesa-obj
... (read more)Why don't people reinforcement-learn to delude themselves? It would be very rewarding for me to believe that alignment is solved, everyone loves me, I've won at life as hard as possible. I think I do reinforcement learning over my own thought processes. So why don't I delude myself?
On my model of people, rewards provide ~"policy gradients" which update everything, but most importantly shards. I think eg the world model will have a ton more data from self-supervised learning, and so on net most of its bits won't come from reward gradients.
For example, if I ... (read more)
Basilisks are a great example of plans which are "trying" to get your plan evaluation procedure to clock in a huge upwards error. Sensible beings avoid considering such plans, and everything's fine. I am somewhat worried about an early-training AI learning about basilisks before the AI is reflectively wise enough to reject the basilisks.
For example:
- Pretraining on a corpus in which people worry about basilisks could elevate reasoning about basilisks to the AI's consideration,
- at which point the AI reasons in more detail because it's not... (read more)
Argument that you can't use a boundedly intelligent ELK solution to search over plans to find one which keeps the diamond in the vault. That is, the ELK solution probably would have to be at least as smart (or smarter) than the plan-generator.
Consider any situation where it's hard to keep the diamond in the vault. Then any successful plan will have relatively few degrees of freedom. Like, a bunch of really smart thieves will execute a cunning plot to extract the diamond. You can't just sit by or deploy some simple traps in this situation.
Therefore, any pla... (read more)
"Goodhart" is no longer part of my native ontology for considering alignment failures. When I hear "The AI goodharts on some proxy of human happiness", I start trying to fill in a concrete example mind design which fits that description and which is plausibly trainable. My mental events are something like:
Condition on: AI with primary value shards oriented around spurious correlate of human happiness; AI exhibited deceptive alignment during training, breaking perceived behavioral invariants during its sharp-capabilities-gainWarning: No history... (read more)Offline RL can work well even with wrong reward labels. I think alignment discourse over-focuses on "reward specification." I think reward specification is important, but far from the full story.
To this end, a new paper (Survival Instinct in Offline Reinforcement Learning) supports Reward is not the optimization target and associated points that reward is a chisel which shapes circuits inside of the network, and that one should fully consider the range of sources of parameter updates (not just those provided by a reward signal).
Some relevant qu... (read more)
Consider trying to use Solomonoff induction to reason about P(I see “Canada goes to war with USA" in next year), emphasis added:
... (read more)Team shard is now accepting applications for summer MATS. SERI MATS is now accepting applications for their 4.0 program this summer. In particular, consider applying to the shard theory stream, especially if you have the following interests:
Feel free to apply if you're interested in shard theory more generally, although I expect to mostly supervise empirical work. Feel free to message me if you have questi... (read more)
The policy of truth is a blog post about why policy gradient/REINFORCE suck. I'm leaving a shortform comment because it seems like a classic example of wrong RL theory and philosophy, since reward is not the optimization target. Quotes:
... (read more)Shard-theoretic model of wandering thoughts: Why trained agents won't just do nothing in an empty room. If human values are contextually activated subroutines etched into us by reward events (e.g. "If candy nearby and hungry, then upweight actions which go to candy"), then what happens in "blank" contexts? Why don't people just sit in empty rooms and do nothing?
Consider that, for an agent with lots of value shards (e.g. candy, family, thrill-seeking, music), the "doing nothing" context is a very unstable equilibrium. I think these shards will activate on t... (read more)
Transplanting algorithms into randomly initialized networks. I wonder if you could train a policy network to walk upright in sim, back out the "walk upright" algorithm, randomly initialize a new network which can call that algorithm as a "subroutine call" (but the walk-upright weights are frozen), and then have the new second model learn to call that subroutine appropriately? Possibly the learned representations would be convergently similar enough to interface quickly via SGD update dynamics.
If so, this provides some (small, IMO) amount of rescue fo... (read more)
How the power-seeking theorems relate to the selection theorem agenda.
- Power-seeking theorems. P(agent behavior | agent decision-making procedure, agent objective, other agent internals, environment).
- The power-seeking theorems also allow some discussion of P(agent behavior | agent training process, trai
... (read more)I've mostly studied the likelihood function for power-seeking behavior: what decision-making procedures, objectives, and environments produce what behavioral tendencies. I've discovered some gears for what situations cause what kinds of behaviors.
Argument sketch for why boxing is doomed if the agent is perfectly misaligned:
Consider a perfectly misaligned agent which has -1 times your utility function—it's zero-sum. Then suppose you got useful output of the agent. This means you're able to increase your EU. This means the AI decreased its EU by saying anything. Therefore, it should have shut up instead. But since we assume it's smarter than you, it realized this possibility, and so the fact that it's saying something means that it expects to gain by hurting your interests via its output. Therefore, the output can't be useful.
My power-seeking theorems seem a bit like Vingean reflection. In Vingean reflection, you reason about an agent which is significantly smarter than you: if I'm playing chess against an opponent who plays the optimal policy for the chess objective function, then I predict that I'll lose the game. I predict that I'll lose, even though I can't predict my opponent's (optimal) moves - otherwise I'd probably be that good myself.
My power-seeking theorems show that most objectives have optimal policies which e.g. avoid shutdown and survive into the far future, even... (read more)
An additional consideration for early work on interpretability: it slightly increases the chance we actually get an early warning shot. If a system misbehaves, we can inspect its cognition and (hopefully) find hints of intentional deception. Could motivate thousands of additional researcher-hours being put into alignment.
ARCHES distinguishes between single-agent / single-user and single-agent/multi-user alignment scenarios. Given assumptions like "everyone in society is VNM-rational" and "societal preferences should also follow VNM rationality", and "if everyone wants a thing, society also wants the thing", Harsanyi's utilitarian theorem shows that the societal utility function is a linear non-negative weighted combination of everyone's utilities. So, in a very narrow (and unrealistic) setting, Harsanyi's theorem tells you how the single-multi solution is built from the si
... (read more)From FLI's AI Alignment Podcast: Inverse Reinforcement Learning and Inferring Human Preferences with Dylan Hadfield-Menell:
... (read more)We can imagine aliens building a superintelligent agent which helps them get what they want. This is a special case of aliens inventing tools. What kind of general process should these aliens use – how should they go about designing such an agent?
Assume that these aliens want things in the colloquial sense (not that they’re eg nontrivially VNM EU maximizers) and that a reasonable observer would say they’re closer to being rational than antirational. Then it seems[1] like these aliens eventually steer towards reflectively coherent rationality (provided they
... (read more)Very rough idea
In 2018, I started thinking about corrigibility as "being the kind of agent lots of agents would be happy to have activated". This seems really close to a more ambitious version of what AUP tries to do (not be catastrophic for most agents).
I wonder if you could build an agent that rewrites itself / makes an agent which would tailor the AU landscape towards its creators' interests, under a wide distribution of creator agent goals/rationalities/capabilities. And maybe you then get a kind of generalization, where most simple algorithms which solve this solve ambitious AI alignment in full generality.
The answer to this seems obvious in isolation: shaping helps with credit assignment, rescaling doesn't (and might complicate certain methods in the advantage vs Q-value way). But I feel like maybe there's an important interaction here that could inform a mathematical theory of how a reward signal guides learners through model space?
Reasoning about learned policies via formal theorems on the power-seeking incentives of optimal policies
One way instrumental subgoals might arise in actual learned policies: we train a proto-AGI reinforcement learning agent with a curriculum including a variety of small subtasks. The current theorems show sufficient conditions for power-seeking tending to be optimal in fully-observable environments; many environments meet these sufficient conditions; optimal policies aren't hard to compute for the subtasks. One highly transferable heuristic would therefore... (read more)
I prompted GPT-3 with modified versions of Eliezer's Beisutsukai stories, where I modified the "class project" to be about solving intent alignment instead of quantum gravity.
... (read more)Transparency Q: how hard would it be to ensure a neural network doesn't learn any explicit NANDs?
An AGI's early learned values will steer its future training and play a huge part in determining its eventual stable values. I think most of the ball game is in ensuring the agent has good values by the time it's smart, because that's when it'll start being reflectively stable. Therefore, we can iterate on important parts of alignment, because the most important parts come relatively early in the training run, and early corresponds to "parts of the AI value formation process which we can test before we hit AGI, without training one all the way out."
I think this, in theory, cuts away a substantial amount of the "But we only get one shot" problem. In practice, maybe OpenMind just YOLOs ahead anyways and we only get a few years in the appropriate and informative regime. But this suggests several kinds of experiments to start running now, like "get a Minecraft agent which robustly cares about chickens", because that tells us about how to map outer signals into inner values.
Are there any alignment techniques which would benefit from the supervisor having a severed corpus callosum, or otherwise atypical neuroanatomy? Usually theoretical alignment discourse focuses on the supervisor's competence / intelligence. Perhaps there are other, more niche considerations.
I'd like to see research exploring the relevance of intragenomic conflict to AI alignment research. Intragenomic conflict constitutes an in-the-wild example of misalignment, where conflict arises "within an agent" even though the agent's genes have strong instrumental incentives to work together (they share the same body).
In an interesting parallel to John Wentworth's Fixing the Good Regulator Theorem, I have an MDP result that says:
Roughly: being able to complete linearly many tasks in the state space means you ha... (read more)