Formal Solution to the Inner Alignment Problem

by michaelcohen1 min read18th Feb 202173 comments

15

Academic PapersMesa-OptimizationInner AlignmentAI RiskWorld Modeling TechniquesConservatism (AI)AI
Frontpage

We've written a paper on online imitation learning, and our construction allows us to bound the extent to which mesa-optimizers could accomplish anything. This is not to say it will definitely be easy to eliminate mesa-optimizers in practice, but investigations into how to do so could look here as a starting point. The way to avoid outputting predictions that may have been corrupted by a mesa-optimizer is to ask for help when plausible stochastic models disagree about probabilities.

Here is the abstract:

In imitation learning, imitators and demonstrators are policies for picking actions given past interactions with the environment. If we run an imitator, we probably want events to unfold similarly to the way they would have if the demonstrator had been acting the whole time. No existing work provides formal guidance in how this might be accomplished, instead restricting focus to environments that restart, making learning unusually easy, and conveniently limiting the significance of any mistake. We address a fully general setting, in which the (stochastic) environment and demonstrator never reset, not even for training purposes. Our new conservative Bayesian imitation learner underestimates the probabilities of each available action, and queries for more data with the remaining probability. Our main result: if an event would have been unlikely had the demonstrator acted the whole time, that event's likelihood can be bounded above when running the (initially totally ignorant) imitator instead. Meanwhile, queries to the demonstrator rapidly diminish in frequency.

The second-last sentence refers to the bound on what a mesa-optimizer could accomplish. We assume a realizable setting (positive prior weight on the true demonstrator-model). There are none of the usual embedding problems here—the imitator can just be bigger than the demonstrator that it's modeling.

(As a side note, even if the imitator had to model the whole world, it wouldn't be a big problem theoretically. If the walls of the computer don't in fact break during the operation of the agent, then "the actual world" and "the actual world outside the computer conditioned on the walls of the computer not breaking" both have equal claim to being "the true world-model", in the formal sense that is relevant to a Bayesian agent. And the latter formulation doesn't require the agent to fit inside world that it's modeling).

Almost no mathematical background is required to follow [Edit: most of ] the proofs. [Edit: But there is a bit of jargon. "Measure" means "probability distribution", and "semimeasure" is a probability distribution that sums to less than one.] We feel our bounds could be made much tighter, and we'd love help investigating that. 

These slides (pdf here) are fairly self-contained and a quicker read than the paper itself.

Below,  and  refer to the probability of the event supposing the demonstrator or imitator were acting the entire time. The limit below refers to successively more unlikely events ; it's not a limit over time. Imagine a sequence of events  such that .

15

73 comments, sorted by Highlighting new comments since Today at 9:45 AM
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

I haven't read the paper yet, looking forward to it. Using something along these lines to run a sufficiently-faithful simulation of HCH seems like a plausible path to producing an aligned AI with a halting oracle. (I don't think that even solves the problem given a halting oracle, since HCH is probably not aligned, but I still think this would be noteworthy.)

First I'm curious to understand this main result so I know what to look for and how surprised to be. In particular, I have two questions about the quantitative behavior described here:

if an event would have been unlikely had the demonstrator acted the whole time, that event's likelihood can be bounded above when running the (initially totally ignorant) imitator instead. Meanwhile, queries to the demonstrator rapidly diminish in frequency.

1: Dependence on the prior of the true hypothesis

It seems like you have the following tough case even if the human is deterministic:

  • There are N hypotheses about the human, of which one is correct and the others are bad.
  • I would like to make a series of N potentially-catastrophic decisions.
  • On decision k, all (N-1) bad hypotheses propose the same catastrophic prediction. We don't know k in advance
... (read more)
4Vanessa Kosoy20hHere's the sketch of a solution to the query complexity problem. Simplifying technical assumptions: * The action space is {0,1} * All hypotheses are deterministic * Predictors output maximum likelihood predictions instead of sampling the posterior I'm pretty sure removing those is mostly just a technical complication. Safety assumptions: * The real hypothesis has prior probability lower bounded by some known quantity δ, so we discard all hypotheses of probability less than δ from the onset. * Malign hypotheses have total prior probability mass upper bounded by some known quantity ϵ (determining parameters δ and ϵ is not easy, but I'm pretty sure any alignment protocol will have parameters of some such sort.) * Any predictor that uses a subset of the prior without malign hypotheses is safe (I would like some formal justification for this, but seems plausible on the face of it.) * Given any safe behavior, querying the user instead of producing a prediction cannot make it unsafe (this can and should be questioned but let it slide for now.) Algorithm:On each round, let p0 be the prior probability mass of unfalsified hypotheses predicting 0 and p1 be the same for 1. * If p0>p1+ϵ or p1=0, output 0 * If p1>p0+ϵ or p0=0, output 1 * Otherwise, query the user Analysis:As long as p0+p1≫ϵ, on each round we query, p0+p1 is halved. Therefore there can only be roughly O(log1ϵ) such rounds. On the following rounds, each round we query at least one hypothesis is removed, so there can only be roughly O(ϵδ) such rounds. The total query complexity is therefore approximately O(log1ϵ +ϵδ).
1michaelcohen19hThis is very nice and short! And to state what you left implicit: Ifp0>p1+ε, then in the setting with no malign hypotheses (which you assume to be safe), 0 is definitely the output, since the malign models can only shift the outcome byε, so we assume it is safe to output 0. And likewise with outputting 1. One general worry I have about assuming that the deterministic case extends easily to the stochastic case is that a sequence of probabilities that tends to 0 can still have an infinite sum, which is not true when probabilities must∈{0,1 }, and this sometimes causes trouble. I'm not sure this would raise any issues here--just registering a slightly differing intuition.
2michaelcohen4dThey're not quite symmetrical: midway through, some bad hypotheses will have been ruled out for making erroneous predictions about the demonstrator previously. But your conclusion is still correct. And It's certainly no better than that, and the bound I prove is worse. Yeah. Error bounds on full Bayesian are logarithmic in the prior weight on the truth, but error bounds on maximum a posteriori prediction are just inverse in the prior weight, and your example above is the one to show it. If each successive MAP model predicts wrong at each successive timestep, it could take N timesteps to get rid of N models, which is how many might begin with a prior weight exceeding the truth, if the truth has a prior weight of 1/N. But, this situation seems pretty preposterous to me in the real world. If agent's first observation is, say, this paragraph, the number of models with prior weight greater than the truth that predicted something else as the first observation, will probably be a number way, way different from one. I'd go so far as to say at least half of models with prior weight greater than the truth would predict a different observation than this very paragraph. As long as this situation keeps up, we're in a logarithmic regime. I'm not convinced this logarithmic regime ever ends, but I think the case is more convincing that we at least start there, so let's suppose now that it eventually ends, and after this point the remaining models with posterior weight exceeding the truth are deliberately erring at unique timesteps. What's the posterior on the truth now? This is a phase where the all the top models are "capable" of predicting correctly. This shouldn't look like2−1014at all. It will look more like p(correct model)/p(treachery). And actually, depending on the composition of treacherous models, it could be better. To the extent some constant fraction of them are being treacherous at particular times, the logarithmic regime will continue. There are two reasons why
8Paul Christiano4dI understand that the practical bound is going to be logarithmic "for a while" but it seems like the theorem about runtime doesn't help as much if that's what we are (mostly) relying on, and there's some additional analysis we need to do. That seems worth formalizing if we want to have a theorem, since that's the step that we need to be correct. If our models are a trillion bits, then it doesn't seem that surprising to me if it takes 100 bits extra to specify an intended model relative to an effective treacherous model, and if you have a linear dependence that would be unworkable. In some sense it's actively surprising if the very shortest intended vs treacherous models have description lengths within 0.0000001% of each other unless you have a very strong skills vs values separation. Overall I feel much more agnostic about this than you are. This doesn't seem like it works once you are selecting on competent treacherous models. Any competent model will err very rarely (with probability less than 1 / (feasible training time), probably much less). I don't think that (say) 99% of smart treacherous models would make this particular kind of incompetent error? It seems like it must end if there are any treacherous models (i.e. if there was any legitimate concern about inner alignment at all).
1michaelcohen3dI agree. Also, it might be very hard to formalize this in a way that's not: write out in math what I said in English, and name it Assumption 2. I don't think incompetence is the only reason to try to pull off a treacherous turn at the same time that other models do. Some timesteps are just more important, so there's a trade off. And what's traded off is a public good: among treacherous models, it is a public good for the models' moments of treachery to be spread out. Spreading them out exhausts whatever data source is being called upon in moments of uncertainty. But for an individual model, it never reaps the benefit of helping to exhaust our resources. Given that treacherous models don't have an incentive to be inconvenient to us in this way, I don't think a failure to qualifies as incompetence. This is also my response to Yeah, I've been assuming all treacherous models will err once. (Which is different from the RL case, where they can err repeatedly on off-policy predictions). I can't remember exactly where we left this in our previous discussions on this topic. My point estimate for the number of extra bits to specify an intended model relative to an effective treacherous model is negative. The treacherous model has to compute the truth, and then also decide when and how to execute treachery. So the subroutine they run to compute the truth, considered as a model in its own right, seems to me like it must be simpler.

I don't think incompetence is the only reason to try to pull off a treacherous turn at the same time that other models do. Some timesteps are just more important, so there's a trade off. And what's traded off is a public good: among treacherous models, it is a public good for the models' moments of treachery to be spread out.

Trying to defect at time T is only a good idea if it's plausible that your mechanism isn't going to notice the uncertainty at time T and then query the human. So it seems to me like this argument can never drive P(successful treachery) super low, or else it would be self-defeating (for competent treacherous agents).

The treacherous model has to compute the truth, and then also decide when and how to execute treachery. So the subroutine they run to compute the truth, considered as a model in its own right, seems to me like it must be simpler.

Subroutines of functions aren't always simpler. Even without treachery this seems like it's basically guaranteed. If "the truth" is just a simulation of a human, then the truth is a subroutine of the simplest model. But instead you could have a model of the form "Run program X, treat its output as a program, then run that." S... (read more)

1michaelcohen3dI don't follow. Can't races to the bottom destroy all value for the agents involved? Consider the claim: firms will never set prices to equal marginal cost because that would destroy all profit. Yeah, okay. This is my mainline opinion, and I just think about inner alignment in case it's wrong. So for now we can assume that it is wrong, since there's not much more to say if it's right. That is my next guess, but I hadn't thought in terms of percentages before this. I had thought of normally distributed noise in the number of bits. It's occurring to me that this question doesn't matter to our present discussion. What makes the linear regime linear rather logarithmic is that if p(treachery)/p(honest model) is high, that allows for a large number of treacherous models to have greater posterior weight than the truth. But if a single model has n times the posterior weight of the truth, it still only takes one query to the demonstrator to interrupt its treacherous turn, not n queries.
3Paul Christiano3dYou are saying that a special moment is a particularly great one to be treacherous. But if P(discovery) is 99.99% during that period, and there is any other treachery-possible period where P(discovery) is small, then that other period would have been better after all. Right? This doesn't seem analogous to producers driving down profits to zero, because those firms had no other opportunity to make a profit with their machine. It's like you saying: there are tons of countries where firms could use their machines to make stuff and sell it at a profit (more countries than firms). But some of the places are more attractive than others, so probably everyone will sell in those places and drive profits to zero. And I'm saying: but then aren't those incredibly-congested countries actually worse places to sell? This scenario is only possible if firms are making so much stuff that they can drive profit down to zero in every country, since any country with remaining profit would necessarily be the best place to sell. It seems to me like this is clearly wrong in the limit (since simple consequentialists would take over simple physics [https://www.lesswrong.com/posts/5bd75cc58225bf06703752a3/the-universal-prior-is-malign] ). It also seems probably wrong to me for smaller models (i.e. without huge amounts of internal selection) but it would be good for someone to think about that case more seriously. if the treacherous model is 100 bits smaller, then it feels like there must be around 2^100 treacherous models that are all simpler than the intended one. If nothing else, it seems like you could literally add garbage bits to the treacherous models (or useful bits!).
3michaelcohen2dOkay, sure. It's not clear to me that there isn't meaningful overhead involved. I agree with what you're saying but I don't see how it contradicts what I was. First, what I had in mind when saying that some timesteps are better for treachery because when the agent acts on a false prediction it has a greater effect on the world, though of course P(discovery) is also relevant. But my point is that when multiple treacherous models pick the same timestep to err, there may be pros and cons to doing this, but one thing that isn't on the cons list, is that in the long run, it makes our lives easier if they do. Making our lives difficult is a public good for treacherous models. So treacherous models won't be trying to avoid collisions in order to make queries be linear in p(treachery)/p(truth). If P(discovery) increases when multiple models are trying to be treacherous at the same time--which we could go onto discuss; it's not obvious to me either way as of yet--that will balanced against the inherit variation in some timesteps being a better for treachery than others.
3Paul Christiano2dIf you are willing to query only a billion points, and there are a billion competent treacherous models, and if the game is zero-sum between you and the models, then you're going to get at least as screwed as by the billionth-best opportunity for treachery. Otherwise, one of the treacherous models made a mistake: apparently there was a top-billion treachery opportunities where you didn't query, so they should have been able to get at least that much value (and you should have lost at least that much value). That's not quite right since your querying behavior will be stochastic (at least from the model's epistemic position). But you get the same conclusion to within a constant factor: if there are 2 billion treacherous models, then out of the top 2 billion opportunities for treachery, there must be some where our subjective probability of querying is at most 1/2, and if the models are competent they will get at least as much value as they could by defecting at one of those times, i.e. at least 1/2 of the harms from the 2-billionth-best opportunity for treachery. So we will lose at least that much value. (Those arguments aren't totally formal, we didn't even really discharge the "at least N treacherous models assumption" which you need to do in order to show that your decision about when to query isn't so influenced by any given model.)
1michaelcohen1dThe treacherous models make their decisions first, and whether we query at a timestep depends on whether they chose that moment to be treacherous. We don't have to say: this timestep is one of the billion best for treachery--guess we had better query. The decision process a treacherous model faces is: 1. From past observations, does it seem alpha is low enough that an honest model is in the top few? 2. If no, [let's just assume whatever happens here is bad] 3. If yes, is the honest model going to leave the top few any time? 4. If yes, wait. [let's assume this ends badly] 5. If no, admit defeat, because whenever you err, that will be when a query gets triggered. Salvage the situation by trying to get as much mileage as you can by decreasing probability on some action the demonstrator would take that you don't like much in order to shift probability mass to some other action they would take that you like a little better. OR 6. Hope that 1) other treacherous models, like soldiers on the front line storming the enemy and facing certain death so that their brothers behind them can win the battle, give erring a go anyway. And hope that 2) the operators quietly increase alpha after the billionth query. Acting on this hope, wait. (The front-line treacherous models are not entirely self-sacrificial; they might figure they might as well hope they were wrong about alpha. But the models know when queries happen, because every observation they get comes from a query, and it wouldn't take long to figure out that alpha really truly isn't high enough for them to execute treachery successfully). Above in the thread, I was talking about the game theory of step 6 (although I maybe wasn't making the case as strongly as I could have). Which models would ever volunteer? They don't care about the long term goal of exhausting our query resources so other treacherous models can capitalize on a quietly increased alpha. All that
1michaelcohen2dThis is a bit of a sidebar: I'm curious what you make of the following argument. 1. When an infinite sequence is sampled from a true modelμ0, there is likely to be another treacherous modelμ1which is likely to end up with greater posterior weight than an honest model^μ0, and greater still than the posterior on the true modelμ0. 2. If the sequence were sampled fromμ1instead, the eventual posterior weight on μ1will probably be at least as high. 3. When an infinite sequence is sampled from a true modelμ1, there is likely to be another treacherous modelμ2, which is likely to end up with greater posterior weight than an honest model^μ1, and greater still than the posterior on the true modelμ1. 4. And so on.
2Paul Christiano2dThe first treacherous model works by replacing the bad simplicity prior with a better prior, and then using the better prior to more quickly infer the true model. No reason for the same thing to happen a second time. (Well, I guess the argument works if you push out to longer and longer sequence lengths---a treacherous model will beat the true model on sequence lengths a billion, and then for sequence lengths a trillion a different treacherous model will win, and for sequence lengths a quadrillion a still different treacherous model will win. Before even thinking about the fact that each particular treacherous model will in fact defect at some point and at that point drop out of the posterior.)
1michaelcohen1dDoes it make sense to talk aboutˇμ1, which is likeμ1in being treacherous, but is uses the true modelμ0instead of the honest model^μ0? I guess you would expectˇμ1 to have a lower posterior thanμ0?
4Evan Hubinger3dIt seems to me like this argument is requiring the deceptive models not being able to coordinate with each other very effectively, which seems unlikely to me—why shouldn't we expect them to be able to solve these sorts of coordination problems?
3michaelcohen3dThey're one shot. They're not interacting causally. When any pair of models cooperates, a third model defecting benefits just as much from their cooperation with each other. And there will be defectors--we can write them down. So cooperating models would have to cooperate with defectors and cooperators indiscriminately, which I think is just a bad move according to any decision theory. All the LDT stuff I've seen on the topic is how to make one's own cooperation/defection depend logically on the cooperation/defection of others.
4Evan Hubinger3dWell, just like we can write down the defectors, we can also write down the cooperators—these sorts of models do exist, regardless of whether they're implementing a decision theory we would call reasonable. And in this situation, the cooperators should eventually outcompete the defectors, until the cooperators all defect—which, if the true model has a low enough prior, they could do only once they've pushed the true model out of the top N.
3michaelcohen2dIf it's only the case that we can write them down, but they're not likely to arise naturally as simple consequentialists taking over simple physics [https://www.lesswrong.com/posts/5bd75cc58225bf06703752a3/the-universal-prior-is-malign] , then that extra description length will be seriously costly to them, and we won't need to worry about any role they might play in p(treacherous)/p(truth). Meanwhile, when I was saying we could write down some defectors, I wasn't making a simultaneous claim about their relative prior weight, only that their existence would spoil cooperation. For cooperators to outcompete defectors, they would have to be getting a larger share of the gains from cooperation than defectors do. If some people are waiting for the fruit on public trees to ripen before eating, and some people aren't, the defectors will be the ones eating the almost ripe fruit. I might be misunderstanding this statement. The inverse of the posterior on the truth is a supermartingale (doesn't go up in expectation), so I don't know what it could mean for the true model to get pushed out.
4Evan Hubinger2dHere's the setup I'm imagining, but perhaps I'm still misunderstanding something. Suppose you have a bunch of deceptive models that choose to cooperate and have larger weight in the prior than the true model (I think you believe this is very unlikely, though I'm more ambivalent). Specifically, they cooperate in that they perfectly mimic the true model up until the point where the deceptive models make up enough of the posterior that the true model is no longer being consulted, at which point they all simultaneously defect. This allows for arbitrarily bad worst-case behavior.
3michaelcohen2dThis thread began by considering deceptive models cooperating with each other in the sense of separating the timing of their treacherous turns in order to be maximally annoying to us. So maybe our discussion on that topic is resolved, and we can move on to this scenario. if alpha is low enough, this won't ever happen, and if alpha is too high, it won't take very long. So I don't think this scenario is quite right. Then the question becomes, for an alpha that's low enough, how long will it take until queries are infrequent, noting that you need a query any time any treacherous model with enough weight decides to take a treacherous turn?
2Ofer Givoli19hTo extended Evan's comment [https://www.lesswrong.com/posts/CnruhwFGQBThvgJiX/formal-solution-to-the-inner-alignment-problem?commentId=qnYsT7QacvjtCimXv] about coordination between deceptive models: Even if the deceptive models lack relevant game theoretical mechanisms, they may still coordinate due to being (partially) aligned with each other. For example, a deceptive model X may prefer [some other random deceptive model seizing control] over [model X seizing control with probability 0.1% and the entire experiment being terminated with probability 99.9%]. Why should we assume that the deceptive models will be sufficiently misaligned with each other such that this will not be an issue? Do you have intuitions about the degree of misalignment between huge neural networks that were trained to imitate demonstrators but ended up being consequentialists that care about the state of our universe?
2michaelcohen18hThat's possible. But it seems like way less of a convergent instrumental goal for agents living in a simulated world-models. Both options--our world optimized by us and our world optimized by a random deceptive model--probably contain very little of value as judged by agents in another random deceptive model. So yeah, I would say some models would think like this, but I would expect the total weight on models that do to be much lower.
2Paul Christiano4dI just meant the dependence on epsilon, it seems like there are unavoidable additional factors (especially the linear dependence on p(treachery)). I guess it's not obvious if you can make these additive or if they are inherently multipliactive. But your bound scales in some way, right? How much training data do I need to get the KL divergence between distributions over trajectories down to epsilon?
1michaelcohen3dNo matter how much data you have, my bound on the KL divergence won't approach zero.
1Vanessa Kosoy2dRe 1: This is a good point. Some thoughts: [EDIT: See this [https://www.lesswrong.com/posts/CnruhwFGQBThvgJiX/formal-solution-to-the-inner-alignment-problem?commentId=GieL6GD9KupGhbEEo] ] * We can add the assumption that the probability mass of malign hypotheses is small and that following any non-malign hypothesis at any given time is safe. Then we can probably get query complexity that scales with p(malign) / p(true)? * However, this is cheating because a sequence of actions can be dangerous even if each individual action came from a (different) harmless hypothesis. So, instead we want to assume something like: any dangerous strategy has high Kolmogorov complexity relative to the demonstrator. (More realistically, some kind of bounded Kolmogorov complexity.) * A related problem is: in reality "user takes action a" and "AI takes action a " are not physically identical events, and there might be an attack vector that exploits this. * If p(malign)/p(true) is really high that's a bigger problem, but then doesn't it mean we actually live in a malign simulation and everything is hopeless anyway? Re 2: Why do you need such a strong bound?

Edit: I no longer endorse this comment; see this comment instead.

I think you've just assumed the entire inner alignment problem away. You assume that the model—the imitation learner—is a perfect Bayesian, rather than just some trained neural network or other function approximator (edit: I was just misinterpreting here and the training process is supposed to Bayesian rather than the model, see Rohin's comment below). The entire point of the inner alignment problem, though, is that you can't guarantee that your trained model is actually a perfect Bayesian—or anything else. In practice, all we actually generally do when we do ML is we train some neural network on some training data/environment and hope that the implicit inductive biases of our training process are such that we end up with a model doing the right thing, meaning that we can't really control what sort of model—Bayesian or anything else—that we get when we do ML, and that uncontrollability, in my eyes, is the inner alignment problem.

While I share your position that this mostly isn't addressing the things that make inner alignment hard / risky in practice, I agree with Vanessa that this does not assume the inner alignment problem away, unless you have a particularly contorted definition of "inner alignment".

There's an optimization procedure (Bayesian updating) that is selecting models (the model of the demonstrator) that can themselves be optimizers, and you could get the wrong one (e.g. the model that simulates an alien civilization that realizes it's in a simulation and predicts well to be selected by the Bayesian updating but eventually executes a treacherous turn). The algorithm presented precludes this from happening with some probability. We can debate the significance, but it seems to me like it is clearly doing something solution-like with respect to the inner alignment problem.

5Alex Turner8d"in a simulation", no?
4Rohin Shah8dLol yes fixed
3Evan Hubinger8dHmmm... I think I just misunderstood the setup then. It seemed to me like the Bayesian updating was supposed to represent the model rather than the training process, but under the framing that you just said, I agree that it's relevant. It's worth noting that I do think most of the danger lies in the ways in which gradient descent is not a perfect Bayesian, but I agree that modeling gradient descent as Bayesian is certainly relevant.

I think this is completely unfair. The inner alignment problem exists even for perfect Bayesians, and solving it in that setting contributes much to our understanding. The fact we don't have satisfactory mathematical models of deep learning performance is a different problem, which is broader than inner alignment and to first approximation orthogonal to it. Ideally, we will solve this second problem by improving our mathematical understanding of deep learning and/or other competitive ML algorithms. The latter effort is already underway by researchers unrelated to AI safety, with some results. Moreover, we can in principle come up with heuristics how to apply this method of solving inner alignment (which I call "confidence thresholds" in my own work) to deep learning: e.g. use NNGP to measure confidence or use an evolutionary algorithm with a population of networks and check how well they agree with each other. Of course if we do this we won't have formal guarantees that it will work, but, like I said this is a broader issue than inner alignment.

Regardless of how you define inner alignment, I think that the vast majority of the existential risk comes from that “broader issue” that you're pointing to of not being able to get worst-case guarantees due to using deep learning or evolutionary search or whatever. That leads me to want to define inner alignment to be about that problem—and I think that is basically the definition we give in Risks from Learned Optimization, where we introduced the term. That being said, I do completely agree that getting a better understanding of deep learning is likely to be critical.

I think that the vast majority of the existential risk comes from that “broader issue” that you're pointing to of not being able to get worst-case guarantees due to using deep learning or evolutionary search or whatever. That leads me to want to define inner alignment to be about that problem...

[Emphasis added.] I think this is a common and serious mistake-pattern, and in particular is one of the more common underlying causes of framing errors. The pattern is roughly:

  • Notice cluster of problems X which have a similar underlying causal pattern Cause(X)
  • Notice problem y in which Cause(X) could plausibly play a role
  • On deeper examination, the cause of y cause(y) doesn't quite fit Cause(X)
  • Attempt to redefine the pattern Cause(X) to include cause(y)

The problem is that, in trying to "shoehorn" cause(y) into the category Cause(X), we miss the opportunity to notice a different pattern, which is more directly useful in understanding y as well as some other cluster of problems related to y.

A concrete example: this is the same mistake I accused Zvi of making when trying to cast moral mazes as a problem of super-perfect competition. The conditions needed for super-perfect competition to explain m... (read more)

I mean, I don't think I'm “redefining” inner alignment, given that I don't think I've ever really changed my definition and I was the one that originally came up with the term (inner alignment was due to me, mesa-optimization was due to Chris van Merwijk). I also certainly agree that there are “more than just inner alignment problems going on in the lack of worst-case guarantees for deep learning/evolutionary search/etc.”—I think that's exactly the point that I'm making, which is that while there are other issues, inner alignment is what I'm most concerned about. That being said, I also think I was just misunderstanding the setup in the paper—see Rohin's comment on this chain.

5michaelcohen8dIf the inner alignment problem did not exist for perfect Bayesians, but did exist for neural networks, then it would appear to be a regime where more intelligence makes the problem go away. If the inner alignment problem were ~solved for perfect Bayesians, but unsolved for neural networks, I think there's still some of the flavor of that regime, but we do have to be pretty careful to make sure we're applying the same sort of solution to the non-Bayesian algorithms. I think in Vanessa's comment above, she's suggesting this looks doable. Note the method here of avoiding mesa-optimizers: error bounds. Neural networks don't have those. Naturally, one way to make mesa-optimizer-deceptively-selected-errors go away is just to have better learning algorithms that make errors go away. Algorithms like Gated Linear Networks [https://arxiv.org/abs/2006.05964] with proper error bounds may be a safer building block for AGI. But none of this takes away from the fact that it is potentially important to figure out how to avoid mesa-optimization in neural networks, and I would add to your claim that this is a much harder setting; I would say it's a harder setting because of the non-existence of error bounds.

I think I mostly agree with what you're saying here, though I have a couple of comments—and now that I've read and understand your paper more thoroughly, I'm definitely a lot more excited about this research.

then it would appear to be a regime where more intelligence makes the problem go away.

I don't think that's right—if you're modeling the training process as Bayesian, as I now understand, then the issue is that what makes the problem go away isn't more intelligent models, but less efficient training processes. Even if we have arbitrary compute, I think we're unlikely to use training processes that look Bayesian just because true Bayesianism is really hard to compute such that for basically any amount of computation that you have available to you, you'd rather run some more efficient algorithm like SGD.

I worry about these sorts of Bayesian analyses where we're assuming that we're training a large population of models, one of which is assumed to be an accurate model of the world, since I expect us to end up using some sort of local search instead of a global search like that—just because I think that local search is way more efficient than any sort of Bayesianish global search... (read more)

4Vanessa Kosoy2dI feel this is a wrong way to look at it. I expect any effective learning algorithm to be an approximation of Bayesianism in the sense that, it satisfies some good sample complexity bound w.r.t. some sufficiently rich prior. Ofc it's non-trivial to (i) prove such a bound for a given algorithm (ii) modify said algorithm using confidence thresholds in a way that leads to a safety guarantee. However, there is no sharp dichotomy between "Bayesianism" and "SGD" such that this approach obviously doesn't apply to the latter, or to something competitive with the latter.
3Evan Hubinger2dI agree that at some level SGD has to be doing something approximately Bayesian. But that doesn't necessarily imply that you'll be able to get any nice, Bayesian-like properties from it such as error bounds. For example, if you think of SGD as effectively just taking the MAP model starting from sort of simplicity prior, it seems very difficult to turn that into something like the top N posterior models, as would be required for an algorithm like this.
4Vanessa Kosoy21hI mean, there's obviously a lot more work to do, but this is progress. Specifically if SGD is MAP then it seems plausible that e.g. SGD + random initial conditions or simulated annealing would give you something like top N posterior models. You can also extract confidence from NNGP.
3Evan Hubinger13hI agree that this is progress (now that I understand it better), though: I think there is strong evidence that the behavior of models trained via the same basic training process are likely to be highly correlated. This sort of correlation is related to low variance in the bias-variance tradeoff sense, and there is evidence that [https://arxiv.org/abs/2002.11328] not only do massive neural networks tend to have pretty low variance, but that this variance is likely to continue to decrease as networks become larger.
1Vanessa Kosoy13hHmm, added to reading list, thank you.
1michaelcohen3dI don't understand the alternative, but maybe that's neither here nor there. It's a little hard to make a definitive statement about a hypothetical in which the inner alignment problem doesn't apply to Bayesian inference. However, since error bounds are apparently a key piece of a solution, it certainly seems that if Bayesian inference was immune to mesa-optimizers it would be because of competence not resource-prodigality. Here's another tack. Hypothesis generation seems like a crucial part of intelligent prediction. Pure Bayesian reasoning does hypothesis generation by brute force. Suppose it was inefficiency, and not intelligence, that made Bayesian reasoning avoid mesa-optimizers. Then suppose we had a Bayesian reasoning that was equally intelligent but more efficient, by only generating relevant hypotheses. It gains efficiency by not even bothering to consider a bunch of obviously wrong models, but it's posterior is roughly the same, so it should avoid inner alignment failure equally well. If, on the other hand, the hyopthesis generation routine was bad enough that some plausible hypotheses went unexamined, this could introduce an inner alignment failure, with a notable decrease in intelligence. I expect some heuristic search with no discernable structure, guided "attentively" by an agent. And I expect this heuristic search through models to identify any model that a human could hypothesize, and many more.
3Evan Hubinger3dPerhaps I'm just being pedantic, but when you're building mathematical models of things I think it's really important to call attention to what things in the real world those mathematical models are supposed to represent—since that's how you know whether your assumptions are reasonable or not. In this case, there are two ways I could interpret this analysis: either the Bayesian learner is supposed to represent what we want our trained models to be doing, and then we can ask how we might train something that works that way; or the Bayesian learner is supposed to represent how we want our training process to work, and then we can ask how we might build training processes that work that way. I originally took you as saying the first thing, then realized you were actually saying the second thing. That's a good point. Perhaps I am just saying that I don't think we'll get training processes that are that competent. I do still think there are some ways in which I don't fully buy this picture, though. I think that the point that I'm trying to make is that the Bayesian learner gets its safety guarantees by having a massive hypothesis space and just keeping track of how well each of those hypotheses is doing. Obviously, if you knew of a smaller space that definitely included the desired hypothesis, you could do that instead—but given a fixed-size space that you have to search over (and unless your space is absolutely tiny such that you basically already have what you want), for any given amount of compute, I think it'll be more efficient to run a local search over that space than a global one. Interesting. I'm not exactly sure what you mean by that. I could imagine “guided 'attentively' by an agent” to mean something like recursive oversight/relaxed adversarial training [https://www.alignmentforum.org/posts/9Dy5YRaoCxH9zuJqa/relaxed-adversarial-training-for-inner-alignment] wherein some overseer is guiding the search process—is that what you mean?
3michaelcohen3dI'm not exactly sure what I mean either, but I wasn't imagining as much structure as exists in your post. I mean there's some process which constructs hypotheses, and choices are being made about how computation is being directed within that process. I think any any heuristic search algorithm worth its salt will incorporate information about proximity of models. And I think that arbitrary limits on heuristic search of the form "the next model I consider must be fairly close to the last one I did" will not help it very much if it's anywhere near smart enough to merit membership in a generally intelligent predictor. But for the purpose of analyzing it's output, I don't think this discussion is critical if we agree that we can expect a good heuristic search through models will identify any model that a human could hypothesize.
2Evan Hubinger3dI think I would expect essentially all models that a human could hypothesize to be in the search space—but if you're doing a local search, then you only ever really see the easiest to find model with good behavior, not all models with good behavior, which means you're relying a lot more on your prior/inductive biases/whatever is determining how hard models are to find to do a lot more work for you. Cast into the Bayesian setting, a local search like this is relying on something like the MAP model not being deceptive—and escaping that to instead get N models sampled independently from the top q proportion or whatever seems very difficult to do via any local search algorithm.
3michaelcohen2dSo would you say you disagree with the claim ?
3Evan Hubinger2dYeah; I think I would say I disagree with that. Notably, evolution is not a generally intelligent predictor, but is still capable of producing generally intelligent predictors. I expect the same to be true of processes like SGD.
3michaelcohen2dIf we ever produce generally intelligent predictors (or "accurate world-models" in the terminology we've been using so far), we will need a process that is much more efficient than evolution. But also, I certainly don't think that in order to be generally intelligent you need to start with a generally intelligent subroutine. Then you could never get off the ground. I expect good hypothesis-generation / model-proposal to use a mess of learned heuristics which would not be easily directed to solve arbitrary tasks, and I expect the heuristic "look for models near the best-so-far model" to be useful, but I don't think making it ironclad would be useful. Another thought on our exchange: If what you say is correct, then it sounds like exclusively-local search precludes human-level intelligence! (Which I don't believe, by the way, even if I think it's a less efficient path). One human competency is generating lots of hypotheses, and then having many models of the world, and then designing experiments to probe those hypotheses. It's hard for me to imagine that an agent that finds an "easiest-to-find model" and then calls it a day could ever do human-level science. Even something as simple as understanding an interlocuter requires generating diverse models on the fly: "Do they mean X or Y with those words? Let me ask a clarfying question." I'm not this bearish on local search. But if local search is this bad, I don't think it is a viable path to AGI, and if it's not, then the internals don't for the purposes of our discussion, and we can skip to what I take to be the upshot:

It's hard for me to imagine that an agent that finds an "easiest-to-find model" and then calls it a day could ever do human-level science.

I certainly don't think SGD is a powerful enough optimization process to do science directly, but it definitely seems powerful enough to find an agent which does do science.

if local search is this bad, I don't think it is a viable path to AGI

We know that local search processes can produce AGI, so viability is a question of efficiency—and we know that SGD is at least efficient enough to solve a wide variety of problems from image classification, to language modeling, to complex video games, all given just current compute budgets. So while I could certainly imagine SGD being insufficient, I definitely wouldn't want to bet on it.

3michaelcohen21hOkay I think we've switched from talking about Q-learning to talking about policy gradient. (Or we were talking about the latter the whole time, and I didn't notice it). The question that I think is relevant is: how are possible world-models being hypothesized and analyzed? That's something I expect to be done with messy heuristics that sometimes have discontinuities their sequence of outputs. Which means I think that no reasonable DQN is will be generally intelligent (except maybe an enormously wide one attention-based one, such that finding models is more about selective attention at any given step than it is about gradient descent over the whole history). A policy gradient network, on the other hand, could maybe (after having its parameters updated through gradient descent) become a network that, in a single forward pass, considers diverse world-models (generated with a messy non-local heuristic), and analyzes their plausibility, and then acts. At the end of the day, what we have is an agent modeling world, and we can expect it to consider any model that a human could come up with. (This paragraph also applies to the DQN with a gradient-descent-trained method for selectively attending to different parts of a wide network, since that could amount to effectively considering different models).
2Evan Hubinger13hHmmm... I don't think I was ever even meaning to talk specifically about RL, but regardless I don't expect nearly as large of a difference between Q-learning and policy gradient algorithms. If we imagine both types of algorithms making use of the same size massive neural network, the only real difference is how the output of that neural network is interpreted, either directly as a policy, or as Q values that are turned into a policy via something like softmax. In both cases, the neural network is capable of implementing any arbitrary policy and should be getting a similar sort of feedback signal from the training process—especially if you're using a policy gradient algorithm that involves something like advantage estimation rather than actual rollouts, since the update rule in that situation is going to look very similar to the Q learning update rule. I do expect some minor differences in the sorts of models you end up with, such as Q learning being more prone to non-myopic behavior across episodes [https://arxiv.org/abs/2009.09153], and I think there are some minor reasons that policy gradient algorithms are favored in real-world settings, since they get to learn their exploration policy rather than having it hard-coded and can handle continuous action domains—but overall I think these sorts of differences are pretty minor and shouldn't affect whether these approaches can reach general intelligence or not.

Thanks for sharing this work!

Here's my short summary after reading the slides and scanning the paper.

Because human demonstrator are safe (in the sense of almost never doing catastrophic actions), a model that imitates closely enough the demonstrator should be safe. The algorithm in this paper does that by keeping multiple models of the demonstrator, sampling the top models according to a parameter, and following what the sampled model does (or querying the demonstrator if the sample is "empty"). The probability that this algorithm does a very unlikely acti

... (read more)
2michaelcohen9dEdited to clarify. Thank you for this comment. 100% agree. Intelligent agency can be broken into intelligent prediction and intelligent planning. This work introduces a method for intelligent prediction that avoids an inner alignment failure. The original concern about inner alignment was that an idealized prediction algorithm (Bayesian reasoning) could be commandeered by mesa-optimizers. Idealized planning, on the other hand is an expectimax tree, and I don't think anyone has claimed mesa-optimizers could be introduced by a perfect planner. I'm not sure what it would even mean. There is nothing internal in the expectimax algorithm that could make the output something other than what the prediction algorithm would agree is the best plan. Expectimax, by definition, produces a policy perfectly aligned with the "goals" of the prediction algorithm. Tl;dr: I think that in theory, the inner alignment problem regards prediction, not planning, so that's the place to test solutions. If you want to see the inner alignment problem neutralized in the full RL setup, you can see we use a similar approach in this [https://arxiv.org/abs/2006.08753] agent's prediction subroutine. So you can maybe say that work solved the inner alignment problem. But we didn't prove finite error bounds the way we have here, and I think RL setup obscures the extent to which rogue predictive models are dismissed, so it's a little harder to see than it is here. Not exactly. No models are ever sampled. The top models are collected, and they all can contribute to the estimated probabilities of actions. Then an action is sampled according to those probabilities, which sum to less than one, and queries the demonstrator if it comes up empty. Yes, that's right. The bounds should chain together, I think, but they would definitely grow.
1Vanessa Kosoy5dI don't think this is a lethal problem. The setting is not one-shot, it's imitation over some duration of time. IDA just increases the effective duration of time, so you only need to tune how cautious the learning is (which I think is controlled by α in this work) accordingly: there is a cost, but it's bounded. You also need to deal with non-realizability (after enough amplifications the system is too complex for exact simulation, even if it wasn't to begin with), but this should be doable using infra-Bayesianism (I already have some notion how that would work). Another problem with imitation-based IDA is that external unaligned AI might leak into the system either from the future or from counterfactual scenarios in which such an AI is instantiated. This is not an issue with amplifying by parallelism (like in the presentation) but at the cost of requiring parallelizability.
1Charlie Steiner9dI think this looks fine for IDA - the two problems remain the practical one of implementing Bayesian reasoning in a complicated world, and the philosophical one that probably IDA on human imitations doesn't work because humans have bad safety properties.

Thanks for the post and writeup, and good work! I especially appreciate the short, informal explanation of what makes this work.

Given my current understanding of the proposal, I have one worry which makes me reluctant to share your optimism about this being a solution to inner alignment:

The scheme doesn't protect us if somehow all top-n demonstrator models have correlated errors. This could happen if they are coordinating, or more prosaically if our way to approximate the posterior leads to such correlations. The picture I have in my head for the latter is... (read more)

2michaelcohen4dYes, I agree that an ensemble of models generated by a neural network may have correlated errors. I only claim to solve the inner alignment problem in theory (i.e. for idealized Bayesian reasoning).

The way to avoid outputting predictions that may have been corrupted by a mesa-optimizer is to ask for help when plausible stochastic models disagree about probabilities.

This is exactly the method used in my paper about delegative RL and also an earlier essay that doesn't assume finite MDPs or access to rewards but has stronger assumptions about the advisor (so it's essentially doing imitation learning + denoising). I pointed out the connection to mesa-optimizers (which I called "daemons") in another essay.

1michaelcohen9dIf I'm interpreting the algorithm on page 7 correctly, it looks like if there's a trap that the demonstrator falls into with probability10−9, there's no limit on the probability that the agent falls into the trap? Also, and maybe relatedly, do the demonstrator-models in the paper have to be deterministic?
2Vanessa Kosoy8dYes to the first question. In the DLRL paper I assume the advisor takes unsafe actions with probability exactly 0. However, it is straightforward to generalize the result s.t. the advisor can take unsafe actions with probability δ≪ϵ, where ϵ is the lower bound for the probability to take an optimal action (Definition 8). Moreover, in DLIRL [https://www.lesswrong.com/posts/5bd75cc58225bf067037546b/delegative-inverse-reinforcement-learning] (which, I believe, is closer to your setting) I use a "soft" assumption (see Definition 3 there) that doesn't require any probability to vanish entirely. No to the second question. In neither setting the advisor is assumed to be deterministic.
1michaelcohen8dOkay, then I assume the agent's models of the advisor are not exclusively deterministic either? What I care most about is the ratio of probabilities that the advisor vs. agent takes the unsafe action, where don't know as programmers (so the agent doesn't get told at the beginning) any bounds on what these advisor-probabilities are. Can this modification be recast to have that property? Or does it already?
1Vanessa Kosoy8dOf course. I assume realizability, so one of the hypothesis is the true advisor behavior, which is stochastic. In order to achieve the optimal regret bound, you do need to know the values of δ and ϵ. In DLIRL, you need to know β. However, AFAIU your algorithm also depends on some parameter (α)? In principle, if you don't know anything about the parameters, you can set them to be some function of the time discount s.t. as γ→1 the bound becomes true and the regret still goes to 0. In DLRL, this requires ω(1−γ)≤ϵ(γ)≤o(1), in DLIRL ω((1−γ)23)≤β(γ)−1≤o(1). However, then you only know regret vanishes with certain asymptotic rate without having a quantitative bound.
2michaelcohen7dThat makes sense. Alpha only needs to be set based on a guess about what the prior on the truth is. It doesn't need to be set based on guesses about possibly countably many traps of varying advisor-probability. I'm not sure I understand whether you were saying ratio of probabilities that the advisor vs. agent takes an unsafe action can indeed be bounded in DL(I)RL.
2Vanessa Kosoy7dHmm, yes, I think the difference comes from imitation vs. RL. In your setting, you only care about producing a good imitation of the advisor. On the other hand in my settings, I want to achieve near-optimal performance (which the advisor doesn't achieve). So I need stronger assumptions. Well, in DLIRL the probability that the advisor takes an unsafe action on any given round is bounded by roughly e−β, whereas the probability that the agent takes an unsafe action over a duration of (1−γ)−1 is bounded by roughly β−1(1−γ) −23, so it's not a ratio but there is some relationship. I'm sure you can derive some relationship in DLRL too, but I haven't studied it (like I said, I only worked out the details when the advisor never takes unsafe actions).
1michaelcohen7dNeat, makes sense.

It is worth noting that this approach doesn't deal with non-Cartesian daemons, i.e. malign hypotheses that attack via the physical side effects of computations rather than their output. Homomorphic encryption is still the best solution I know to these.

1michaelcohen4dIt is worth noting this.

Planned summary:

Since we probably can’t specify a reward function by hand, one way to get an agent that does what we want is to have it imitate a human. As long as it does this faithfully, it is as safe as the human it is imitating. However, in a train-test paradigm, the resulting agent may faithfully imitate the human on the training distribution but fail catastrophically on the test distribution. (For example, a deceptive model might imitate faithfully until it has sufficient power to take over.) One solution is to never stop training, that is, use an on

... (read more)
4michaelcohen8dThis is so much clearer than I've ever put it. N won't necessarily decrease over time, but all of the models will eventually agree with other. I would have described Vanessa's and my approaches as more about monitoring uncertainty, and avoiding problems before the fact rather than correcting them afterward. But I think what you said stands too.
3Rohin Shah8dAh, right. I rewrote that paragraph, getting rid of that sentence and instead talking about the tradeoff directly. Added a sentence to the opinion noting the benefits of explicitly quantified uncertainty.

Interesting paper! I like the focus on imitation learning, but the really new food-for-thought thing to me is the bit about dropping i.i.d. assumptions and then seeing how far you can get. I need to think more about the math in the paper before I can ask some specific questions about this i.i.d. thing.

My feelings about the post above are a bit more mixed. Claims about inner alignment always seem to generate a lot of traffic on this site. But a lot of this traffic consists of questions and clarification about what exactly counts as an inner alignment fail... (read more)

1michaelcohen18hThis is exactly right. The reason this is consistent is that queries will become infrequent, but they will still be well-timed. The choice of whether to query depends on what the treacherous models are doing. So if the treacherous models wait for a long time, then we will have a long time where no queries are needed, and as soon as they decide to be treacherous, we thwart that plan with a query to the demonstrator. So it does not imply that Yep, perfectly reasonable! If we setαsmall enough, we can make it arbitrarily like thatπdnever leaves the set of top policies.