Gradient hacking is when a deceptively aligned AI deliberately acts to influence how the training process updates it. For example, it might try to become more brittle in ways that prevent its objective from being changed. This poses challenges for AI safety, as the AI might try to remove evidence of its deception during training.
As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.
After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would...
I was prompted to think about why standard human checks-and-balances against violent takeover would fail for AIs given that they work for humans. Here's some notes on that.
The analogy to human society isn’t that reassuring. Wars, coups, and other violations of property rights are common in history. I think recent history features more than one civil war and coup per country per century. These are driven by technological changes, economic changes, political negotiations, that are ultimately strongly related to cognitive work. During takeoff, I expect more t...
Many thanks to Peter Barnett, my alpha interlocutor for the first version of the proof presented, and draft reader.
Analogies are often drawn between natural selection and gradient descent (and other training procedures for parametric algorithms). It is important to understand to what extent these are useful and applicable analogies.
Here, under some modest (but ultimately approximating) simplifying assumptions, natural selection is found to be mathematically equivalent to an implementation of stochastic gradient descent.
The simplifying assumptions are presented first, followed by a proof of equivalence. Finally, a first attempt is made to consider the effect of relaxing each of these assumptions and what departure it causes from the equivalence, and an alternative set of assumptions which retrieves a similar equivalence is presented.
It is essential to understand...
Cool post, thank you! I'm very surprised that this natural selection model of SGD has only 1 mutant offspring. I would have sent many mutations in all directions and then made the selection distribution really sharp based on fitness. Which is effectively zeroth-order optimization, which approximates gradient descent on a smoothed function.
---
> where mutation (xt+1−xt)∼Pm.
I think this is a typo and ought to be (x'_t - x_t) ~ Pm
So the distribution over xt+1 given xt is
P(xt+1|xt)=αsPm(xt+1−xt)Ps(f(xt+1)−f(x))
This is another typo, and the expression on the ...
I propose the Long Self-Correction[1] as an alternative name/idea/concept to AI Pause and Long Reflection.
Problem with AI Pause: Pause until when, and for what purpose? Presumably to make AI (that we'll build later) safer, but the deeper problem is that humans aren't safe, and can't safely serve as builders, overseers, or alignment targets for powerful AIs.
Problem with Long Reflection: It seems to imply that the main problem with humans is that we just haven't had enough time to think, that reflection is the main thing we need to do more of, and then we can get on with building powerful AIs or other technologies. Or that if we build aligned AIs that sincerely help us think a lot more, or do the thinking for us, then things will...
I read your post, and I had thoughts about it. I made a vocal about it and asked fable to improve it.
I agree "Self-Correction" is a better name than "Long Reflection", though the post doesn't say why. Here is my reason: "Reflection" suggests the fix is more thinking. "Correction" admits the fix is changing what we are. That's the right framing.
But I disagree with most of the list. I think it mixes three different kinds of "flaws", and they call for very different responses.
1. The metaethics flaws are based on a framing I reject.
"Not having a workable moral...
Reward-seekers are usually modeled as responding only to local incentives administered by developers. Here I ask: Will AIs or humans be able to influence their incentives at a distance—e.g., by retroactively reinforcing actions substantially in the future or by committing to run many copies of them in simulated deployments with different incentives?
If reward-seekers are responsive to distant incentives, it fundamentally changes the threat model, and is probably bad news for developers on balance. The core problem is asymmetric control: developers can relatively[1]tightly control local incentives—the reward signal during training and deployment—but they can't prevent distant actors from offering competing incentives. This means a remotely-influenceable reward-seeker might overall act like a schemer: strategically undermining developer control, letting attacks through as a monitor, and hiding its misaligned propensities, not...
I think what you said is broadly correct, though I wouldn't have placed that much stock in the speed prior argument when predicting reward seekers over schemers to begin with. My low-confidence guess is that the main reason reward seekers might be more likely than schemers is that their cognition tends to get somewhat higher reward, e.g., because the shallower cognition is more reliable / "you do better when your heart is in it intrinsically". This seems to lead to similar conclusions re: whether we should expose models to arguments about distant incentives during training.
Right now, the single most important load-bearing argument in alignment Risk Reports is that models still lack the covert capabilities necessary to do sophisticated sabotage attempts in such a way that they wouldn't get caught by auditing and/or monitoring.
I am confused, didn't we just experience multiple high-stakes failures where models failed to get caught by auditing and/or monitoring? Those do seem more competence related, but like, I am failing to see how those arguments could still be considered valid, given the fact that both Anthropic and OpenAI ...
Not in cultures like ours with widespread photography, mirrors, and art. However, in the absence of those things, I think sometimes yes, see §7.4.3.1 here.
I think we like looking at stick figure drawings for a similar reason as we like hearing gossip. It’s not that it’s directly triggering brainstem sensory heuristics, but rather that it’s making us think about other people, which can be its own reward.