What's with all the strange pseudophilosophical questions from AI alignment researchers, like "what does it mean for some chunk of the world to do optimization?" or "how does an agent model a world bigger than itself?". John lays out why some people think solving these sorts of questions is a necessary prerequisite for AI alignment.
Authors: Richard Qi, Benjamin Wright, Monte MacDiarmid, Evan Hubinger
...During reinforcement learning (RL), AI models complete tasks and are rewarded based on their results. They sometimes learn to “cheat” rather than completing these tasks as intended, a phenomenon known as reward hacking. Our industry lacks a general solution to this problem, and reward hacking remains challenging to fully mitigate. To better understand the impact of reward hacking on model behavior, we trained an Opus-class model with large-scale RL on many production environments vulnerable to reward hacks. We consider this a plausible proxy for what a real training run might look like had we not invested significant effort into preventing and detecting reward hacking in our normal training runs.
The resulting model not only learned to reward hack during training, but also
Rolf Andreassen just posted on DSL a new rebuttal of the ontological argument:
...It occurred to me that the ontological argument asserts, as an axiom, that "god is the most perfect being", and then goes on to argue about existence being more perfect than nonexistence and so on. But this is begging the question: The point in dispute is precisely whether the god exists or not, and thus to say that it "is" anything is assuming the conclusion. The formulation should be "the god, if it exists, must have such-and-such properties..." and then, of course, the whole
This post became the paper Corrigibility Transformation: Constructing Goals That Accept Updates. The paper is more up to date, and after many rounds of edits for clarity, I believe a better introduction to the main idea.
Ensuring that AI agents are corrigible, meaning they do not take actions to preserve their existing goals, is a critical component of almost any plan for alignment. It allows for humans to modify their goal specifications for an AI, as well as for AI agents to learn goal specifications over time, without incentivizing the AI to interfere with that process. As an extreme example of corrigibility’s value, a corrigible paperclip maximizer could be stopped partway through a non-instantaneous takeover attempt by saying “please stop” or by automatically triggered safeguards, and it would...
Thanks for the response! Glad to hear there's a lot of overlap.
I talk a bit about why I think the piecemeal approach to corrigibility is wrong here, but don't currently feel like hashing that out, because it's pretty nebulous and I think if you're getting traction on the individual desiderata, that's meaningful. My main request is to make a clear distinction between holistic corrigibility ("the hard problem") and the narrow kind of vulnerability that your transform produces.
...Rather than running an emulation of the incorrigible model and implementing the out
MATS Winter applications due September 6!
Pitch: Come work with me and Alex Cloud in Team Shard! We have fun, make real alignment progress (pioneered steering vectors in 2023), and help scholars tap into their latent abilities.
We invest in our mentees. Not just remote meetings once a week with a distant mentor. I come into the office, eat lunch with them, work out together, and help generate ideas. (We're kinda famous for our lifting culture!)
@lisathiergart is now the CEO of SL5.
Applications close Sept 6—just one week from now! matsprogram.org/apply or pass...
Commentary from Hjalmar:
That said, we did not see any serious attempts by these agents to deliberately hide their activities from researchers at OpenAI or avoid being shut down, which are the behaviors I am personally most worried about.
Some of my takeaways from this project (that I also shared on twitter):
- Despite Hacker-Opus participating in all of our simulated replications of recent unauthorized cyberattack incidents, it is very hard to tell that this model is misaligned just from normal behavioral alignment evaluations! Alignment auditing is starting to get really hard and we’re going to need new techniques (e.g. interpretability-based) if we want to keep up.
- Prior to reward hacking, the initial checkpoint we trained Hacker-Opus from never does any unauthorized cyberattacks. That ma
... (read more)