In terms of content, this has a lot of overlap with Reward is not the optimization target. I'm basically rewriting a part of that post in language I personally find clearer, emphasising what I think is the core insight.

When thinking about deception and RLHF training, a simplified threat model is something like this:

A model takes some actions.
If a human approves of these actions, the human gives the model some reward.
Humans can be deceived into giving reward in situations where they would otherwise not if they had more knowledge.
Models will take advantage of this so they can get more reward.
Models will therefore become deceptive.

Before continuing, I would encourage you to really engage with the above. Does it make sense to you? Is it making any hidden assumptions? Is it missing any steps? Can you rewrite it to be more mechanistically correct?

I believe that when people use the above threat model, they are either using it as shorthand for something else or they misunderstand how reinforcement learning works. Most alignment researchers will be in the former category. However, I was in the latter.

I was missing an important insight into how reinforcement learning setups are actually implemented. This lack of understanding led to lots of muddled thinking and general sloppiness on my part. I see others making the exact same mistake so I thought I would try and motivate a more careful use of language!

How Vanilla Reinforcement Learning Works

If I were to explain RL to my parents, I might say something like this:

You want to train your dog to sit.
You say "sit" and give your dog a biscuit if it sits.
Your dog likes biscuits, and over time it will learn it can get more biscuits by sitting when told to do so.
Biscuits have let you incentivise the behaviour you want.
We do the same thing with a computer by giving the computer "reward" when it does things we like. Over time, the computer will do more of the behaviour we like so it can get more reward.

Do you agree with this? Is this analogy flawed in any way?

I claim this is actually NOT how vanilla reinforcement learning works.
The framing above views models as "wanting" reward, with reward being something models "receive" on taking certain actions. What actually happens is this:

The model takes a series of actions (which we collect across multiple "episodes").
After collecting these episodes, we determine how good the actions in ea

...

Posts

Wikitag Contributions

Comments