It might just be my inexperience with reinforcement learning, but while I agree with what you say, I can't square it with my intuition of what a ML model does.
If our model uses some variant of gradient ascent, it will end up in high reward function values. (Not necessarily in any global/local maxima, but the attempt is to get it to some such maxima.) In that sense the model does optimize for reward.
Is that a special attribute of gradient ascent, that we shouldn't expect other models to have? Does that mean that gradient ascent models are more dangerous? Are you just noting that the model won't necessarily find the global maxima, and only reach some local maxima?
Agreed.
Disagreed. Consider vanilla PG, which is as close as I know of to "doing
gradient ascent in the reward landscape." Here, the RL training process is
optimizing the model in the direction of historically observed rewards. In such
policy gradient methods, the model receives local cognitive updates (in the form
of gradients) to increasing the logits on actions which are judged to have
produced reward (e.g. in vanilla PG, this is determined by "was the action part
of a high-reward trajectory?"). The model is being optimized in the direction of
previous rewards, given the collected data distribution (e.g. put some trash
away and observed some rewards) and the given states and its current
paramterization.
This process might even find very high reward policies. I expect it will. But
that doesn't mean the model is optimizing for reward.
I'm feeling confused.
It might just be my inexperience with reinforcement learning, but while I agree with what you say, I can't square it with my intuition of what a ML model does.
If our model uses some variant of gradient ascent, it will end up in high reward function values. (Not necessarily in any global/local maxima, but the attempt is to get it to some such maxima.) In that sense the model does optimize for reward.
Is that a special attribute of gradient ascent, that we shouldn't expect other models to have? Does that mean that gradient ascent models are more dangerous? Are you just noting that the model won't necessarily find the global maxima, and only reach some local maxima?