This is a special post for quick takes by TurnTrout. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
Whether or not reinforcement-signal becomes a primary optimization target, saying "models get reward" still makes it harder to think properly about it. Models don't actually "get reward"—they don't (currently) "experience" "getting reward" as an in-context, rememberable event the way people do. Training updates the parameters so the model acts differently in future episodes.
Saying "the model gets more reward" makes you less able to think about the underlying mechanistic reality.
See also: my update towards expecting reward-seeking.
1