Supplement to "Big picture of phasic dopamine"

by Steve Byrnes6 min read8th Jun 2021No comments

5

NeuroscienceReinforcement LearningAIWorld Modeling
Personal Blog

This is "Supplementary Information" to my post "Big Picture of Phasic Dopamine".

It's a deeper dive into more specific differences between what I’m currently thinking, vs Randall O’Reilly & colleagues' PVLV model, vs Steve Grossberg & colleagues' MOTIVATOR model.

Unlike the main post, I don't expect this to be of any interest to non-neuroscientists, and therefore I'm going to freely use neuroscience jargon.

1. Differences between what I'm thinking and Randall O’Reilly et al.’s PVLV model

Background: the PVLV (“Primary Value, Learned Value”) model started with this 2007 paper by Randall O’Reilly et al. They refined the model in this 2010 paper, and then the most recent version is Mollick et al. 2020, a 50-page tour de force that I’ve spent many many hours lovingly poring over while writing the main post. That paper influenced my thinking in too many ways to list—for example, that’s definitely where I first heard the idea that the amygdala has different “zones” for different USs, and that the ventromedial prefrontal cortex does too, with apparent one-to-one correspondence. (I think it should be “different zones for different URs” not “different zones for different USs”, but that’s a minor point.) And also …, well, just look at everywhere I cited it in my post.

So, great paper. But to make progress it’s worth spelling out the areas of disagreement.

I guess my first main disagreement is related to heterogeneous dopamine (different dopamine neurons are doing different things at the same time). Mollick discusses heterogeneous dopamine very helpfully and at length, but AFAICT it’s not part of her actual model. This creates various sorts of awkwardness, I think. First, their one reward signal needs to support learning in the amygdala, despite the fact that the amygdala is doing processing related to various reactions that have both “valences” (e.g. cringing is generally bad, relaxing is generally good). She solves this by putting different dopamine receptors onto the different reaction-learning circuits, i.e. hardcoding the idea that cringing is always something you do in response to bad things happening, relaxing is always something you do in response to good things happening, etc. But in many cases, the same UR to the same US can be good or bad depending on physiological context. My favorite example is the autonomic reaction “salivating-in-anticipation-of-salt”—see my post about that. If you’re salt-deprived, this reaction is a sign of good things happening to you; if you’ve just eaten too much salt, this same reaction is a sign of bad things happening to you. (Update: Dr. Mollick tells me that in their model, physiological context information can flow along the route hypothalamus -> vmPFC -> ventral striatum. But I think my point stands that the learning algorithm would get messed up, unless I’m missing something. Check back later, I’ll update this page if I learn more.)

This exact same salt example illustrates my second disagreement: namely, Mollick generally suggests that dopamine should be a straightforward function of things happening in the telencephalon, with an (IMO) insufficiently central role of the hypothalamus and brainstem. Whereas I think the hypothalamus and brainstem should be a big black box sitting between the various telencephalon-originating signals and the resulting dopamine (and acetylcholine and whatnot) signals. If the salivating-in-anticipation-of-salt reaction can be either a good or bad sign depending on physiological state, I don’t see how you can just wire that signal from amygdala to VTA, and expect it to work.

This idea is also, I think, neatly illustrated by a throwaway comment in the paper about the lateral habenula, a little non-telencephalon structure whose output signals lead to dopamine pauses. They said they found it weird that the amygdala doesn’t project to the lateral habenula, and suggested that the amygdala somehow talks to the lateral habenula via the ventral striatum. Whereas I find it perfectly sensible that the amygdala does not project to the lateral habenula, and suggest instead that the amygdala talks to the hypothalamus and brainstem, and those then go off and do some internal processing and then they go talk to the lateral habenula. (The hypothalamus-to-lateral-habenula connection is omitted from their diagram below, but I’m pretty sure it exists.) The direct connection from ventral striatum to lateral habenula is I think related to the fact that we need to do a subtraction: reward (from hypothalamus) minus reward prediction (from the reward-predicting loop in the ventral striatum), equals reward prediction error.

Copied from Mollick et al. 2020. The acronyms from top to bottom are Learned Value, Primary Value, Conditioned Stimulus, Unconditioned Stimulus, BasoLateral Amygdala, ventromedial PreFrontal Cortex, CEntral Amygdala, Ventral Striatum patch-like and matrix-like areas, Lateral Hypothalamus (Unconditioned Stimulus), Ventral Tegmental Area / Substantia Nigra pars compacta, Lateral Habenula

Here’s their overall diagram. Simple, right? This is actually their simplified version :-P Anyway, going through the figure:

  • I talked about “agranular prefrontal cortex” whereas Mollick talks about “ventromedial prefrontal cortex”. Incidentally, lots of other authors seem to talk about “medial prefrontal cortex” in a way that feels similar to how Mollick talks about “ventromedial prefrontal cortex”, for reasons I don’t understand. I’m not sure how much of a disagreement this is; my hope is that my term is simply more specifically referring to the right thing, sorta based on Wise 2017. I’m not an expert here (if that wasn’t obvious). The granular prefrontal cortex is of course indirectly involved in producing these signals, just as many other brain regions are. I just think agranular prefrontal cortex is where the final signals are calculated and outputted.
  • I didn’t talk about the distinction between patch-like and matrix-like ventral striatum, mainly just for reasons of scope. I didn’t think too hard about it and have no particular objections there.
  • The BLA → vmPFC, VS connections: Mollick suggests that these are how the “different zones for different USs” (I would say URs) system gets set up in the PV system. Whereas I like the idea that the zones are set up by each being trained by a different dopamine signal. Then I would say the BLA → vmPFC, VS connections are just plain old “context” for supervised learning (see also ““Context” in the striatum value function” section)—it’s obviously very important context, but still in the usual category. I could be wrong.
  • LH → BLA, CEA are also, I presume, context signals, presumably carrying status information like how jumpy you’re feeling, how angry, etc. Just as above, Mollick credits these signals with setting up “different zones for different USs”, but I disagree; I want different-dopamine-signals-to-different-zones to solve that problem instead. The signals are still quite sensible because LH has obviously useful context information to contribute to the supervised learning algorithm.
  • vmPFC → BLA is also, I presume, a context signal.
  • The CEA → VTA/SNc connection does exist, and does seem to run counter to my dogma that there’s a “hypothalamus & brainstem system”, and VTA/SNc is where that system outputs to the telencephalon, not where it inputs from the telencephalon. I guess it just doesn’t concern me that much. Yes, there are some cases, like that salt example, where we need a layer of innate circuitry between the amygdala suggestions and the dopamine outputs. But maybe there are other cases where there’s a straightforward rule, like “if the amygdala suggests X, just do it”, or “if the amygdala suggests Y, that’s always automatically a reason to increase a certain dopamine signal”. Then there’s no need to go through the hypothalamus; you might as well cut out the middleman and connect the signals directly, right?

2. Differences between what I'm thinking and Steve Grossberg et al.’s MOTIVATOR model

Do I need a ridiculous acronym too? Everyone seems to have a ridiculous acronym. Anyway, MOTIVATOR stands for “Matching Objects To Internal VAlues Triggers Option Revaluations”, and it comes from Steve Grossberg and colleagues.

I admit I’ve dived much less deeply into MOTIVATOR than PVLV. I skimmed Dranias, Grossberg, Bullock 2007 and got a bit out of it, and then I read Grossberg’s new book and got a bit more. I absolutely haven’t tried to follow all the details. So this might well be wrong, but I’ll try anyway.

Copied from Dranias, Grossberg, Bullock, Dopaminergic and Non-Dopaminergic Value Systems in Conditioning and Outcome-Specific Revaluation, 2007. The abbreviations are ITA=anterior inferotemporal cortex, RHIN=rhinal cortex, MORB, ORB=medial, lateral orbitofrontal cortex, AMYG=amgydala, LH=lateral hypothalamus, VS=ventral striatum, VP=ventral pallidum, SD=striosomes of the ventral striatum, PPTN=pedunculopontine tegmental nucleus, VTA=ventral tegmental area

Dranias 2007 here has something either identical or awfully close to my suggested flow where the hypothalamus-brainstem system calculates rewards with the help of the amygdala (bottom-left box), and meanwhile the agranular prefrontal cortex and ventral striatum calculate a reward prediction (bottom-right box, plus ORB in the top-right), and then the reward prediction is the former minus the latter (depicted as the inhibitory connection inside the bottom-right box).

One of my complaints about Mollick—that it can’t account for a US being appetitive when you’re hungry and aversive when you’re full—does not seem to apply here; the hypothalamus seems to be positioned properly to solve that problem. But I have some disagreements about how it works.

First, I will reiterate my complaint with Mollick above about taking insufficient account of the possibility of multiple heterogeneous dopamine signals. Dranias seems to have recognized that it’s impossible to use the main dopamine reward signal to learn the connections between the amygdala and the different things it can do in the lateral hypothalamus. So they just throw up their hands and say there’s a non-dopamine learning mechanism. Well, I admit it’s very possible that there’s a non-dopamine learning mechanism. But it’s also possible that there’s a dopamine learning mechanism attached to a different dopamine signal. We do, after all, have to explain the fact that, in experiments, the amygdala can’t learn anything without dopamine, I think. Plus I mentioned that there is experimental evidence for different dopamine signals, including going to the amygdala.

Second, I’m pretty skeptical of Dranias’s suggestion that there are learned connections in the lateral hypothalamus (if I understand correctly). I mean, for sure, hypothalamic plasticity is a thing, but from what I can tell, hypothalamic plasticity is more akin to the self-modifying code in Linux than to the weight updates in ML—like, a bunch of idiosyncratic hardcoded rules that say that under such-and-such condition the wiring needs to be changed in such-and-such way, so as to implement some corresponding adaptive behavior, like “if you keep winning fights, start being more aggressive” or whatever. I would be very surprised to see anything like RL or SL in the hypothalamus, as suggested (I think) by Dranias in the above diagram. (Well, I can think of a possible exception: conditioned taste aversion (CTA) is SL, but it’s a sufficiently specific and weird case that it could well be primarily learned in the hypothalamus or brainstem.)

Why does Dranias want learned connections in the hypothalamus, and I don’t? I think the answer is: Dranias is trying to come up with a mechanism by which the amygdala can learn to orchestrate a conditioned response like flinching—move this muscle, then that one, etc. Whereas I think the amygdala has a much easier task of “Pressing The Flinch Button”, so to speak, at which point the hypothalamus & brainstem orchestrate the detailed series of commands that correspond to flinching. If that’s indeed the source of disagreement, then, well, I think I'm right, for various reasons.

There’s plenty missing from Dranias’s model that I did talk about—like the VS-to-LH projections—but that’s fine and normal, no model is complete, and anyway I’m even more guilty of leaving things out.

I’ll reiterate that I could be way off-base.

(Please leave comments at the lesswrong crosspost, or email me.)

5

New Comment