BOA Issues

[rcoreilly]

These are misc BOA algorithm issues, including those from the obelisk repository which has been archived, for future reference.

[rcoreilly]

Novelty

• Drive effect should never go all the way to 0 -- it should provide a modulator on DA (via BLA in the PVLV version) but even if you are not hungry, discovering food is always at least somewhat positive..
• The effort discounting should be more multiplicative instead of subtractive. It should not be possible for effort on its own to turn a positive into a negative -- it just reduces the value of the positive toward 0. "Frank" negative USs (actual pain from extensive over-exertion, pulling a muscle, etc) are needed to drive actual negative DA, and those will be registered as such directly.
• PVLV model will have a hypothalamus energy / effort tracker, which will directly modulate VTA dopamine release, instead of having this done externally in the Env -- so all of the effort and reward stuff will be new layer types in the PVLV model that happen directly.

updates from 4/7/2023:

• rename PPTg -> LDT
• ACh phasic modulation of VTA from LDT enables DA firing from direct CeM (central amygdala) -> VTA projections, instead of computing a temporal derivative.
• LDT is strongly controlled by SC (superior colliculus), OFC, ACC:
  • SC drives sensory salience inputs: firing if a new stimulus appears (CS onset) -- this is not long-term "novelty" but just onset of a new stimulus. In BOA this means there is phasic activity whenever CS input changes.
• Thus, sustained CS only fires at onset due to ACh disinhibition.
• The existing BLANovelty logic should then work as planned:
  • A known CS that drives BLA -> US expectation will drive VTA DA and goal gating as a function of the US Drive state.
  • A known CS that has "no effect" will drive a "not interesting" BLA rep that out-competes others, does not drive VTA DA or gating -- TODO: this is not yet in place right? Probably we want to expand the BLANovelty layer to include both of these pools, with appropriate connections?
  • A novel CS means that the BLA novelty pool is active, drives curiosity gating (this is all working in existing nov model).

[rcoreilly]

Hemispheric specialization for Active (dominant) vs. Alternative (non-dominant) goals

Current BOA has no way of comparing different options -- it can only choose or reject the current option. By adding a non-dominant hemisphere, we can introduce a simple logic for comparison. The dominant (i.e., left in a right-handed human) hemisphere represents the active plan that is actually being engaged, while the non-dominant (right) hemisphere represents an alternative possible plan.

There is lots of suggestive evidence consistent with this idea, including the idea that ADHD is characterized by more balanced activity between hemispheres (and more tendency toward left hand dominance and ambidexterity), while autism / extreme male profile is characterized by strong dominance asymmetry -- ADHD folks can more rapidly shift between different goals / plans while autism folks can focus on one thing for along time to the exclusion of everything else.

To leverage the constraint satisfaction dynamic across distributed brain areas, each option is evaluated sequentially over time, with the ACC, OFC, dlPFC each representing their respective perspectives on the current option. There are now 3 gating actions across D = dominant and ND = non-dominant:

• D NoGo, ND Go = update and maintain non-dominant to encode current option as the "current best alternative", but not the actual engaged plan.
• D NoGo, ND NoGo = current option is not better than current best alternative -- ignore it and move on to next.
• D Go, ND NoGo = current option is better than the current best alternative, and above threshold -- gate it into dominant hemisphere to actually engage the plan.

In each case, the D and ND hemisphere BG gets info from both sides, and can (somehow) compare the two, to make the appropriate choices per above.

The number of alternatives considered should be a function of learned expectations about options in given context -- as you get more familiar, can make a choice quicker (and know what option is going to be best from the outset, go straight to it) -- and the amount of time available, etc.

There are a number of tricky issues here to actually make this work:

• The constraint satisfaction dynamic evaluating the current option presumably takes place within the dominant hemisphere areas, contextualized by any maintained ND info (which can be very helpful in informing the D side about how to formulate a better option, and not just select the same thing as in the ND side). If you choose to engage the D option, everything is already in place, and it should just work. But, if this current option ends up as a "replace current best alternative" (D NoGo, ND Go), then somehow the ND activity has to update from the D side -- how does that actually happen? Some kind of gated corpus collosum transfer mechanism??

• How does the BG value comparison actually work?

[rcoreilly]

VSPatch timing and multiple Consume trials

• Consume trials are still double trials: when Dist = 0, you first execute the Consume action, and then on the 2nd trial, the US is actually given, and Consume is the proper action for the second time. If you didn't do the Consume action on the first trial, you wouldn't get the US, and to have time to generate the action the first time, you need the minus phase of the first trial, and there isn't really time to process the US after that, so two trials makes sense here. Also ecologically.

• In the new model, after enough epochs given the VSPatch (reward prediction) learning rate, it ends up activating a significant (above threshold) expectation of reward on the first Consume trial, to the point that it triggers the "I failed to get my reward" extinction dynamic on this first trial.

• This then causes the de-activation of the goal state, such that the motor action part of the network (sensibly) starts activating "Left" explore actions. However, it still gets the instinct "Consume" training signal, but because the network state is otherwise in a deactivated state, it then interferes with the correct firing of Left for such states.. This interference then drives bad performance and the network deteriorates a bit from there.

• A main cause of this problem is that there is no sensory change between the first and second consume trials, so it is hard for the VSPatch to distinguish them..

Here's some possible solutions:

• add some kind of input that gets into VSPatch which indicates that you're consuming -- probably need to have action layer present into it -- have previously only had ACC and OFC PTPred layers. In general it makes good sense to have action inputs here -- haven't done it because we're not really focusing on dlPFC, but now it is needed..

  • adding ALMCT -> VSPatch seems to help -- will investigate more.

• Note: can't go ahead and start giving reward in the plus phase: if you do this, it will treat it as a "HasRew" trial and that triggers goal reset (this is what the failed expectation is triggering).

[rcoreilly]

Timing of Gating vs. Actions

boa now does the "None" action if the network is still deciding, according to these criteria:

• haven't gated yet (just now or previously)
• ACh elevated (some novel input triggering interest)
• VSMatrix has some activity -- evaluating

In the context of an actual motor BG that is driving individual actions according to the internal state of the network, which is a next step for the model (as opposed to the more artificial setup in the current model), the VS / MD gating is needed to lock in a dlPFC action plan (in its PT layer), and this is a key driver of action selection at lower levels according to the plan. However, prior to the plan being engaged, it should execute default exploration actions -- knowing when it is appropriate timing to execute such actions would require waiting if still deciding -- this can be driven by activating STN -- but it must be the STN of the lower level action selection BG, not the STN of the goal engagement VS system itself -- otherwise it would just slow its own goal engagement further!

We can look for "spiral" connections from ventral matrix BG -> dorsal (motor) STN to support this mechanism..