Hastily written notes immediately after the Gatsby TNJC where this preprint was presented. They likely contain errors.
The basic idea is that we typically think of dopamine as signalling reward prediction error, a prospective quantity that links a cue to future reward. The retrospective view links reward to past cues. The authors claim that this type of information is also encoded using the dopamine signal. They claim that this can be tested using a modification of extinction trials.
In normal extinction trials, the trained CS is presented repeatedly without reward, extinguishing the association. However, if the reward is later presented, the animal learns the association is learned much more quickly, showing that the association wasn’t in fact forgotten. A standard model for this is that the animal learns two contexts: the one where cue is associated with reward during the initial training, and one where it isn’t during extinction. When the reward returns, the animal reverts to the previous context, and relearns the task quickly.
The authors’ explanation for this pseudo-forgetting is that classical extinction destroys the prospective association (cue to reward), but not the retrospective one (reward to cue). They predict that they can induce true forgetting in a different kind of extinction trial, where the reward is presented repeatedly without the cue. This should break the retrospective association of the cue with the reward, and animals should revert to their, slower, pre-training rates of acquistion when the reward is presented again. This is indeed what they observe:
I will need to think more about this, and how it plugs into the standard TD learning explanation of dopamine, which seems to be purely prospective.
$$\blacksquare$$
Leave a Reply