UCL logo

UCL Discovery

UCL home » Library Services » Electronic resources » UCL Discovery

Prospective and retrospective temporal difference learning

Dayan, P; (2009) Prospective and retrospective temporal difference learning. NETWORK-COMP NEURAL , 20 (1) 32 - 46. 10.1080/09548980902759086.

Full text not available from this repository.

Abstract

A striking recent finding is that monkeys behave maladaptively in a class of tasks in which they know that reward is going to be systematically delayed. This may be explained by a malign Pavlovian influence arising from states with low predicted values. However, by very carefully analyzing behavioral data from such tasks, La Camera and Richmond (2008) observed the additional important characteristic that subjects perform differently on states in the task that are at equal distances from the future reward, depending on what has happened in the recent past. The authors pointed out that this violates the definition of state value in the standard reinforcement learning models that are ubiquitous as accounts of operant and classical conditioned behavior; they suggested and analyzed an alternative temporal difference (TD) model in which past and future are melded. Here, we show that, in fact, a standard TD model can actually exhibit the same behavior, and that this avoids deleterious consequences for choice. At the heart of the model is the average reward per step, which acts as a baseline for measuring immediate rewards. Relatively subtle changes to this baseline occasioned by the past can markedly influence predictions and thus behavior.

Type:Article
Title:Prospective and retrospective temporal difference learning
DOI:10.1080/09548980902759086
Keywords:Emotional processing, reinforcement learning, PAVLOVIAN-INSTRUMENTAL TRANSFER, NUCLEUS-ACCUMBENS DOPAMINE, GOAL-DIRECTED BEHAVIOR, REWARD SCHEDULES, NEURONAL SIGNALS, DECISION-MAKING, ANTERIOR CINGULATE, RHINAL CORTEX, FOOD-SEEKING, REINFORCEMENT
UCL classification:UCL > School of Life and Medical Sciences > Faculty of Life Sciences > Gatsby Computational Neuroscience Unit

Archive Staff Only: edit this record