Miranda, B;
Malalasekera, WMN;
Behrens, TE;
Dayan, P;
Kennerley, SW;
(2020)
Combined model-free and model-sensitive reinforcement learning in non-human primates.
PLOS Computer Biology
, 16
(6)
, Article e1007944. 10.1371/journal.pcbi.1007944.
Preview |
Text
journal.pcbi.1007944.pdf - Published Version Download (1MB) | Preview |
Abstract
Contemporary reinforcement learning (RL) theory suggests that potential choices can be evaluated by strategies that may or may not be sensitive to the computational structure of tasks. A paradigmatic model-free (MF) strategy simply repeats actions that have been rewarded in the past; by contrast, model-sensitive (MS) strategies exploit richer information associated with knowledge of task dynamics. MF and MS strategies should typically be combined, because they have complementary statistical and computational strengths; however, this tradeoff between MF/MS RL has mostly only been demonstrated in humans, often with only modest numbers of trials. We trained rhesus monkeys to perform a two-stage decision task designed to elicit and discriminate the use of MF and MS methods. A descriptive analysis of choice behaviour revealed directly that the structure of the task (of MS importance) and the reward history (of MF and MS importance) significantly influenced both choice and response vigour. A detailed, trial-by-trial computational analysis confirmed that choices were made according to a combination of strategies, with a dominant influence of a particular form of model sensitivity that persisted over weeks of testing. The residuals from this model necessitated development of a new combined RL model which incorporates a particular credit assignment weighting procedure. Finally, response vigor exhibited a subtly different collection of MF and MS influences. These results provide new illumination onto RL behavioural processes in non-human primates.
| Type: | Article |
|---|---|
| Title: | Combined model-free and model-sensitive reinforcement learning in non-human primates |
| Location: | United States |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1371/journal.pcbi.1007944 |
| Publisher version: | http://doi.org/10.1371/journal.pcbi.1007944 |
| Language: | English |
| Additional information: | This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > UCL Queen Square Institute of Neurology UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > UCL Queen Square Institute of Neurology > Clinical and Movement Neurosciences |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10105531 |
Archive Staff Only
 |
View Item |
