Moran, R;
Dayan, P;
Dolan, RJ;
(2021)
Human subjects exploit a cognitive map for credit assignment.
Proceedings of the National Academy of Sciences of the United States of America (PNAS)
, 118
(4)
, Article e2016884118. 10.1073/pnas.2016884118.
Preview |
Text
e2016884118.full.pdf - Published Version Download (1MB) | Preview |
Abstract
An influential reinforcement learning framework proposes that behavior is jointly governed by model-free (MF) and model-based (MB) controllers. The former learns the values of actions directly from past encounters, and the latter exploits a cognitive map of the task to calculate these prospectively. Considerable attention has been paid to how these systems interact during choice, but how and whether knowledge of a cognitive map contributes to the way MF and MB controllers assign credit (i.e., to how they revaluate actions and states following the receipt of an outcome) remains underexplored. Here, we examine such sophisticated credit assignment using a dual-outcome bandit task. We provide evidence that knowledge of a cognitive map influences credit assignment in both MF and MB systems, mediating subtly different aspects of apparent relevance. Specifically, we show MF credit assignment is enhanced for those rewards that are related to a choice, and this contrasted with choice-unrelated rewards that reinforced subsequent choices negatively. This modulation is only possible based on knowledge of task structure. On the other hand, MB credit assignment was boosted for outcomes that impacted on differences in values between offered bandits. We consider mechanistic accounts and the normative status of these findings. We suggest the findings extend the scope and sophistication of cognitive map-based credit assignment during reinforcement learning, with implications for understanding behavioral control.
| Type: | Article |
|---|---|
| Title: | Human subjects exploit a cognitive map for credit assignment |
| Location: | United States |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1073/pnas.2016884118 |
| Publisher version: | http://doi.org/10.1073/pnas.2016884118 |
| Language: | English |
| Additional information: | This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ |
| Keywords: | cognitive maps, decision making, model-based, model-free, reinforcement learning |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > UCL Queen Square Institute of Neurology UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > UCL Queen Square Institute of Neurology > Imaging Neuroscience |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10119939 |
Archive Staff Only
 |
View Item |
