UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

A State Representation for Diminishing Rewards

Moskovitz, Theodore; Hromadka, Samo; Touati, Ahmed; Borsa, Diana; Sahani, Maneesh; (2023) A State Representation for Diminishing Rewards. In: Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023). NeurIPS (In press).

[thumbnail of lambda_representation (3).pdf] Text
lambda_representation (3).pdf - Accepted Version
Access restricted to UCL open access staff until 23 April 2024.

Download (1MB)

Abstract

A common setting in multitask reinforcement learning (RL) demands that an agent rapidly adapt to various stationary reward functions randomly sampled from a fixed distribution. In such situations, the successor representation (SR) is a popular framework which supports rapid policy evaluation by decoupling a policy's expected discounted, cumulative state occupancies from a specific reward function. However, in the natural world, sequential tasks are rarely independent, and instead reflect shifting priorities based on the availability and subjective perception of rewarding stimuli. Reflecting this disjunction, in this paper we study the phenomenon of diminishing marginal utility and introduce a novel state representation, the $\lambda$ representation ($\lambda$R) which, surprisingly, is required for policy evaluation in this setting and which generalizes the SR as well as several other state representations from the literature. We establish the $\lambda$R's formal properties and examine its normative advantages in the context of machine learning, as well as its usefulness for studying natural behaviors, particularly foraging.

Type: Proceedings paper
Title: A State Representation for Diminishing Rewards
Event: Thirty-seventh Conference on Neural Information Processing Systems
Publisher version: https://neurips.cc/virtual/2023/poster/72705
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Gatsby Computational Neurosci Unit
URI: https://discovery.ucl.ac.uk/id/eprint/10179331
Downloads since deposit
1Download
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item