Moskovitz, Ted; Hromadka, Samo; Touati, Ahmed; Borsa, Diana; Sahani, Maneesh; (2023) A State Representation for Diminishing Rewards. In: Proceedings of the Thirty-seventh Annual Conference on Neural Information Processing Systems. (pp. pp. 1-38). NeurIPS: San Diego, CA, USA.

Preview

Text lambda_representation (3).pdf - Accepted Version Download (1MB) | Preview

Abstract

A common setting in multitask reinforcement learning (RL) demands that an agent rapidly adapt to various stationary reward functions randomly sampled from a fixed distribution. In such situations, the successor representation (SR) is a popular framework which supports rapid policy evaluation by decoupling a policy's expected discounted, cumulative state occupancies from a specific reward function. However, in the natural world, sequential tasks are rarely independent, and instead reflect shifting priorities based on the availability and subjective perception of rewarding stimuli. Reflecting this disjunction, in this paper we study the phenomenon of diminishing marginal utility and introduce a novel state representation, the $\lambda$ representation ($\lambda$R) which, surprisingly, is required for policy evaluation in this setting and which generalizes the SR as well as several other state representations from the literature. We establish the $\lambda$R's formal properties and examine its normative advantages in the context of machine learning, as well as its usefulness for studying natural behaviors, particularly foraging.

Download activity - last month

Export as CSVXMLJSON

Download activity - last 12 months

Export as CSVJSONXML

Downloads by country - last 12 months

Export as JSONXMLCSV

Archive Staff Only

View Item