UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Reward Maximization Through Discrete Active Inference

Da Costa, Lancelot; Sajid, Noor; Parr, Thomas; Friston, Karl; Smith, Ryan; (2023) Reward Maximization Through Discrete Active Inference. Neural Computation pp. 1-46. 10.1162/neco_a_01574. (In press). Green open access

[thumbnail of neco_a_01574.pdf]
Preview
Text
neco_a_01574.pdf - Published Version

Download (1MB) | Preview

Abstract

Active inference is a probabilistic framework for modeling the behavior of biological and artificial agents, which derives from the principle of minimizing free energy. In recent years, this framework has been applied successfully to a variety of situations where the goal was to maximize reward, often offering comparable and sometimes superior performance to alternative approaches. In this article, we clarify the connection between reward maximization and active inference by demonstrating how and when active inference agents execute actions that are optimal for maximizing reward. Precisely, we show the conditions under which active inference produces the optimal solution to the Bellman equation, a formulation that underlies several approaches to model-based reinforcement learning and control. On partially observed Markov decision processes, the standard active inference scheme can produce Bellman optimal actions for planning horizons of 1 but not beyond. In contrast, a recently developed recursive active inference scheme (sophisticated inference) can produce Bellman optimal actions on any finite temporal horizon. We append the analysis with a discussion of the broader relationship between active inference and reinforcement learning.

Type: Article
Title: Reward Maximization Through Discrete Active Inference
Location: United States
Open access status: An open access version is available from UCL Discovery
DOI: 10.1162/neco_a_01574
Publisher version: https://doi.org/10.1162/neco_a_01574
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > UCL Queen Square Institute of Neurology
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Institute of Epidemiology and Health
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > UCL Queen Square Institute of Neurology > Imaging Neuroscience
URI: https://discovery.ucl.ac.uk/id/eprint/10168059
Downloads since deposit
34Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item