UCL logo

UCL Discovery

UCL home » Library Services » Electronic resources » UCL Discovery

Indexability of bandit problems with response delays

Caro, F; Yoo, OS; (2010) Indexability of bandit problems with response delays. Probability in the Engineering and Informational Sciences , 24 (3) pp. 349-374. 10.1017/S0269964810000021.

[img] Text
Access restricted to UCL open access staff

Download (198kB)


This article considers an important class of discrete time restless bandits, given by the discounted multiarmed bandit problems with response delays. The delays in each period are independent random variables, in which the delayed responses do not cross over. For a bandit arm in this class, we use a coupling argument to show that in each state there is a unique subsidy that equates the pulling and nonpulling actions (i.e., the bandit satisfies the indexibility criterion introduced by Whittle (1988). The result allows for infinite or finite horizon and holds for arbitrary delay lengths and infinite state spaces. We compute the resulting marginal productivity indexes (MPI) for the Beta-Bernoulli Bayesian learning model, formulate and compute a tractable upper bound, and compare the suboptimality gap of the MPI policy to those of other heuristics derived from different closed-form indexes. The MPI policy performs near optimally and provides a theoretical justification for the use of the other heuristics.

Type: Article
Title: Indexability of bandit problems with response delays
DOI: 10.1017/S0269964810000021
Publisher version: http://dx.doi.org/10.1017/S0269964810000021
Language: English
Additional information: Copyright © Cambridge University Press 2010.
UCL classification: UCL > School of BEAMS
UCL > School of BEAMS > Faculty of Engineering Science
UCL > School of BEAMS > Faculty of Engineering Science > Management Science and Innovation
URI: http://discovery.ucl.ac.uk/id/eprint/1315595
Downloads since deposit
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item