UCL logo

UCL Discovery

UCL home » Library Services » Electronic resources » UCL Discovery

Analytical mean squared error curves for temporal difference learning

Singh, S; Dayan, P; (1998) Analytical mean squared error curves for temporal difference learning. MACH LEARN , 32 (1) 5 - 40.

Full text not available from this repository.

Abstract

We provide analytical expressions governing changes to the bias and variance of the lookup table estimators provided by various Monte Carlo and temporal difference value estimation algorithms with offline updates over trials in absorbing Markov reward processes. We have used these expressions to develop software that serves as an analysis tool: given a complete description of a Markov reward process, it rapidly yields an exact mean-square-error curve, the curve one would get from averaging together sample mean-square-error curves from an infinite number of learning trials on the given problem. We use our analysis tool to illustrate classes of mean-square-error curve behavior in a variety of example reward processes, and we show that although the various temporal difference algorithms are quite sensitive to the choice of step-size and eligibility-trace parameters, there are values of these parameters that make them similarly competent, and generally good.

Type:Article
Title:Analytical mean squared error curves for temporal difference learning
Keywords:reinforcement learning, temporal difference, Monte Carlo, MSE, bias, variance, eligibility trace, Markov reward process, CONVERGENCE, TD(LAMBDA)
UCL classification:UCL > School of Life and Medical Sciences > Faculty of Life Sciences > Gatsby Computational Neuroscience Unit

Archive Staff Only: edit this record