UCL logo

UCL Discovery

UCL home » Library Services » Electronic resources » UCL Discovery

Fast gradient-descent methods for temporal-difference learning with linear function approximation

Sutton, RS; Maei, HR; Silver, D; Szepesv ari, C; Wiewiora, E; Precup, D; Bhatnagar, S; (2009) Fast gradient-descent methods for temporal-difference learning with linear function approximation. In: ACM International Conference Proceeding Series.

Full text not available from this repository.

Abstract

Sutton, Szepesv́ari and Maei (2009) recently introduced the first temporal-difference learning algorithm compatible with both linear function approximation and off-policy training, and whose complexity scales only linearly in the size of the function approximator. Although their gradient temporal difference (GTD) algorithm converges reliably, it can be very slow compared to conventional linear TD (on on-policy problems where TD is convergent), calling into question its practical utility. In this paper we introduce two new related algorithms with better convergence rates. The first algorithm, GTD2, is derived and proved convergent just as GTD was, but uses a different objective function and converges significantly faster (but still not as fast as conventional TD). The second new algorithm, linear TD with gradient correction, or TDC, uses the same update rule as conventional TD except for an additional term which is initially zero. In our experiments on small test problems and in a Computer Go application with a million features, the learning rate of this algorithm was comparable to that of conventional TD. This algorithm appears to extend linear TD to off-policy learning with no penalty in performance while only doubling computational requirements. Copyright 2009.

Type:Proceedings paper
Title:Fast gradient-descent methods for temporal-difference learning with linear function approximation
DOI:10.1145/1553374.1553501
UCL classification:UCL > School of BEAMS > Faculty of Engineering Science > Computer Science

Archive Staff Only: edit this record