Forward and Reverse Gradient-Based Hyperparameter Optimization

Advanced search
Browse by:

Department | Year

UCL Theses | Latest

Deposit your research

Forward and Reverse Gradient-Based Hyperparameter Optimization

Franceschi, L; Donini, M; Frasconi, P; Pontil, M; (2017) Forward and Reverse Gradient-Based Hyperparameter Optimization. In: Precup, D and Teh, YW, (eds.) Proceedings of the 34th International Conference on Machine Learning 2017. (pp. pp. 1165-1173). JMLR: Sydney, Australia. Green open access

[thumbnail of Franceschi_Forward_Reverse_Gradient-Based.pdf]

Preview

Text
Franceschi_Forward_Reverse_Gradient-Based.pdf - Published Version
Download (414kB) | Preview

Abstract

We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have different trade-offs in terms of running time and space requirements. Our formulation of the reverse-mode procedure is linked to previous work by Maclaurin et al. (2015) but does not require reversible dynamics. The forward-mode procedure is suitable for real-time hyperparameter updates, which may significantly speed up hyperparameter optimization on large datasets. We present experiments on data cleaning and on learning task interactions. We also present one large-scale experiment where the use of previous gradient-based methods would be prohibitive.

Type:	Proceedings paper
Title:	Forward and Reverse Gradient-Based Hyperparameter Optimization
Event:	34th International Conference on Machine Learning, 6-11 August 2017, Sydney, Australia
ISBN-13:	9781510855144
Open access status:	An open access version is available from UCL Discovery
Publisher version:	http://proceedings.mlr.press/v70/franceschi17a.htm...
Language:	English
Additional information:	This is the published version of record. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification:	UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI:	https://discovery.ucl.ac.uk/id/eprint/1568472