UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Forward and Reverse Gradient-Based Hyperparameter Optimization

Franceschi, L; Donini, M; Frasconi, P; Pontil, M; (2017) Forward and Reverse Gradient-Based Hyperparameter Optimization. In: Precup, D and Teh, YW, (eds.) Proceedings of the 34th International Conference on Machine Learning 2017. (pp. pp. 1165-1173). JMLR: Sydney, Australia. Green open access

[img]
Preview
Text
Franceschi_Forward_Reverse_Gradient-Based.pdf - Published version

Download (414kB) | Preview

Abstract

We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have different trade-offs in terms of running time and space requirements. Our formulation of the reverse-mode procedure is linked to previous work by Maclaurin et al. (2015) but does not require reversible dynamics. The forward-mode procedure is suitable for real-time hyperparameter updates, which may significantly speed up hyperparameter optimization on large datasets. We present experiments on data cleaning and on learning task interactions. We also present one large-scale experiment where the use of previous gradient-based methods would be prohibitive.

Type: Proceedings paper
Title: Forward and Reverse Gradient-Based Hyperparameter Optimization
Event: 34th International Conference on Machine Learning, 6-11 August 2017, Sydney, Australia
ISBN-13: 9781510855144
Open access status: An open access version is available from UCL Discovery
Publisher version: http://proceedings.mlr.press/v70/franceschi17a.htm...
Language: English
Additional information: This is the published version of record. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/1568472
Downloads since deposit
11Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item