UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Practical Gauss-Newton Optimisation for Deep Learning

Botev, A; Ritter, J; Barber, D; (2017) Practical Gauss-Newton Optimisation for Deep Learning. In: Precup, D and Teh, YW, (eds.) Proceedings of the 34th International Conference on Machine Learning. (pp. pp. 557-565). Proceedings of Machine Learning Research: Sydney, Australia. Green open access

[thumbnail of Barber_botev17a.pdf]
Preview
Text
Barber_botev17a.pdf - Published Version

Download (968kB) | Preview

Abstract

We present an efficient block-diagonal approximation to the Gauss-Newton matrix for feedforward neural networks. Our resulting algorithm is competitive against state-of-the-art first-order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for which hyperparameter tuning of the optimisation parameters is often a laborious process, our approach can provide good performance even when used with default settings. A side result of our work is that for piecewise linear transfer functions, the network objective function can have no differentiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation.

Type: Proceedings paper
Title: Practical Gauss-Newton Optimisation for Deep Learning
Event: 34th International Conference on Machine Learning
Open access status: An open access version is available from UCL Discovery
Publisher version: http://proceedings.mlr.press/v70/botev17a.html
Language: English
Additional information: This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10038402
Downloads since deposit
267Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item