UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting

Ritter, H; Botev, A; Barber, D; (2018) Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting. In: Bengio, S and Wallach, H and Larochelle, H and Grauman, K and CesaBianchi, N and Garnett, R, (eds.) Proceedings of the 32nd Conference on Neural Information Processing Systems (NIPS 2018). Neural Information Processing Systems (NIPS): Montréal, Canada. Green open access

[thumbnail of osla.pdf]
Preview
Text
osla.pdf - Published Version

Download (520kB) | Preview

Abstract

We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the weights. The Laplace approximation requires calculating the Hessian around a mode, which is typically intractable for modern architectures. In order to make our method scalable, we leverage recent block-diagonal Kronecker factored approximations to the curvature. Our algorithm achieves over 90% test accuracy across a sequence of 50 instantiations of the permuted MNIST dataset, substantially outperforming related methods for overcoming catastrophic forgetting.

Type: Proceedings paper
Title: Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting
Event: NIPS 2018
Location: Montreal, CANADA
Dates: 02 December 2018 - 08 December 2018
Open access status: An open access version is available from UCL Discovery
Publisher version: https://papers.nips.cc/book/advances-in-neural-inf...
Language: English
Additional information: This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Science & Technology, Technology, Computer Science, Artificial Intelligence, Computer Science
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10076100
Downloads since deposit
44Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item