UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Gaussian mean field regularizes by limiting learned information

Kunze, J; Kirsch, L; Ritter, H; Barber, D; (2019) Gaussian mean field regularizes by limiting learned information. Entropy , 21 (8) , Article 758. 10.3390/e21080758. Green open access

[thumbnail of entropy-21-00758-v2.pdf]
Preview
Text
entropy-21-00758-v2.pdf - Published Version

Download (814kB) | Preview

Abstract

Variational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is scaled by a constant. Our experiments suggest that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks.

Type: Article
Title: Gaussian mean field regularizes by limiting learned information
Open access status: An open access version is available from UCL Discovery
DOI: 10.3390/e21080758
Publisher version: https://doi.org/10.3390/e21080758
Language: English
Additional information: This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/
Keywords: Information theory; variational inference; machine learning
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10080069
Downloads since deposit
74Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item