Biggs, Felix;
Guedj, Benjamin;
(2021)
Differentiable PAC–Bayes Objectives with Partially Aggregated Neural Networks.
Entropy
, 23
(10)
, Article 1280. 10.3390/e23101280.
Preview |
PDF
entropy-23-01280-v3.pdf - Published Version Download (333kB) | Preview |
Abstract
We make two related contributions motivated by the challenge of training stochastic neural networks, particularly in a PAC–Bayesian setting: (1) we show how averaging over an ensemble of stochastic neural networks enables a new class of partially-aggregated estimators, proving that these lead to unbiased lower-variance output and gradient estimators; (2) we reformulate a PAC–Bayesian bound for signed-output networks to derive in combination with the above a directly optimisable, differentiable objective and a generalisation guarantee, without using a surrogate loss or loosening the bound. We show empirically that this leads to competitive generalisation guarantees and compares favourably to other methods for training such networks. Finally, we note that the above leads to a simpler PAC–Bayesian training scheme for sign-activation networks than previous work.
| Type: | Article |
|---|---|
| Title: | Differentiable PAC–Bayes Objectives with Partially Aggregated Neural Networks |
| Location: | Switzerland |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.3390/e23101280 |
| Publisher version: | http://dx.doi.org/10.3390/e23101280 |
| Language: | English |
| Additional information: | © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
| Keywords: | statistical learning theory; PAC–Bayes theory; deep learning |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10182539 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
