TY  - JOUR
SN  - 1099-4300
EP  - 14
IS  - 10
KW  - statistical learning theory; PAC?Bayes theory; deep learning
A1  - Biggs, Felix
A1  - Guedj, Benjamin
AV  - public
TI  - Differentiable PAC?Bayes Objectives with Partially Aggregated Neural Networks
PB  - MDPI AG
UR  - http://dx.doi.org/10.3390/e23101280
ID  - discovery10182539
VL  - 23
JF  - Entropy
N1  - © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
N2  - We make two related contributions motivated by the challenge of training stochastic neural networks, particularly in a PAC?Bayesian setting: (1) we show how averaging over an ensemble of stochastic neural networks enables a new class of partially-aggregated estimators, proving that these lead to unbiased lower-variance output and gradient estimators; (2) we reformulate a PAC?Bayesian bound for signed-output networks to derive in combination with the above a directly optimisable, differentiable objective and a generalisation guarantee, without using a surrogate loss or loosening the bound. We show empirically that this leads to competitive generalisation guarantees and compares favourably to other methods for training such networks. Finally, we note that the above leads to a simpler PAC?Bayesian training scheme for sign-activation networks than previous work.
Y1  - 2021/09/29/
ER  -