Zhou, Wenda; Veitch, Victor; Austern, Morgane; Adams, Ryan P; Orbanz, Peter; (2019) Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach. In: ICLR 2019 International Conference on Learning Representations. ICLR: New Orleans, LA, United States.

Preview

Text 557_non_vacuous_generalization_bou.pdf - Published Version Download (302kB) | Preview

Abstract

Modern neural networks are highly overparameterized, with capacity to substantially overfit to training data. Nevertheless, these networks often generalize well in practice. It has also been observed that trained networks can often be “compressed” to much smaller representations. The purpose of this paper is to connect these two empirical observations. Our main technical result is a generalization bound for compressed networks based on the compressed size that, combined with off-theshelf compression algorithms, leads to state-of-the-art generalization guarantees. In particular, we provide the first non-vacuous generalization guarantees for realistic architectures applied to the ImageNet classification problem. Additionally, we show that compressibility of models that tend to overfit is limited. Empirical results show that an increase in overfitting increases the number of bits required to describe a trained network.

Download activity - last month

Export as JSONXMLCSV

Download activity - last 12 months

Export as JSONXMLCSV

Downloads by country - last 12 months

Export as JSONXMLCSV

Archive Staff Only

View Item