UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization

Xie, J; Ma, Z; Lei, J; Zhang, G; Xue, J-H; Tan, Z-H; Guo, J; (2022) Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44 (9) pp. 4605-4625. 10.1109/tpami.2021.3083089. Green open access

[thumbnail of TPAMI-JiyangXie-final.pdf]
Preview
Text
TPAMI-JiyangXie-final.pdf - Accepted Version

Download (5MB) | Preview

Abstract

Due to lack of data, overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs). We propose advanced dropout, a model-free methodology, to mitigate overfitting and improve the performance of DNNs. The advanced dropout technique applies a model-free and easily implemented distribution with parametric prior, and adaptively adjusts dropout rate. Specifically, the distribution parameters are optimized by stochastic gradient variational Bayes in order to carry out an end-to-end training. We evaluate the effectiveness of the advanced dropout against nine dropout techniques on seven computer vision datasets (five small-scale datasets and two large-scale datasets) with various base models. The advanced dropout outperforms all the referred techniques on all the datasets.We further compare the effectiveness ratios and find that advanced dropout achieves the highest one on most cases. Next, we conduct a set of analysis of dropout rate characteristics, including convergence of the adaptive dropout rate, the learned distributions of dropout masks, and a comparison with dropout rate generation without an explicit distribution. In addition, the ability of overfitting prevention is evaluated and confirmed. Finally, we extend the application of the advanced dropout to uncertainty inference, network pruning, text classification, and regression. The proposed advanced dropout is also superior to the corresponding referred methods.

Type: Article
Title: Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization
Open access status: An open access version is available from UCL Discovery
DOI: 10.1109/tpami.2021.3083089
Publisher version: http://doi.org/10.1109/TPAMI.2021.3083089
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions.
Keywords: Deep neural network, dropout, model-free distribution, Bayesian approximation, stochastic gradient variational Bayes
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences > Dept of Statistical Science
URI: https://discovery.ucl.ac.uk/id/eprint/10128664
Downloads since deposit
115Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item