UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Soft Dropout And Its Variational Bayes Approximation

Xie, J; Ma, Z; Zhang, G; Xue, J-H; Tan, Z-H; Guo, J; (2019) Soft Dropout And Its Variational Bayes Approximation. In: Proceedings of the 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE Green open access

[img]
Preview
Text
MLSP-softdropout-camera_ready-checked.pdf - Accepted version

Download (403kB) | Preview

Abstract

Soft dropout, a generalization of standard “hard” dropout, is introduced to regularize the parameters in neural networks and prevent overfitting. We replace the “hard” dropout mask following a Bernoulli distribution with the “soft” mask following a beta distribution to drop the hidden nodes in different levels. The soft dropout method can introduce continuous mask coefficients in the interval of [0, 1], rather than only zero and one. Meanwhile, in order to implement the adaptive dropout rate via adaptive distribution parameters, we respectively utilize the half-Gaussian distributed and the half-Laplace distributed variables to approximate the beta distributed masks and apply a variation of variational Bayes optimization called stochastic gradient variational Bayes (SGVB) algorithm to optimize the distribution parameters. In the experiments, compared with the standard soft dropout with fixed dropout rate, the adaptive soft dropout method generally improves the performance. In addition, the proposed soft dropout and its adaptive versions achieve performance improvement compared with the referred methods on both image classification and regression tasks.

Type: Proceedings paper
Title: Soft Dropout And Its Variational Bayes Approximation
Event: 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP)
Location: Pittsburgh (PA), USA
Dates: 13th-16th October 2019
ISBN-13: 978-1-7281-0824-7
Open access status: An open access version is available from UCL Discovery
DOI: 10.1109/mlsp.2019.8918818
Publisher version: https://doi.org/10.1109/MLSP.2019.8918818
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Neural networks, soft dropout, beta distribution, Bayesian approximation
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences > Dept of Statistical Science
URI: https://discovery.ucl.ac.uk/id/eprint/10089423
Downloads since deposit
5Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item