Mansbridge, A;
Fierimonte, R;
Feige, I;
Barber, D;
(2019)
Improving latent variable descriptiveness by modelling rather than ad-hoc factors.
Machine Learning
, 108
(8-9)
pp. 1601-1611.
10.1007/s10994-019-05830-1.
Preview |
Text
ImprovingLatentVariableDescriptiveness.pdf - Published Version Download (343kB) | Preview |
Abstract
Powerful generative models, particularly in natural language modelling, are commonly trained by maximizing a variational lower bound on the data log likelihood. These models often suffer from poor use of their latent variable, with ad-hoc annealing factors used to encourage retention of information in the latent variable. We discuss an alternative and general approach to latent variable modelling, based on an objective that encourages a perfect reconstruction by tying a stochastic autoencoder with a variational autoencoder (VAE). This ensures by design that the latent variable captures information about the observations, whilst retaining the ability to generate well. Interestingly, although our model is fundamentally different to a VAE, the lower bound attained is identical to the standard VAE bound but with the addition of a simple pre-factor; thus, providing a formal interpretation of the commonly used, ad-hoc pre-factors in training VAEs.
| Type: | Article |
|---|---|
| Title: | Improving latent variable descriptiveness by modelling rather than ad-hoc factors |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/s10994-019-05830-1 |
| Publisher version: | https://doi.org/10.1007/s10994-019-05830-1 |
| Language: | English |
| Additional information: | © The Author(s) 2019. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License http://creativecommons.org/ licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. |
| Keywords: | Generative modelling, Latent variable modelling, Variational autoencoders, Variational inference, Natural language processing |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10080050 |
Archive Staff Only
 |
View Item |
