Dorrell, William;
Hsu, Kyle;
Whittington, James;
Wu, Jiajun;
Finn, Chelsea;
(2023)
Disentanglement via Latent Quantization.
In:
Proceedings - Advances in Neural Information Processing Systems 37 (NeurIPS 2023).
NeurIPS: New Orleans, USA.
Preview |
Text
3834_disentanglement_via_latent_qua.pdf - Published Version Download (24MB) | Preview |
Abstract
In disentangled representation learning, a model is asked to tease apart a dataset’s underlying sources of variation and represent them independently of one another. Since the model is provided with no ground truth information about these sources, inductive biases take a paramount role in enabling disentanglement. In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. Concretely, we do this by (i) quantizing the latent space into discrete code vectors with a separate learnable scalar codebook per dimension and (ii) applying strong model regularization via an unusually high weight decay. Intuitively, the latent space design forces the encoder to combinatorially construct codes from a small number of distinct scalar values, which in turn enables the decoder to assign a consistent meaning to each value. Regularization then serves to drive the model towards this parsimonious strategy. We demonstrate the broad applicability of this approach by adding it to both basic data-reconstructing (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models. For reliable evaluation, we also propose InfoMEC, a new set of metrics for disentanglement that is cohesively grounded in information theory and fixes well-established shortcomings in previous metrics. Together with regularization, latent quantization dramatically improves the modularity and explicitness of learned representations on a representative suite of benchmark datasets. In particular, our quantized-latent autoencoder (QLAE) consistently outperforms strong methods from prior work in these key disentanglement properties without compromising data reconstruction.
Type: | Proceedings paper |
---|---|
Title: | Disentanglement via Latent Quantization |
Event: | 37th Conference on Neural Information Processing Systems (NeurIPS 2023) |
Open access status: | An open access version is available from UCL Discovery |
Publisher version: | https://papers.nips.cc/paper_files/paper/2023/file... |
Language: | English |
Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Gatsby Computational Neurosci Unit |
URI: | https://discovery.ucl.ac.uk/id/eprint/10181681 |
Archive Staff Only
View Item |