Shi, Y;
Paige, B;
Torr, PHS;
Siddharth, N;
(2021)
Relating by Contrasting: A Data-Efficient Framework for Multimodal DGMs.
In:
Proceedings of the 9th International Conference on Learning Representations (ICLR 2021).
(pp. pp. 1-25).
ICLR
Preview |
Text
1601_relating_by_contrasting_a_data.pdf - Published Version Download (6MB) | Preview |
Abstract
Multimodal learning for generative models often refers to the learning of abstract concepts from the commonality of information in multiple modalities, such as vision and language. While it has proven effective for learning generalisable representations, the training of such models often requires a large amount of “related” multimodal data that shares commonality, which can be expensive to come by. To mitigate this, we develop a novel contrastive framework for generative model learning, allowing us to train the model not just by the commonality between modalities, but by the distinction between “related” and “unrelated” multimodal data. We show in experiments that our method enables data-efficient multimodal learning on challenging datasets for various multimodal variational autoencoder (VAE) models. We also show that under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.
| Type: | Proceedings paper |
|---|---|
| Title: | Relating by Contrasting: A Data-Efficient Framework for Multimodal DGMs |
| Event: | The Ninth International Conference on Learning Representations (Virtual Only) |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://iclr.cc/Conferences/2021 |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher's terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10171347 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
