Pantazis, Omiros;
Brostow, Gabriel;
Jones, Katherine;
Mac Aodha, Oisin;
(2022)
SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models.
In:
Proceedings of The 33rd British Machine Vision Conference.
The British Machine Vision Association (BMVA): London, UK.
Preview |
Text
0580.pdf - Other Download (1MB) | Preview |
Abstract
Vision-language models such as CLIP are pretrained on large volumes of internet sourced image and text pairs, and have been shown to sometimes exhibit impressive zero- and low-shot image classification performance. However, due to their size, fine-tuning these models on new datasets can be prohibitively expensive, both in terms of the supervision and compute required. To combat this, a series of light-weight adaptation methods have been proposed to efficiently adapt such models when limited supervision is available. In this work, we show that while effective on internet-style datasets, even those remedies under-deliver on classification tasks with images that differ significantly from those commonly found online. To address this issue, we present a new approach called SVL-Adapter that combines the complementary strengths of both vision-language pretraining and self-supervised representation learning. We report an average classification accuracy improvement of 10% in the low-shot setting when compared to existing methods, on a set of challenging visual classification tasks. Further, we present a fully automatic way of selecting an important blending hyperparameter for our model that does not require any held-out labeled validation data. Code for our project is available here: https://github.com/omipan/svl_adapter.
Type: | Proceedings paper |
---|---|
Title: | SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models |
Event: | BMVC 2022: The 33rd British Machine Vision Conference |
Dates: | 21 Nov 2022 - 24 Nov 2022 |
Open access status: | An open access version is available from UCL Discovery |
Publisher version: | https://bmvc2022.mpi-inf.mpg.de/580/ |
Language: | English |
Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Div of Biosciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Div of Biosciences > Genetics, Evolution and Environment |
URI: | https://discovery.ucl.ac.uk/id/eprint/10165606 |
Archive Staff Only
View Item |