UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models

Pantazis, Omiros; Brostow, Gabriel; Jones, Katherine; Mac Aodha, Oisin; (2022) SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models. In: Proceedings of The 33rd British Machine Vision Conference. The British Machine Vision Association (BMVA): London, UK. Green open access

[thumbnail of 0580.pdf]
Preview
Text
0580.pdf - Other

Download (1MB) | Preview

Abstract

Vision-language models such as CLIP are pretrained on large volumes of internet sourced image and text pairs, and have been shown to sometimes exhibit impressive zero- and low-shot image classification performance. However, due to their size, fine-tuning these models on new datasets can be prohibitively expensive, both in terms of the supervision and compute required. To combat this, a series of light-weight adaptation methods have been proposed to efficiently adapt such models when limited supervision is available. In this work, we show that while effective on internet-style datasets, even those remedies under-deliver on classification tasks with images that differ significantly from those commonly found online. To address this issue, we present a new approach called SVL-Adapter that combines the complementary strengths of both vision-language pretraining and self-supervised representation learning. We report an average classification accuracy improvement of 10% in the low-shot setting when compared to existing methods, on a set of challenging visual classification tasks. Further, we present a fully automatic way of selecting an important blending hyperparameter for our model that does not require any held-out labeled validation data. Code for our project is available here: https://github.com/omipan/svl_adapter.

Type: Proceedings paper
Title: SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models
Event: BMVC 2022: The 33rd British Machine Vision Conference
Dates: 21 Nov 2022 - 24 Nov 2022
Open access status: An open access version is available from UCL Discovery
Publisher version: https://bmvc2022.mpi-inf.mpg.de/580/
Language: English
Additional information: This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Div of Biosciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Div of Biosciences > Genetics, Evolution and Environment
URI: https://discovery.ucl.ac.uk/id/eprint/10165606
Downloads since deposit
9Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item