UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models

Li, Kaican; Xie, Weiyan; Huang, Yongxiang; Deng, Didan; Hong, Lanqing; Li, Zhenguo; Silva, Ricardo; (2024) Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models. In: Proceedings of NeurIPS 2024: Thirty-Eighth Annual Conference on Neural Information Processing Systems. (pp. pp. 1-33). NeurIPS: San Diego, CA, USA. Green open access

[thumbnail of 4_Dual_Risk_Minimization_Towar.pdf]
Preview
Text
4_Dual_Risk_Minimization_Towar.pdf - Accepted Version

Download (1MB) | Preview

Abstract

Fine-tuning foundation models often compromises their robustness to distribution shifts. To remedy this, most robust fine-tuning methods aim to preserve the pre-trained features. However, not all pre-trained features are robust and those methods are largely indifferent to which ones to preserve. We propose dual risk minimization (DRM), which combines empirical risk minimization with worst-case risk minimization, to better preserve the core features of downstream tasks. In particular, we utilize core-feature descriptions generated by LLMs to induce core-based zero-shot predictions which then serve as proxies to estimate the worst-case risk. DRM balances two crucial aspects of model robustness: expected performance and worst-case performance, establishing a new state of the art on various real-world benchmarks. DRM significantly improves the out-of-distribution performance of CLIP ViT-L/14@336 on ImageNet (75.9 → 77.1), WILDS-iWildCam (47.1 → 51.8), and WILDS-FMoW (50.7 → 53.1); opening up new avenues for robust fine-tuning. Our code is available at https://github.com/vaynexie/DRM.

Type: Proceedings paper
Title: Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models
Event: NeurIPS 2024: Thirty-Eighth Annual Conference on Neural Information Processing Systems
ISBN-13: 9798331314385
Open access status: An open access version is available from UCL Discovery
Publisher version: https://proceedings.neurips.cc/paper_files/paper/2...
Language: English
Additional information: This version is the version of record. For information on re-use, please refer to the publisher's terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences > Dept of Statistical Science
URI: https://discovery.ucl.ac.uk/id/eprint/10199062
Downloads since deposit
15Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item