Logan, Robert L;
Balazevic, Ivana;
Wallace, Eric;
Petroni, Fabio;
Singh, Sameer;
Riedel, Sebastian;
(2022)
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models.
In:
Findings of the Association for Computational Linguistics: ACL 2022.
(pp. pp. 2824-2835).
ASSOC COMPUTATIONAL LINGUISTICS-ACL
Preview |
Text
Riede_Cutting Down on Prompts and Parameters_VoR.pdf - Published Version Download (664kB) | Preview |
Abstract
Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetuning LMs in the few-shot setting can considerably reduce the need for prompt engineering. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive accuracy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each downstream task, we show that this memory overhead can be substantially reduced-finetuning only the bias terms can achieve comparable or better accuracy than standard finetuning while only updating 0.1% of the parameters. All in all, we recommend finetuning LMs for few-shot learning as it is more accurate, has relatively stable performance across different prompts, and can be made nearly as efficient as using frozen LMs.
| Type: | Proceedings paper |
|---|---|
| Title: | Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models |
| Event: | 60th Annual Meeting of the Association-for-Computational-Linguistics (ACL) |
| Location: | Dublin, IRELAND |
| Dates: | 22 May 2022 - 27 May 2022 |
| ISBN-13: | 9781955917254 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://aclanthology.org/2022.findings-acl.222/ |
| Language: | English |
| Additional information: | ACL materials are Copyright © 1963–2023 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. |
| Keywords: | Science & Technology, Social Sciences, Technology, Computer Science, Artificial Intelligence, Computer Science, Theory & Methods, Linguistics, Computer Science |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10166550 |
Archive Staff Only
 |
View Item |
