Kirstain, Y;
Lewis, P;
Riedel, S;
Levy, O;
(2022)
A Few More Examples May Be Worth Billions of Parameters.
In:
Findings of the Association for Computational Linguistics: EMNLP 2022.
(pp. pp. 1017-1029).
ACL Anthology: Abu Dhabi, United Arab Emirates.
Preview |
Text
2022.findings-emnlp.72.pdf - Published Version Download (766kB) | Preview |
Abstract
We investigate the dynamics of increasing the number of model parameters versus the number of labeled examples across a wide variety of tasks. Our exploration reveals that while scaling parameters consistently yields performance improvements, the contribution of additional examples highly depends on the task's format. Specifically, in open question answering tasks, enlarging the training set does not improve performance. In contrast, classification, extractive question answering, and multiple choice tasks benefit so much from additional examples that collecting a few hundred examples is often “worth” billions of parameters. We hypothesize that unlike open question answering, which involves recalling specific information, solving strategies for tasks with a more restricted output space transfer across examples, and can therefore be learned with small amounts of labeled data.
| Type: | Proceedings paper |
|---|---|
| Title: | A Few More Examples May Be Worth Billions of Parameters |
| Event: | Findings of the Association for Computational Linguistics: EMNLP 2022 |
| ISBN-13: | 978-1-955917-25-4 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://aclanthology.org/2022.findings-emnlp.72/ |
| Language: | English |
| Additional information: | ©2022 Association for Computational Linguistics. Licensed on a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10167454 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
