Yong, ZX;
Schoelkopf, H;
Muennighoff, N;
Aji, AF;
Adelani, DI;
Almubarak, K;
Bari, MS;
... Nikoulina, V; + view all
(2023)
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting.
In:
Proceedings of the Annual Meeting of the Association for Computational Linguistics.
(pp. pp. 11682-11703).
ACL: Toronto, Canada.
Preview |
Text
2023.acl-long.653.pdf - Published Version Download (971kB) | Preview |
Abstract
The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. To extend the benefits of BLOOM to other languages without incurring prohibitively large costs, it is desirable to adapt BLOOM to new languages not seen during pretraining. In this work, we apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. We find language adaptation to be effective at improving zero-shot performance in new languages. Surprisingly, we find that adapter-based finetuning is more effective than continued pretraining for large models. In addition, we discover that prompting performance is not significantly affected by language specifics, such as the writing system. It is primarily determined by the size of the language adaptation data. We also add new languages to BLOOMZ, which is a multitask finetuned version of BLOOM capable of following task instructions zero-shot. We find including a new language in the multitask fine-tuning mixture to be the most effective method to teach BLOOMZ a new language. We conclude that with sufficient training data language adaptation can generalize well to diverse languages. Our code is available at https://github.com/bigscience-workshop/multilingual-modeling.
| Type: | Proceedings paper |
|---|---|
| Title: | BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting |
| Event: | 61st Annual Meeting of the Association for Computational Linguistics |
| ISBN-13: | 9781959429722 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | http://dx.doi.org/10.18653/v1/2023.acl-long.653 |
| Language: | English |
| Additional information: | ACL materials are Copyright © 1963–2023 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10181861 |
Archive Staff Only
 |
View Item |
