Ruder, S;
Clark, JH;
Gutkin, A;
Kale, M;
Ma, M;
Nicosia, M;
Rijhwani, S;
... Talukdar, P; + view all
(2023)
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages.
In: Augenstein, I and Vlachos, A, (eds.)
Findings of the Association for Computational Linguistics: EMNLP 2023.
(pp. pp. 1856-1884).
Association for Computational Linguistics: Dubrovnik, Croatia.
Preview |
Text
2023.findings-emnlp.125.pdf - Published Version Download (1MB) | Preview |
Abstract
Data scarcity is a crucial issue for the development of highly multilingual NLP systems. Yet for many under-represented languages (ULs)-languages for which NLP research is particularly far behind in meeting user needs-it is feasible to annotate small amounts of data. Motivated by this, we propose XTREME-UP, a benchmark defined by: its focus on the scarce-data scenario rather than zero-shot; its focus on user-centric tasks-tasks with broad adoption by speakers of high-resource languages; and its focus on under-represented languages where this scarce-data scenario is most realistic. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks that are of general utility. We create new datasets for OCR, autocomplete, question answering, semantic parsing, and transliteration, and build on and refine existing datasets for other tasks. XTREME-UP provides a methodology for evaluating many modeling scenarios including text-only, multi-modal (vision, audio, and text), supervised parameter tuning, and in-context learning. We evaluate commonly used models on the benchmark.
| Type: | Proceedings paper |
|---|---|
| Title: | XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages |
| ISBN-13: | 9798891760615 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://aclanthology.org/2023.findings-emnlp.125/ |
| Language: | English |
| Additional information: | ACL materials are Copyright © 1963–2024 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10186778 |
Archive Staff Only
 |
View Item |
