Kiela, Douwe;
Bartolo, Max;
Nie, Yixin;
Kaushik, Divyansh;
Geiger, Atticus;
Wu, Zhengxuan;
Vidgen, Bertie;
... Williams, Adina; + view all
(2021)
Dynabench: Rethinking Benchmarking in NLP.
In:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
(pp. pp. 4110-4124).
Association for Computational Linguistics: Online.
Preview |
PDF
2021.naacl-main.324.pdf - Published Version Download (480kB) | Preview |
Abstract
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-in-the-loop dataset creation: annotators seek to create examples that a target model will misclassify, but that another person will not. In this paper, we argue that Dynabench addresses a critical need in our community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios. With Dynabench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and informative benchmarks. We report on four initial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic benchmarking as a new standard for the field.
| Type: | Proceedings paper |
|---|---|
| Title: | Dynabench: Rethinking Benchmarking in NLP |
| Event: | NAACL 2021 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://aclanthology.org/2021.naacl-main.324/ |
| Language: | English |
| Additional information: | © 1963–2022 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. |
| Keywords: | cs.CL, cs.CL, cs.AI |
| UCL classification: | UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science UCL > Provost and Vice Provost Offices > UCL BEAMS UCL |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10154326 |
Archive Staff Only
 |
View Item |
