Question and answer test-train overlap in open-domain question answering datasets

Advanced search
Browse by:

Department | Year

UCL Theses | Latest

Deposit your research

Question and answer test-train overlap in open-domain question answering datasets

Lewis, P; Stenetorp, P; Riedel, S; (2021) Question and answer test-train overlap in open-domain question answering datasets. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics: Online. Green open access

Preview

Text
2021.eacl-main.86.pdf - Published Version
Download (229kB) | Preview

Abstract

Ideally Open-Domain Question Answering models should exhibit a number of competencies, ranging from simply memorizing questions seen at training time, to answering novel question formulations with answers seen during training, to generalizing to completely novel questions with novel answers. However, single aggregated test set scores do not show the full picture of what capabilities models truly have. In this work, we perform a detailed study of the test sets of three popular open-domain benchmark datasets with respect to these competencies. We find that 30% of test-set questions have a near-duplicate paraphrase in their corresponding train sets. In addition, we find that 60-70% of answers in the test sets are also present in the train sets. Using these findings, we evaluate a variety of popular open-domain models to obtain greater insight into what extent they can generalize, and what drives their overall performance. We find that all models perform substantially worse on questions that cannot be memorized from train sets, with a mean absolute performance difference of 61% between repeated and non-repeated data. Finally we show that simple nearest-neighbor models outperform a BART closed-book QA model, further highlighting the role that train set memorization plays in these benchmarks.

Type:	Proceedings paper
Title:	Question and answer test-train overlap in open-domain question answering datasets
Event:	EACL 2021
Open access status:	An open access version is available from UCL Discovery
DOI:	10.18653/v1/2021.eacl-main.86
Publisher version:	https://aclanthology.org/2021.eacl-main.86
Language:	English
Additional information:	This is an open access article under the CC BY 4.0 license Attribution 4.0 International (https://creativecommons.org/licenses/by/4.0/)
UCL classification:	UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI:	https://discovery.ucl.ac.uk/id/eprint/10130059