UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason

Inoue, N; Stenetorp, P; Inui, K; (2020) R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. (pp. pp. 6740-6750). Association for Computational Linguistics: Seattle, WA, USA. Green open access

[img]
Preview
Text
Saito Stenetorp_1910.04601.pdf - Accepted version

Download (1MB) | Preview

Abstract

Recent studies have revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. This prevents the community from reliably measuring the progress of RC systems. To address this issue, we introduce R4C, a new task for evaluating RC systems’ internal reasoning. R4C requires giving not only answers but also derivations: explanations that justify predicted answers. We present a reliable, crowdsourced framework for scalably annotating RC datasets with derivations. We create and publicly release the R4C dataset, the first, quality-assured dataset consisting of 4.6k questions, each of which is annotated with 3 reference derivations (i.e. 13.8k derivations). Experiments show that our automatic evaluation metrics using multiple reference derivations are reliable, and that R4C assesses different skills from an existing benchmark.

Type: Proceedings paper
Title: R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason
Event: 58th Annual Meeting of the Association for Computational Linguistics
Dates: July 2020 - July 2020
Open access status: An open access version is available from UCL Discovery
DOI: 10.18653/v1/2020.acl-main.602
Publisher version: https://www.aclweb.org/anthology/2020.acl-main.602
Language: English
Additional information: This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10109731
Downloads since deposit
11Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item