Inoue, N; Stenetorp, P; Inui, K; (2020) R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. (pp. pp. 6740-6750). Association for Computational Linguistics: Seattle, WA, USA.

Preview

Text Saito Stenetorp_1910.04601.pdf - Accepted Version Download (1MB) | Preview

Abstract

Recent studies have revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. This prevents the community from reliably measuring the progress of RC systems. To address this issue, we introduce R4C, a new task for evaluating RC systems’ internal reasoning. R4C requires giving not only answers but also derivations: explanations that justify predicted answers. We present a reliable, crowdsourced framework for scalably annotating RC datasets with derivations. We create and publicly release the R4C dataset, the first, quality-assured dataset consisting of 4.6k questions, each of which is annotated with 3 reference derivations (i.e. 13.8k derivations). Experiments show that our automatic evaluation metrics using multiple reference derivations are reliable, and that R4C assesses different skills from an existing benchmark.

Download activity - last month

Export as CSVJSONXML

Download activity - last 12 months

Export as CSVJSONXML

Downloads by country - last 12 months

Export as CSVJSONXML

Archive Staff Only

View Item