TY  - GEN
N1  - © 1963?2022 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License.
EP  - 1612
Y1  - 2021/04/23/
AV  - public
SP  - 1592
TI  - Benchmarking Machine Reading Comprehension: A Psychological Perspective
KW  - cs.CL
KW  -  cs.CL
A1  - Sugawara, Saku
A1  - Stenetorp, Pontus
A1  - Aizawa, Akiko
CY  - Online
UR  - https://aclanthology.org/2021.eacl-main.137/
PB  - Association for Computational Linguistics
ID  - discovery10154325
N2  - Machine reading comprehension (MRC) has received considerable attention as a benchmark for natural language understanding. However, the conventional task design of MRC lacks explainability beyond the model interpretation, i.e., reading comprehension by a model cannot be explained in human terms. To this end, this position paper provides a theoretical basis for the design of MRC datasets based on psychology as well as psychometrics, and summarizes it in terms of the prerequisites for benchmarking MRC. We conclude that future datasets should (i) evaluate the capability of the model for constructing a coherent and grounded representation to understand context-dependent situations and (ii) ensure substantive validity by shortcut-proof questions and explanation as a part of the task design.
ER  -