eprintid: 10154325 rev_number: 6 eprint_status: archive userid: 699 dir: disk0/10/15/43/25 datestamp: 2022-08-25 09:16:46 lastmod: 2022-08-25 09:16:46 status_changed: 2022-08-25 09:16:46 type: proceedings_section metadata_visibility: show sword_depositor: 699 creators_name: Sugawara, Saku creators_name: Stenetorp, Pontus creators_name: Aizawa, Akiko title: Benchmarking Machine Reading Comprehension: A Psychological Perspective ispublished: pub divisions: C05 divisions: F48 divisions: B04 divisions: UCL keywords: cs.CL, cs.CL note: © 1963–2022 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. abstract: Machine reading comprehension (MRC) has received considerable attention as a benchmark for natural language understanding. However, the conventional task design of MRC lacks explainability beyond the model interpretation, i.e., reading comprehension by a model cannot be explained in human terms. To this end, this position paper provides a theoretical basis for the design of MRC datasets based on psychology as well as psychometrics, and summarizes it in terms of the prerequisites for benchmarking MRC. We conclude that future datasets should (i) evaluate the capability of the model for constructing a coherent and grounded representation to understand context-dependent situations and (ii) ensure substantive validity by shortcut-proof questions and explanation as a part of the task design. date: 2021-04-23 date_type: published publisher: Association for Computational Linguistics official_url: https://aclanthology.org/2021.eacl-main.137/ oa_status: green full_text_type: pub language: eng primo: open primo_central: open_green verified: verified_manual elements_id: 1958161 lyricists_name: Otias Protenest, Pontus lyricists_id: PLEPS00 actors_name: Flynn, Bernadette actors_id: BFFLY94 actors_role: owner full_text_status: public pres_type: paper place_of_pub: Online pagerange: 1592-1612 event_title: EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics book_title: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics citation: Sugawara, Saku; Stenetorp, Pontus; Aizawa, Akiko; (2021) Benchmarking Machine Reading Comprehension: A Psychological Perspective. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. (pp. pp. 1592-1612). Association for Computational Linguistics: Online. Green open access document_url: https://discovery.ucl.ac.uk/id/eprint/10154325/1/2021.eacl-main.137.pdf