eprintid: 10154325
rev_number: 6
eprint_status: archive
userid: 699
dir: disk0/10/15/43/25
datestamp: 2022-08-25 09:16:46
lastmod: 2022-08-25 09:16:46
status_changed: 2022-08-25 09:16:46
type: proceedings_section
metadata_visibility: show
sword_depositor: 699
creators_name: Sugawara, Saku
creators_name: Stenetorp, Pontus
creators_name: Aizawa, Akiko
title: Benchmarking Machine Reading Comprehension: A Psychological Perspective
ispublished: pub
divisions: C05
divisions: F48
divisions: B04
divisions: UCL
keywords: cs.CL, cs.CL
note: © 1963–2022 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License.
abstract: Machine reading comprehension (MRC) has received considerable attention as a benchmark for natural language understanding. However, the conventional task design of MRC lacks explainability beyond the model interpretation, i.e., reading comprehension by a model cannot be explained in human terms. To this end, this position paper provides a theoretical basis for the design of MRC datasets based on psychology as well as psychometrics, and summarizes it in terms of the prerequisites for benchmarking MRC. We conclude that future datasets should (i) evaluate the capability of the model for constructing a coherent and grounded representation to understand context-dependent situations and (ii) ensure substantive validity by shortcut-proof questions and explanation as a part of the task design.
date: 2021-04-23
date_type: published
publisher: Association for Computational Linguistics
official_url: https://aclanthology.org/2021.eacl-main.137/
oa_status: green
full_text_type: pub
language: eng
primo: open
primo_central: open_green
verified: verified_manual
elements_id: 1958161
lyricists_name: Otias Protenest, Pontus
lyricists_id: PLEPS00
actors_name: Flynn, Bernadette
actors_id: BFFLY94
actors_role: owner
full_text_status: public
pres_type: paper
place_of_pub: Online
pagerange: 1592-1612
event_title: EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics
book_title: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics
citation:        Sugawara, Saku;    Stenetorp, Pontus;    Aizawa, Akiko;      (2021)    Benchmarking Machine Reading Comprehension: A Psychological Perspective.                     In:  Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics.  (pp. pp. 1592-1612).  Association for Computational Linguistics: Online.       Green open access   
 
document_url: https://discovery.ucl.ac.uk/id/eprint/10154325/1/2021.eacl-main.137.pdf