?url_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Adc&rft.title=Benchmarking+Machine+Reading+Comprehension%3A+A+Psychological+Perspective&rft.creator=Sugawara%2C+Saku&rft.creator=Stenetorp%2C+Pontus&rft.creator=Aizawa%2C+Akiko&rft.description=Machine+reading+comprehension+(MRC)+has+received+considerable+attention+as+a+benchmark+for+natural+language+understanding.+However%2C+the+conventional+task+design+of+MRC+lacks+explainability+beyond+the+model+interpretation%2C+i.e.%2C+reading+comprehension+by+a+model+cannot+be+explained+in+human+terms.+To+this+end%2C+this+position+paper+provides+a+theoretical+basis+for+the+design+of+MRC+datasets+based+on+psychology+as+well+as+psychometrics%2C+and+summarizes+it+in+terms+of+the+prerequisites+for+benchmarking+MRC.+We+conclude+that+future+datasets+should+(i)+evaluate+the+capability+of+the+model+for+constructing+a+coherent+and+grounded+representation+to+understand+context-dependent+situations+and+(ii)+ensure+substantive+validity+by+shortcut-proof+questions+and+explanation+as+a+part+of+the+task+design.&rft.subject=cs.CL%2C+cs.CL&rft.publisher=Association+for+Computational+Linguistics&rft.date=2021-04-23&rft.type=Proceedings+paper&rft.language=eng&rft.source=+++++In%3A++Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics.++(pp.+pp.+1592-1612).++Association+for+Computational+Linguistics%3A+Online.+(2021)+++++&rft.format=application%2Fpdf&rft.identifier=https%3A%2F%2Fdiscovery.ucl.ac.uk%2Fid%2Feprint%2F10154325%2F1%2F2021.eacl-main.137.pdf&rft.identifier=https%3A%2F%2Fdiscovery.ucl.ac.uk%2Fid%2Feprint%2F10154325%2F&rft.rights=open