Giulianelli, Mario;
Opedal, Andreas;
Cotterell, Ryan;
(2024)
Generalized Measures of Anticipation and Responsivity in Online Language Processing.
In: Al-Onaizan,, Yaser and Bansal,, Mohit and Chen, Yun-Nung, (eds.)
Findings of the Association for Computational Linguistics: EMNLP 2024.
(pp. pp. 11648-11669).
Association for Computational Linguistics: Miami, FL, USA.
Preview |
Text
2024.findings-emnlp.682.pdf - Published Version Download (2MB) | Preview |
Abstract
We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations of incremental linguistic contexts. Our framework provides a formal definition of anticipatory and responsive measures, and it equips experimenters with the tools to define new, more expressive measures beyond standard next-symbol entropy and surprisal. While extracting these standard quantities from language models is convenient, we demonstrate that using Monte Carlo simulation to estimate alternative responsive and anticipatory measures pays off empirically: New special cases of our generalized formula exhibit enhanced predictive power compared to surprisal for human cloze completion probability as well as ELAN, LAN, and N400 amplitudes, and greater complementarity with surprisal in predicting reading times
| Type: | Proceedings paper |
|---|---|
| Title: | Generalized Measures of Anticipation and Responsivity in Online Language Processing |
| Event: | Findings of the Association for Computational Linguistics: EMNLP 2024 |
| Dates: | Nov 2024 - Nov 2024 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.18653/v1/2024.findings-emnlp.682 |
| Publisher version: | https://doi.org/10.18653/v1/2024.findings-emnlp.68... |
| Language: | English |
| Additional information: | ACL materials are Copyright © 1963–2025 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Linguistics |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10216478 |
Archive Staff Only
 |
View Item |
