Giulianelli, Mario;
Malagutti, Luca;
Gastaldi, Juan Luis;
DuSell, Brian;
Vieira, Tim;
Cotterell, Ryan;
(2024)
On the Proper Treatment of Tokenization in Psycholinguistics.
In: Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung, (eds.)
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.
(pp. pp. 18556-18572).
Association for Computational Linguistics: Miami, FL, USA.
Preview |
Text
2024.emnlp-main.1032.pdf - Published Version Download (874kB) | Preview |
Abstract
Language models are widely used in computational psycholinguistics to test theories that relate the negative log probability (the surprisal) of a region of interest (a substring of characters) under a language model to its cognitive cost experienced by readers, as operationalized, for example, by gaze duration on the region. However, the application of modern language models to psycholinguistic studies is complicated by the practice of using tokenization as an intermediate step in training a model. Doing so results in a language model over token strings rather than one over character strings. Vexingly, regions of interest are generally misaligned with these token strings. The paper argues that token-level language models should be (approximately) marginalized into character-level language models before they are used in psycholinguistic studies to compute the surprisal of a region of interest; then, the marginalized character-level language model can be used to compute the surprisal of an arbitrary character substring, which we term a focal area, that the experimenter may wish to use as a predictor. Our proposal of marginalizing a token-level model into a character-level one solves this misalignment issue independently of the tokenization scheme. Empirically, we discover various focal areas whose surprisal is a better psychometric predictor than the surprisal of the region of interest itself.
| Type: | Proceedings paper |
|---|---|
| Title: | On the Proper Treatment of Tokenization in Psycholinguistics |
| Event: | Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing |
| Dates: | Nov 2024 - Nov 2024 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.18653/v1/2024.emnlp-main.1032 |
| Publisher version: | https://doi.org/10.18653/v1/2024.emnlp-main.1032 |
| Language: | English |
| Additional information: | ACL materials are Copyright © 1963–2025 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Linguistics |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10216476 |
Archive Staff Only
 |
View Item |
