Tsipidi, Eleftheria;
Nowak, Franz;
Cotterell, Ryan;
Wilcox, Ethan;
Giulianelli, Mario;
Warstadt, Alex;
(2024)
Surprise! Uniform Information Density Isn’t the Whole Story: Predicting Surprisal Contours in Long-form Discourse.
In: Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung, (eds.)
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.
(pp. pp. 18820-18836).
Association for Computational Linguistics: Miami, FL, USA.
Preview |
Text
2024.emnlp-main.1047.pdf - Published Version Download (417kB) | Preview |
Abstract
The Uniform Information Density (UID) hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. Of course, information rate in texts and discourses is not perfectly uniform. While these fluctuations can be viewed as theoretically uninteresting noise on top of a uniform target, another explanation is that UID is not the only functional pressure regulating information content in a language. Speakers may also seek to maintain interest, adhere to writing conventions, and build compelling arguments. In this paper, we propose one such functional pressure; namely that speakers modulate information rate based on location within a hierarchically-structured model of discourse. We term this the Structured Context Hypothesis and test it by predicting the surprisal contours of naturally occurring discourses extracted from large language models using predictors derived from discourse structure. We find that hierarchical predictors are significant predictors of a discourse’s information contour and that deeply nested hierarchical predictors are more predictive than shallow ones. This work takes an initial step beyond UID to propose testable hypotheses for why the information rate fluctuates in predictable ways.
| Type: | Proceedings paper |
|---|---|
| Title: | Surprise! Uniform Information Density Isn’t the Whole Story: Predicting Surprisal Contours in Long-form Discourse |
| Event: | Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing |
| Dates: | Nov 2024 - Nov 2024 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.18653/v1/2024.emnlp-main.1047 |
| Publisher version: | https://doi.org/10.18653/v1/2024.emnlp-main.1047 |
| Language: | English |
| Additional information: | ACL materials are Copyright © 1963–2025 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Linguistics |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10216477 |
Archive Staff Only
 |
View Item |
