Giulianelli, Mario;
Baan, Joris;
Aziz, Wilker;
Fernández, Raquel;
Plank, Barbara;
(2023)
What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability.
In: Bouamor, Houda and Pino, Juan and Bali, Kalika, (eds.)
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
(pp. pp. 14349-14371).
Association for Computational Linguistics: Singapore.
Preview |
Text
2023.emnlp-main.887.pdf - Published Version Download (2MB) | Preview |
Abstract
In Natural Language Generation (NLG) tasks, for any input, multiple communicative goals are plausible, and any goal can be put into words, or produced, in multiple ways. We characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty. We then inspect the space of output strings shaped by a generation system’s predicted probability distribution and decoding algorithm to probe its uncertainty. For each test input, we measure the generator’s calibration to human production variability. Following this instance-level approach, we analyse NLG models and decoding strategies, demonstrating that probing a generator with multiple samples and, when possible, multiple references, provides the level of detail necessary to gain understanding of a model’s representation of uncertainty.
| Type: | Proceedings paper |
|---|---|
| Title: | What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability |
| Event: | Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing |
| Dates: | Dec 2023 - Dec 2023 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.18653/v1/2023.emnlp-main.887 |
| Publisher version: | https://doi.org/10.18653/v1/2023.emnlp-main.887 |
| Language: | English |
| Additional information: | In Natural Language Generation (NLG) tasks, for any input, multiple communicative goals are plausible, and any goal can be put into words, or produced, in multiple ways. We characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty. We then inspect the space of output strings shaped by a generation system’s predicted probability distribution and decoding algorithm to probe its uncertainty. For each test input, we measure the generator’s calibration to human production variability. Following this instance-level approach, we analyse NLG models and decoding strategies, demonstrating that probing a generator with multiple samples and, when possible, multiple references, provides the level of detail necessary to gain understanding of a model’s representation of uncertainty. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Linguistics |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10216482 |
Archive Staff Only
 |
View Item |
