Massarelli, L;
Petroni, F;
Piktus, A;
Ott, M;
Rocktäschel, T;
Plachouras, V;
Silvestri, F;
(2019)
How Decoding Strategies Affect the Verifiability of Generated Text.
In:
Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020.
(pp. pp. 223-235).
Association for Computational Linguistics
Preview |
Text
Rocktaschel_2020.findings-emnlp.22.pdf - Published Version Download (570kB) | Preview |
Abstract
Language models are of considerable importance. They are used for pretraining, finetuning, and rescoring in downstream applications, and as is as a test-bed and benchmark for progress in natural language understanding. One fundamental question regards the way we should generate text from a language model. It is well known that different decoding strategies can have dramatic impact on the quality of the generated text and using the most likely sequence under the model distribution, e.g., via beam search, generally leads to degenerate and repetitive outputs. While generation strategies such as top-k and nucleus sampling lead to more natural and less repetitive generations, the true cost of avoiding the highest scoring solution is hard to quantify. In this paper, we argue that verifiability, i.e., the consistency of the generated text with factual knowledge, is a suitable metric for measuring this cost. We use an automatic fact-checking system to calculate new metrics as a function of the number of supported claims per sentence and find that sampling-based generation strategies, such as top-k, indeed lead to less verifiable text. This finding holds across various dimensions, such as model size, training data size and parameters of the generation strategy. Based on this finding, we introduce a simple and effective generation strategy for producing non-repetitive and more verifiable (in comparison to other methods) text.
| Type: | Proceedings paper |
|---|---|
| Title: | How Decoding Strategies Affect the Verifiability of Generated Text |
| Event: | Findings of the Association for Computational Linguistics: EMNLP 2020 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.18653/v1/2020.findings-emnlp.22 |
| Publisher version: | https://doi.org/10.18653/v1/2020.findings-emnlp.22 |
| Language: | English |
| Additional information: | © 1963–2021 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10086496 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
