Petroni, F;
Lewis, PSH;
Piktus, A;
Rocktäschel, T;
Wu, Y;
Miller, AH;
Riedel, S;
(2020)
How Context Affects Language Models' Factual Predictions.
In: McCallum, Andrew and Singh, Sameer and Halevy, Alon, (eds.)
Proceedings of the Automated Knowledge Base Construction (AKBC) 2020.
AKBC
Preview |
Text
2005.04611v1.pdf - Published Version Download (599kB) | Preview |
Abstract
When pre-trained on large unsupervised textual corpora, language models are able to store and retrieve factual knowledge to some extent, making it possible to use them directly for zero-shot cloze-style question answering. However, storing factual knowledge in a fixed number of weights of a language model clearly has limitations. Previous approaches have successfully provided access to information outside the model weights using supervised architectures that combine an information retrieval system with a machine reading component. In this paper, we go a step further and integrate information from a retrieval system with a pre-trained language model in a purely unsupervised way. We report that augmenting pre-trained language models in this way dramatically improves performance and that the resulting system, despite being unsupervised, is competitive with a supervised machine reading baseline. Furthermore, processing query and context with different segment tokens allows BERT to utilize its Next Sentence Prediction pre-trained classifier to determine whether the context is relevant or not, substantially improving BERT’s zeroshot cloze-style question-answering performance and making its predictions robust to noisy contexts.
| Type: | Proceedings paper |
|---|---|
| Title: | How Context Affects Language Models' Factual Predictions |
| Event: | Automated Knowledge Base Construction (AKBC) 2020 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://www.akbc.ws/2020/papers/025X0zPfn |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10100505 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
