Wu, Y;
Zhao, Y;
Hu, B;
Minervini, P;
Stenetorp, P;
Riedel, S;
(2022)
An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks.
In: Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue, (eds.)
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
(pp. pp. 5184-5196).
Association for Computational Linguistics: Abu Dhabi, United Arab Emirates.
Preview |
Text
2022.emnlp-main.346.pdf - Published Version Download (423kB) | Preview |
Abstract
Access to external knowledge is essential for many natural language processing tasks, such as question answering and dialogue. Existing methods often rely on a parametric model that stores knowledge in its parameters, or use a retrieval-augmented model that has access to an external knowledge source. Parametric and retrieval-augmented models have complementary strengths in terms of computational efficiency and predictive accuracy. To combine the strength of both approaches, we propose the Efficient Memory-Augmented Transformer (EMAT) – it encodes external knowledge into a key-value memory and exploits the fast maximum inner product search for memory querying. We also introduce pre-training tasks that allow EMAT to encode informative key-value representations, and to learn an implicit strategy to integrate multiple memory slots into the transformer. Experiments on various knowledge-intensive tasks such as question answering and dialogue datasets show that, simply augmenting parametric models (T5-base) using our method produces more accurate results (e.g., 25.8 → 44.3 EM on NQ) while retaining a high throughput (e.g., 1000 queries/s on NQ). Compared to retrievalaugmented models, EMAT runs substantially faster across the board and produces more accurate results on WoW and ELI5.1
| Type: | Proceedings paper |
|---|---|
| Title: | An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks |
| Event: | 2022 Conference on Empirical Methods in Natural Language Processing |
| Location: | Abu Dhabi, United Arab Emirates |
| Dates: | 7 Dec 2022 - 11 Dec 2022 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://aclanthology.org/2022.emnlp-main.346/ |
| Language: | English |
| Additional information: | ACL materials are Copyright © 1963–2023 ACL. Licensed on a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10166593 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
