Wu, Y; Zhao, Y; Hu, B; Minervini, P; Stenetorp, P; Riedel, S; (2022) An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks. In: Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue, (eds.) Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. (pp. pp. 5184-5196). Association for Computational Linguistics: Abu Dhabi, United Arab Emirates.

Preview

Text 2022.emnlp-main.346.pdf - Published Version Download (423kB) | Preview

Abstract

Access to external knowledge is essential for many natural language processing tasks, such as question answering and dialogue. Existing methods often rely on a parametric model that stores knowledge in its parameters, or use a retrieval-augmented model that has access to an external knowledge source. Parametric and retrieval-augmented models have complementary strengths in terms of computational efficiency and predictive accuracy. To combine the strength of both approaches, we propose the Efficient Memory-Augmented Transformer (EMAT) – it encodes external knowledge into a key-value memory and exploits the fast maximum inner product search for memory querying. We also introduce pre-training tasks that allow EMAT to encode informative key-value representations, and to learn an implicit strategy to integrate multiple memory slots into the transformer. Experiments on various knowledge-intensive tasks such as question answering and dialogue datasets show that, simply augmenting parametric models (T5-base) using our method produces more accurate results (e.g., 25.8 → 44.3 EM on NQ) while retaining a high throughput (e.g., 1000 queries/s on NQ). Compared to retrievalaugmented models, EMAT runs substantially faster across the board and produces more accurate results on WoW and ELI5.1

Download activity - last month

Export as JSONXMLCSV

Download activity - last 12 months

Export as XMLCSVJSON

Downloads by country - last 12 months

Export as JSONXMLCSV

Archive Staff Only

View Item