UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Behavior analysis of NLI models: Uncovering the influence of three factors on robustness

Ivan Sanchez Carmona, V; Mitchell, J; Riedel, S; (2018) Behavior analysis of NLI models: Uncovering the influence of three factors on robustness. In: Walker, M and Ji, H and Stent, A, (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). (pp. pp. 1975-1985). Association for Computational Linguistics (ACL): New Orleans, LA, USA. Green open access

[img]
Preview
Text
N18-1179.pdf - Published version

Download (238kB) | Preview

Abstract

Natural Language Inference is a challenging task that has received substantial attention, and state-of-the-art models now achieve impressive test set performance in the form of accuracy scores. Here, we go beyond this single evaluation metric to examine robustness to semantically-valid alterations to the input data. We identify three factors - insensitivity, polarity and unseen pairs - and compare their impact on three SNLI models under a variety of conditions. Our results demonstrate a number of strengths and weaknesses in the models’ ability to generalise to new in-domain instances. In particular, while strong performance is possible on unseen hypernyms, unseen antonyms are more challenging for all the models. More generally, the models suffer from an insensitivity to certain small but semantically significant alterations, and are also often influenced by simple statistical correlations between words and training labels. Overall, we show that evaluations of NLI models can benefit from studying the influence of factors intrinsic to the models or found in the dataset used.

Type: Proceedings paper
Title: Behavior analysis of NLI models: Uncovering the influence of three factors on robustness
Event: 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Open access status: An open access version is available from UCL Discovery
DOI: 10.18653/v1/N18-1179
Publisher version: http://dx.doi.org/10.18653/v1/N18-1179
Language: English
Additional information: This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10098371
Downloads since deposit
14Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item