Kocijan, Vid; Cretu, Ana-Maria; Camburu, Oana-Maria; Yordanov, Yordan; Lukasiewicz, Thomas; (2019) A Surprisingly Robust Trick for the Winograd Schema Challenge. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics: Florence, Italy.

Preview

Text trick.pdf - Published Version Download (232kB) | Preview

Abstract

The Winograd Schema Challenge (WSC) dataset WSC273 and its inference counterpart WNLI are popular benchmarks for natural language understanding and commonsense reasoning. In this paper, we show that the performance of three language models on WSC273 consistently and robustly improves when finetuned on a similar pronoun disambiguation problem dataset (denoted WSCR). We additionally generate a large unsupervised WSClike dataset. By fine-tuning the BERT language model both on the introduced and on the WSCR dataset, we achieve overall accuracies of 72.5% and 74.7% on WSC273 and WNLI, improving the previous state-of-theart solutions by 8.8% and 9.6%, respectively. Furthermore, our fine-tuned models are also consistently more accurate on the “complex” subsets of WSC273, introduced by Trichelair et al. (2018).

Download activity - last month

Export as XMLCSVJSON

Download activity - last 12 months

Export as CSVJSONXML

Downloads by country - last 12 months

Export as JSONCSVXML

Archive Staff Only

View Item