UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Mitigating the Position Bias of Transformer Models in Passage Re-Ranking

Hofstatter, S; Lipani, A; Althammer, S; Zlabinger, M; Hanbury, A; (2021) Mitigating the Position Bias of Transformer Models in Passage Re-Ranking. In: Advances in Information Retrieval. ECIR 2021. (pp. pp. 238-253). Springer, Cham Green open access

[img]
Preview
Text
Mitigating_the_Position_Bias_of_Transformer_Based_Contextualization_for_Passage_Re_Ranking.pdf - Accepted version

Download (941kB) | Preview

Abstract

Supervised machine learning models and their evaluation strongly depends on the quality of the underlying dataset. When we search for a relevant piece of information it may appear anywhere in a given passage. However, we observe a bias in the position of the correct answer in the text in two popular Question Answering datasets used for passage re-ranking. The excessive favoring of earlier positions inside passages is an unwanted artefact. This leads to three common Transformer-based re-ranking models to ignore relevant parts in unseen passages. More concerningly, as the evaluation set is taken from the same biased distribution, the models overfitting to that bias overestimate their true effectiveness. In this work we analyze position bias on datasets, the contextualized representations, and their effect on retrieval results. We propose a debiasing method for retrieval datasets. Our results show that a model trained on a position-biased dataset exhibits a significant decrease in re-ranking effectiveness when evaluated on a debiased dataset. We demonstrate that by mitigating the position bias, Transformer-based re-ranking models are equally effective on a biased and debiased dataset, as well as more effective in a transfer-learning setting between two differently biased datasets.

Type: Proceedings paper
Title: Mitigating the Position Bias of Transformer Models in Passage Re-Ranking
Event: ECIR 2021
Open access status: An open access version is available from UCL Discovery
DOI: 10.1007/978-3-030-72113-8_16
Publisher version: https://doi.org/10.1007/978-3-030-72113-8_16
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Civil, Environ and Geomatic Eng
URI: https://discovery.ucl.ac.uk/id/eprint/10119400
Downloads since deposit
4Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item