UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

The impact of fixed-cost pooling strategies on test collection bias

Lipani, A; Zuccon, G; Lupu, M; Koopman, B; Hanbury, A; (2016) The impact of fixed-cost pooling strategies on test collection bias. In: Carterette, B and Fang, H and Lalmas, M and Nie, J-Y, (eds.) ICTIR '16: Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval. (pp. pp. 105-108). ACM: New York, USA. Green open access

[thumbnail of main.pdf]
Preview
Text
main.pdf - Accepted Version

Download (198kB) | Preview

Abstract

In Information Retrieval, test collections are usually built using the pooling method. Many pooling strategies have been developed for the pooling method. Herein, we address the question of identifying the best pooling strategy when evaluating systems using precision-oriented measures in presence of budget constraints on the number of documents to be evaluated. As a quality measurement we use the bias introduced by the pooling strategy, measured both in terms of Mean Absolute Error of the scores and in terms of ranking errors. Based on experiments on 15 test collections, we conclude that, for precision-oriented measures, the best strategies are based on Rank-Biased Precision (RBP). These results can inform collection builders because they suggest that, under fixed assessment budget constraints, RBP-based sampling produces less biased pools than other alternatives.

Type: Proceedings paper
Title: The impact of fixed-cost pooling strategies on test collection bias
Event: ICTIR '16: ACM SIGIR International Conference on the Theory of Information Retrieval, 12 - 16 September 2016, Newark, Delaware, USA
ISBN-13: 9781450344975
Open access status: An open access version is available from UCL Discovery
DOI: 10.1145/2970398.2970429
Publisher version: https://doi.org/10.1145/2970398.2970429
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Pooling Method, Pooling Strategies, Pool Bias
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Civil, Environ and Geomatic Eng
URI: https://discovery.ucl.ac.uk/id/eprint/10059980
Downloads since deposit
150Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item