UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Fixed-Cost Pooling Strategies

Lipani, A; Losada, D; Zuccon, G; Lupu, M; (2019) Fixed-Cost Pooling Strategies. IEEE Transactions on Knowledge and Data Engineering 10.1109/TKDE.2019.2947049. (In press). Green open access

Lipani_Fixed-Cost Pooling Strategies_AAM.pdf - Accepted version

Download (3MB) | Preview


The empirical nature of Information Retrieval (IR) mandates strong experimental practices. A keystone of such experimental practices is the Cranfield evaluation paradigm. Within this paradigm, the collection of relevance judgments has been the subject of intense scientific investigation. This is because, on one hand, consistent, precise, and numerous judgments are keys to reducing evaluation uncertainty and test collection bias; on the other hand, however, relevance judgments are costly to collect. In this paper, we focus on the bias introduced by the pooling method, known as pool bias, which affects the reusability of test collections, in particular when building test collections with a limited budget. In this paper, we formalize and evaluate a set of 22 pooling strategies based on: traditional strategies, voting systems, retrieval fusion methods, evaluation measures, and multi-armed bandit models. To do this we run a large-scale evaluation by considering a set of 9 standard TREC test collections, in which we show that the choice of the pooling strategy has significant effects on the cost needed to obtain an unbiased test collection. We also identify the least biased pooling strategy in terms of pool bias according to three IR evaluation measures: AP, NDCG, and P@10.

Type: Article
Title: Fixed-Cost Pooling Strategies
Open access status: An open access version is available from UCL Discovery
DOI: 10.1109/TKDE.2019.2947049
Publisher version: https://doi.org/10.1109/TKDE.2019.2947049
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Resource management, Organizations, Q measurement, Uncertainty, Buildings, Standards, Benchmark testing
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Civil, Environ and Geomatic Eng
URI: https://discovery.ucl.ac.uk/id/eprint/10083262
Downloads since deposit
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item