Lipani, A; Palotti, J; Lupu, M; Piroi, F; Zuccon, G; Hanbury, A; (2017) Fixed-Cost Pooling Strategies Based on IR Evaluation Measures. In: Jose, JM and Hauff, C and Altingovde, IS and Song, D and Albakour, D and Watt, S and Tait, J, (eds.) Advances in Information Retrieval, 39th European Conference on IR Research: Proceedings. (pp. pp. 357-368). Springer: Cham, Switzerland.

Preview

Text main.pdf - Accepted Version Download (348kB) | Preview

Abstract

Recent studies have reconsidered the way we operationalise the pooling method, by considering the practical limitations often encountered by test collection builders. The biggest constraint is often the budget available for relevance assessments and the question is how best – in terms of the lowest pool bias – to select the documents to be assessed given a fixed budget. Here, we explore a series of 3 new pooling strategies introduced in this paper against 3 existing ones and a baseline. We show that there are significant differences depending on the evaluation measure ultimately used to assess the runs. We conclude that adaptive strategies are always best, but in their absence, for top-heavy evaluation measures we can continue to use the baseline, while for P@100 we should use any of the other non-adaptive strategies.

Download activity - last month

Export as CSVJSONXML

Download activity - last 12 months

Export as JSONXMLCSV

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item