UCL logo

UCL Discovery

UCL home » Library Services » Electronic resources » UCL Discovery

Fixed-Cost Pooling Strategies Based on IR Evaluation Measures

Lipani, A; Palotti, J; Lupu, M; Piroi, F; Zuccon, G; Hanbury, A; (2017) Fixed-Cost Pooling Strategies Based on IR Evaluation Measures. In: Jose, JM and Hauff, C and Altingovde, IS and Song, D and Albakour, D and Watt, S and Tait, J, (eds.) Advances in Information Retrieval, 39th European Conference on IR Research: Proceedings. (pp. pp. 357-368). Springer: Cham, Switzerland. Green open access

[img]
Preview
Text
main.pdf - ["content_typename_Accepted version" not defined]

Download (348kB) | Preview

Abstract

Recent studies have reconsidered the way we operationalise the pooling method, by considering the practical limitations often encountered by test collection builders. The biggest constraint is often the budget available for relevance assessments and the question is how best – in terms of the lowest pool bias – to select the documents to be assessed given a fixed budget. Here, we explore a series of 3 new pooling strategies introduced in this paper against 3 existing ones and a baseline. We show that there are significant differences depending on the evaluation measure ultimately used to assess the runs. We conclude that adaptive strategies are always best, but in their absence, for top-heavy evaluation measures we can continue to use the baseline, while for P@100 we should use any of the other non-adaptive strategies.

Type: Proceedings paper
Title: Fixed-Cost Pooling Strategies Based on IR Evaluation Measures
Event: ECIR 2017, 39th European Conference on Information Retrieval Research, 8-13 April 2017, Aberdeen, UK
Location: Aberdeen, SCOTLAND
Dates: 08 April 2017 - 13 April 2017
ISBN-13: 978-3-319-56607-8
Open access status: An open access version is available from UCL Discovery
DOI: 10.1007/978-3-319-56608-5_28
Publisher version: https://doi.org/10.1007/978-3-319-56608-5_28
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: http://discovery.ucl.ac.uk/id/eprint/10059981
Downloads since deposit
41Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item