eprintid: 10059980
rev_number: 21
eprint_status: archive
userid: 608
dir: disk0/10/05/99/80
datestamp: 2018-10-30 18:36:42
lastmod: 2021-09-17 23:01:10
status_changed: 2018-10-30 18:36:42
type: proceedings_section
metadata_visibility: show
creators_name: Lipani, A
creators_name: Zuccon, G
creators_name: Lupu, M
creators_name: Koopman, B
creators_name: Hanbury, A
title: The impact of fixed-cost pooling strategies on test collection bias
ispublished: pub
divisions: UCL
divisions: B04
divisions: C05
divisions: F44
keywords: Pooling Method, Pooling Strategies, Pool Bias
note: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
abstract: In Information Retrieval, test collections are usually built using the pooling method. Many pooling strategies have been developed for the pooling method. Herein, we address the question of identifying the best pooling strategy when evaluating systems using precision-oriented measures in presence of budget constraints on the number of documents to be evaluated. As a quality measurement we use the bias introduced by the pooling strategy, measured both in terms of Mean Absolute Error of the scores and in terms of ranking errors. Based on experiments on 15 test collections, we conclude that, for precision-oriented measures, the best strategies are based on Rank-Biased Precision (RBP). These results can inform collection builders because they suggest that, under fixed assessment budget constraints, RBP-based sampling produces less biased pools than other alternatives.
date: 2016-09-12
date_type: published
publisher: ACM
official_url: https://doi.org/10.1145/2970398.2970429
oa_status: green
full_text_type: other
language: eng
primo: open
primo_central: open_green
verified: verified_manual
elements_id: 1588679
doi: 10.1145/2970398.2970429
isbn_13: 9781450344975
lyricists_name: Lipani, Aldo
lyricists_id: ALIPA33
actors_name: Lipani, Aldo
actors_id: ALIPA33
actors_role: owner
full_text_status: public
series: ACM International Conference on the Theory of Information Retrieval
publication: ICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval
place_of_pub: New York, USA
pagerange: 105-108
event_title: ICTIR '16: ACM SIGIR International Conference on the Theory of Information Retrieval, 12 - 16 September 2016, Newark, Delaware, USA
book_title: ICTIR '16: Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval
editors_name: Carterette, B
editors_name: Fang, H
editors_name: Lalmas, M
editors_name: Nie, J-Y
citation:        Lipani, A;    Zuccon, G;    Lupu, M;    Koopman, B;    Hanbury, A;      (2016)    The impact of fixed-cost pooling strategies on test collection bias.                     In: Carterette, B and Fang, H and Lalmas, M and Nie, J-Y, (eds.) ICTIR '16: Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval.  (pp. pp. 105-108).  ACM: New York, USA.       Green open access   
 
document_url: https://discovery.ucl.ac.uk/id/eprint/10059980/1/main.pdf