Lipani, A;
Lupu, M;
Palotti, J;
Zuccon, G;
Hanbury, A;
(2017)
Fixed budget pooling strategies based on fusion methods.
In: Shin, SY and Shin, D and Lencastre, M, (eds.)
SAC '17: Proceedings of the Symposium on Applied Computing.
(pp. pp. 919-924).
ACM
Preview |
Text
paper.pdf - Accepted Version Download (382kB) | Preview |
Abstract
The empirical nature of Information Retrieval (IR) mandates strong experimental practices. The Cranfield/TREC evaluation paradigm represents a keystone of such experimental practices. Within this paradigm, the generation of relevance judgments has been the subject of intense scientific investigation. This is because, on one hand, consistent, precise and numerous judgements are key to reduce evaluation uncertainty and test collection bias; on the other hand, however, relevance judgements are costly to collect. The selection of which documents to judge for relevance (known as pooling) has therefore great impact in IR evaluation. In this paper, we contribute a set of 8 novel pooling strategies based on retrieval fusion methods. We show that the choice of the pooling strategy has significant effects on the cost needed to obtain an unbiased test collection; we also identify the best performing pooling strategy according to three evaluation measure.
| Type: | Proceedings paper |
|---|---|
| Title: | Fixed budget pooling strategies based on fusion methods |
| Event: | SAC 2017, 32nd ACM SIGAPP Symposium On Applied Computing, 3-7 April 2017, Marrakech, Morocco |
| ISBN-13: | 9781450344869 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3019612.3019692 |
| Publisher version: | https://doi.org/10.1145/3019612.3019692 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Pooling Method, Pooling Strategies, Pool Bias |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Civil, Environ and Geomatic Eng |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10059982 |
Archive Staff Only
 |
View Item |
