?url_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Adc&rft.title=The+impact+of+fixed-cost+pooling+strategies+on+test+collection+bias&rft.creator=Lipani%2C+A&rft.creator=Zuccon%2C+G&rft.creator=Lupu%2C+M&rft.creator=Koopman%2C+B&rft.creator=Hanbury%2C+A&rft.description=In+Information+Retrieval%2C+test+collections+are+usually+built+using+the+pooling+method.+Many+pooling+strategies+have+been+developed+for+the+pooling+method.+Herein%2C+we+address+the+question+of+identifying+the+best+pooling+strategy+when+evaluating+systems+using+precision-oriented+measures+in+presence+of+budget+constraints+on+the+number+of+documents+to+be+evaluated.+As+a+quality+measurement+we+use+the+bias+introduced+by+the+pooling+strategy%2C+measured+both+in+terms+of+Mean+Absolute+Error+of+the+scores+and+in+terms+of+ranking+errors.+Based+on+experiments+on+15+test+collections%2C+we+conclude+that%2C+for+precision-oriented+measures%2C+the+best+strategies+are+based+on+Rank-Biased+Precision+(RBP).+These+results+can+inform+collection+builders+because+they+suggest+that%2C+under+fixed+assessment+budget+constraints%2C+RBP-based+sampling+produces+less+biased+pools+than+other+alternatives.&rft.subject=Pooling+Method%2C+Pooling+Strategies%2C+Pool+Bias&rft.publisher=ACM&rft.contributor=Carterette%2C+B&rft.contributor=Fang%2C+H&rft.contributor=Lalmas%2C+M&rft.contributor=Nie%2C+J-Y&rft.date=2016-09-12&rft.type=Proceedings+paper&rft.language=eng&rft.source=+++++In%3A+Carterette%2C+B+and+Fang%2C+H+and+Lalmas%2C+M+and+Nie%2C+J-Y%2C+(eds.)+ICTIR+'16%3A+Proceedings+of+the+2016+ACM+International+Conference+on+the+Theory+of+Information+Retrieval.++(pp.+pp.+105-108).++ACM%3A+New+York%2C+USA.+(2016)+++++&rft.format=text&rft.identifier=https%3A%2F%2Fdiscovery.ucl.ac.uk%2Fid%2Feprint%2F10059980%2F1%2Fmain.pdf&rft.identifier=https%3A%2F%2Fdiscovery.ucl.ac.uk%2Fid%2Feprint%2F10059980%2F&rft.rights=open