Hosseini, M; Cox, I; (2011) Prioritizing relevance judgments to improve the construction of IR test collections. (CS Research Notes ). UCL Department of Computer Science: London, UK.

Preview

PDF RN_11_16.pdf Download (347kB)

Abstract

We consider the problem of optimally allocating a fixed budget to construct a test collection with associated relevance judgements, such that it can (i) accurately evaluate the relative performance of the participating systems, and (ii) generalize to new, previously unseen systems. We propose a two stage approach. For a given set of queries, we adopt the traditional pooling method and use a portion of the budget to evaluate a set of documents retrieved by the participating systems. Next, prioritize the queries and associated documents for further refinement of the test collection. Our objective is to increase the effectiveness of the test collection for comparative evaluation and extendibility to new systems. The query prioritization is formulated as a convex optimization problem, thereby permitting efficient solution and providing a flexible framework to incorporate various constraints. We use the remaining budget to evaluate query-document pairs with the highest priority scores. The budgets for the initial and the refinement phase are expended during the construction of the test collection and consider only the documents that have been retrieved by the participating systems. We evaluate our resource optimization approach on two TREC test collections namely TREC 8 and TREC 2004 Robust Track. We demonstrate that our optimization techniques are cost efficient and yield a significant improvement in the reusability of the test collections.

Download activity - last month

Export as CSVXMLJSON

Download activity - last 12 months

Export as XMLCSVJSON

Downloads by country - last 12 months

Export as JSONXMLCSV

Archive Staff Only

View Item