UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

How Am I Doing?: Evaluating Conversational Search Systems Offline

Lipani, A; Carterette, B; Yilmaz, E; (2021) How Am I Doing?: Evaluating Conversational Search Systems Offline. ACM Transactions on Information Systems , 39 (4) , Article 51. 10.1145/3451160. Green open access

[thumbnail of How_Am_I_Doing-Evaluating_Conversational_Search_Systems_Offline.pdf]
Preview
Text
How_Am_I_Doing-Evaluating_Conversational_Search_Systems_Offline.pdf - Accepted Version

Download (1MB) | Preview

Abstract

As conversational agents like Siri and Alexa gain in popularity and use, conversation is becoming a more and more important mode of interaction for search. Conversational search shares some features with traditional search, but differs in some important respects: conversational search systems are less likely to return ranked lists of results (a SERP), more likely to involve iterated interactions, and more likely to feature longer, well-formed user queries in the form of natural language questions. Because of these differences, traditional methods for search evaluation (such as the Cranfield paradigm) do not translate easily to conversational search. In this work, we propose a framework for offline evaluation of conversational search, which includes a methodology for creating test collections with relevance judgments, an evaluation measure based on a user interaction model, and an approach to collecting user interaction data to train the model. The framework is based on the idea of “subtopics”, often used to model novelty and diversity in search and recommendation, and the user model is similar to the geometric browsing model introduced by RBP and used in ERR. As far as we know, this is the first work to combine these ideas into a comprehensive framework for offline evaluation of conversational search.

Type: Article
Title: How Am I Doing?: Evaluating Conversational Search Systems Offline
Open access status: An open access version is available from UCL Discovery
DOI: 10.1145/3451160
Publisher version: https://doi.org/10.1145/3451160
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Information retrieval, conversational search, evaluation, test collections
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Civil, Environ and Geomatic Eng
URI: https://discovery.ucl.ac.uk/id/eprint/10125575
Downloads since deposit
374Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item