Mehrotra, R; Anderson, A; Diaz, F; Sharma, A; Wallach, HM; Yilmaz, E; (2017) Auditing Search Engines for Differential Satisfaction Across Demographics. In: Barrett, R and Cummings, R and Agichtein, E and Gabrilovich, E, (eds.) WWW '17 Companion: Proceedings of the 26th International Conference on World Wide Web Companion. (pp. pp. 626-633). Association for Computing Machinery (ACM): New York, NY, USA.

Preview

Text p626.pdf - Published Version Download (1MB) | Preview

Abstract

Many online services, such as search engines, social media platforms, and digital marketplaces, are advertised as being available to any user, regardless of their age, gender, or other demographic factors. However, there are growing concerns that these services may systematically underserve some groups of users. In this paper, we present a framework for internally auditing such services for differences in user satisfaction across demographic groups, using search engines as a case study. We first explain the pitfalls of naively comparing the behavioral metrics that are commonly used to evaluate search engines. We then propose three methods for measuring latent differences in user satisfaction from observed differences in evaluation metrics. To develop these methods, we drew on ideas from the causal inference literature and the multilevel modeling literature. Our framework is broadly applicable to other online services, and provides general insight into interpreting their evaluation metrics.

Download activity - last month

Export as CSVXMLJSON

Download activity - last 12 months

Export as XMLCSVJSON

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item