Kırnap, Ö;
Diaz, F;
Biega, A;
Ekstrand, M;
Carterette, B;
Yilmaz, E;
(2021)
Estimation of Fair Ranking Metrics with Incomplete Judgments.
In:
(Proceedings) WWW '21: The Web Conference 2021.
(pp. pp. 1065-1075).
ACM
Preview |
Text
3442381.3450080.pdf - Published Version Download (1MB) | Preview |
Abstract
There is increasing attention to evaluating the fairness of search system ranking decisions. These metrics often consider the membership of items to particular groups, often identified using protected attributes such as gender or ethnicity. To date, these metrics typically assume the availability and completeness of protected attribute labels of items. However, the protected attributes of individuals are rarely present, limiting the application of fair ranking metrics in large scale systems. In order to address this problem, we propose a sampling strategy and estimation technique for four fair ranking metrics. We formulate a robust and unbiased estimator which can operate even with very limited number of labeled items. We evaluate our approach using both simulated and real world data. Our experimental results demonstrate that our method can estimate this family of fair ranking metrics and provides a robust, reliable alternative to exhaustive or random data annotation.
| Type: | Proceedings paper |
|---|---|
| Title: | Estimation of Fair Ranking Metrics with Incomplete Judgments |
| Event: | WWW '21: The Web Conference 2021 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3442381.3450080 |
| Publisher version: | https://doi.org/10.1145/3442381.3450080 |
| Language: | English |
| Additional information: | This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10139263 |
Archive Staff Only
 |
View Item |
