eprintid: 10130186
rev_number: 23
eprint_status: archive
userid: 608
dir: disk0/10/13/01/86
datestamp: 2021-06-25 16:50:20
lastmod: 2021-11-10 01:57:10
status_changed: 2021-11-05 16:10:35
type: proceedings_section
metadata_visibility: show
creators_name: Hort, M
creators_name: Zhang, J
creators_name: Sarro, F
creators_name: Harman, M
title: Fairea: A Model Behaviour Mutation Approach to Benchmarking Bias Mitigation Methods
ispublished: inpress
divisions: UCL
divisions: B04
divisions: C05
divisions: F48
note: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
abstract: The increasingly wide uptake of Machine Learning (ML) has raised the significance of the problem of tackling bias (i.e., unfairness), making it a primary software engineering concern. In this paper, we introduce Fairea, a model behaviour mutation approach to benchmarking ML bias mitigation methods. We also report on a large-scale empirical study to test the effectiveness of 12 widely-studied bias mitigation methods. Our results reveal that, surprisingly, bias mitigation methods have a poor effectiveness in 49% of the cases. In particular, 15% of the mitigation cases have worse fairness-accuracy trade-offs than the baseline established by Fairea; 34% of the cases have a decrease in accuracy and an increase in bias.

Fairea has been made publicly available for software engineers and researchers to evaluate their bias mitigation methods.
date: 2021-08-23
date_type: published
publisher: Association for Computing Machinery
official_url: https://doi.org/10.6084/m9.figshare.13712827.v2
oa_status: green
full_text_type: other
language: eng
primo: open
primo_central: open_green
verified: verified_manual
elements_id: 1872514
doi: 10.6084/m9.figshare.13712827.v2
lyricists_name: Harman, Mark
lyricists_name: Hort, Max
lyricists_name: Sarro, Federica
lyricists_id: MHARM36
lyricists_id: MBHOR72
lyricists_id: FSSAR72
actors_name: Hort, Max
actors_id: MBHOR72
actors_role: owner
full_text_status: public
place_of_pub: New York, NY, United States
pagerange: 994-1006
event_title: ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
institution: ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
book_title: ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
citation:        Hort, M;    Zhang, J;    Sarro, F;    Harman, M;      (2021)    Fairea: A Model Behaviour Mutation Approach to Benchmarking Bias Mitigation Methods.                     In:  ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.  (pp. pp. 994-1006).  Association for Computing Machinery: New York, NY, United States.    (In press).    Green open access   
 
document_url: https://discovery.ucl.ac.uk/id/eprint/10130186/1/Fairea-rps.pdf