UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Cost Measures Matter for Mutation Testing Study Validity

Guizzo, G; Sarro, F; Harman, M; (2020) Cost Measures Matter for Mutation Testing Study Validity. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringNovember 2020 (ESEC/FSE 2020). ACM: Newyork, USA. Green open access

[img]
Preview
Text
fse-v1.9.pdf - Accepted version

Download (675kB) | Preview

Abstract

Mutation testing research has often used the number of mutants as a surrogate measure for the true execution cost of generating and executing mutants. This poses a potential threat to the validity of the scientific findings reported in the literature. Out of 75 works surveyed in this paper, we found that 54 (72%) are vulnerable to this threat. To investigate the magnitude of the threat, we conducted an empirical evaluation using 10 real-world programs. The results reveal that: i) percentages of randomly sampled mutants differ from the true execution time, on average, by 44%, varying in difference from 19% to 91%; ii) errors arising from using the surrogate correlate with program size (ρ = 0.74) and number of mutants (ρ = 0.76), making the problem more pernicious for more realistic programs; iii) scientific findings concerning sampling strategies would have approximately 37% rank disagreement, indicating potentially dramatic impact on experiment validity. To investigate whether this threat matters in practice, we reproduced a seminal study on Selective Mutation (widely relied upon for more than two decades). The impact is stark: an inconclusive scientific finding using the surrogate is transformed to an unequivocal finding when using the true execution cost.

Type: Proceedings paper
Title: Cost Measures Matter for Mutation Testing Study Validity
Event: ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)
Location: Sacramento, California, United States
Dates: 08 November 2020 - 13 November 2020
ISBN-13: 978-1-4503-7043-1/20/11
Open access status: An open access version is available from UCL Discovery
DOI: 10.1145/ 3368089.3409742
Publisher version: https://doi.org/10.1145/3368089.3409742
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10100738
Downloads since deposit
27Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item