Guizzo, G;
Sarro, F;
Harman, M;
(2020)
Cost Measures Matter for Mutation Testing Study Validity.
In:
Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringNovember 2020 (ESEC/FSE 2020).
ACM: Newyork, USA.
Preview |
Text
fse-v1.9.pdf - Accepted Version Download (675kB) | Preview |
Abstract
Mutation testing research has often used the number of mutants as a surrogate measure for the true execution cost of generating and executing mutants. This poses a potential threat to the validity of the scientific findings reported in the literature. Out of 75 works surveyed in this paper, we found that 54 (72%) are vulnerable to this threat. To investigate the magnitude of the threat, we conducted an empirical evaluation using 10 real-world programs. The results reveal that: i) percentages of randomly sampled mutants differ from the true execution time, on average, by 44%, varying in difference from 19% to 91%; ii) errors arising from using the surrogate correlate with program size (ρ = 0.74) and number of mutants (ρ = 0.76), making the problem more pernicious for more realistic programs; iii) scientific findings concerning sampling strategies would have approximately 37% rank disagreement, indicating potentially dramatic impact on experiment validity. To investigate whether this threat matters in practice, we reproduced a seminal study on Selective Mutation (widely relied upon for more than two decades). The impact is stark: an inconclusive scientific finding using the surrogate is transformed to an unequivocal finding when using the true execution cost.
| Type: | Proceedings paper |
|---|---|
| Title: | Cost Measures Matter for Mutation Testing Study Validity |
| Event: | ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) |
| Location: | Sacramento, California, United States |
| Dates: | 08 November 2020 - 13 November 2020 |
| ISBN-13: | 978-1-4503-7043-1/20/11 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/ 3368089.3409742 |
| Publisher version: | https://doi.org/10.1145/3368089.3409742 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10100738 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
