UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Are mutants really natural? A study on how "naturalness" helps mutant selection

Jimenez, M; Checkam, TT; Cordy, M; Papadakis, M; Kintis, M; Le Traon, Y; Harman, M; (2018) Are mutants really natural? A study on how "naturalness" helps mutant selection. In: International Symposium on Empirical Software Engineering and Measurement. (pp. pp. 1-10). ACM: Oulu, Finland. Green open access

[img]
Preview
Text
esem18.pdf - Published version

Download (6MB) | Preview

Abstract

Background: Code is repetitive and predictable in a way that is similar to the natural language. This means that code is "natural" and this "naturalness" can be captured by natural language modelling techniques. Such models promise to capture the program semantics and identify source code parts that `smell', i.e., they are strange, badly written and are generally error-prone (likely to be defective). Aims: We investigate the use of natural language modelling techniques in mutation testing (a testing technique that uses artificial faults). We thus, seek to identify how well artificial faults simulate real ones and ultimately understand how natural the artificial faults can be. Our intuition is that natural mutants, i.e., mutants that are predictable (follow the implicit coding norms of developers), are semantically useful and generally valuable (to testers). We also expect that mutants located on unnatural code locations (which are generally linked with error-proneness) to be of higher value than those located on natural code locations. Method: Based on this idea, we propose mutant selection strategies that rank mutants according to a) their naturalness (naturalness of the mutated code), b) the naturalness of their locations (naturalness of the original program statements) and c) their impact on the naturalness of the code that they apply to (naturalness differences between original and mutated statements). We empirically evaluate these issues on a benchmark set of 5 open-source projects, involving more than 100k mutants and 230 real faults. Based on the fault set we estimate the utility (i.e. capability to reveal faults) of mutants selected on the basis of their naturalness, and compare it against the utility of randomly selected mutants. Results: Our analysis shows that there is no link between naturalness and the fault revelation utility of mutants. We also demonstrate that the naturalness-based mutant selection performs similar (slightly worse) to the random mutant selection. Conclusions: Our findings are negative but we consider them interesting as they confute a strong intuition, i.e., fault revelation is independent of the mutants' naturalness.

Type: Proceedings paper
Title: Are mutants really natural? A study on how "naturalness" helps mutant selection
Event: 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) as part of the Empirical Software Engineering International Week (ESEIW)
Location: Oulu, FINLAND
Dates: 11 October 2018 - 12 October 2018
Open access status: An open access version is available from UCL Discovery
DOI: 10.1145/3239235.3240500
Publisher version: https://doi.org/10.1145/3239235.3240500
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Mutation testing, Fault Revelation, Language Models
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10102863
Downloads since deposit
9Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item