Papadakis, M; Henard, C; Harman, M; Jia, Y; Traon, YL; (2016) Threats to the validity of mutation-based test assessment. In: Zeller, A and Roychoudhury, A, (eds.) Proceedings of the 25th International Symposium on Software Testing and Analysis ISSTA 2016. (pp. pp. 354-365). ACM: Saarbrücken, Germany.

Preview

Text ISSTA16.pdf - Accepted Version Download (796kB) | Preview

Abstract

Much research on software testing and test techniques relies on experimental studies based on mutation testing. In this paper we reveal that such studies are vulnerable to a potential threat to validity, leading to possible Type I errors; incorrectly rejecting the Null Hypothesis. Our findings indicate that Type I errors occur, for arbitrary experiments that fail to take countermeasures, approximately 62% of the time. Clearly, a Type I error would potentially compromise any scientific conclusion. We show that the problem derives from such studies’ combined use of both subsuming and subsumed mutants. We collected articles published in the last two years at three leading software engineering conferences. Of those that use mutation-based test assessment, we found that 68% are vulnerable to this threat to validity.

Download activity - last month

Export as XMLJSONCSV

Download activity - last 12 months

Export as XMLJSONCSV

Downloads by country - last 12 months

Export as XMLCSVJSON

Archive Staff Only

View Item