The Importance of Accounting for Execution Failures when Predicting Test Flakiness

Advanced search
Browse by:

Department | Year

UCL Theses | Latest

Deposit your research

The Importance of Accounting for Execution Failures when Predicting Test Flakiness

Haben, Guillaume; Habchi, Sarra; Micco, John; Harman, Mark; Papadakis, Mike; Cordy, Maxime; Le Traon, Yves; (2024) The Importance of Accounting for Execution Failures when Predicting Test Flakiness. In: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. (pp. pp. 1979-1989). ACM Green open access

Preview

Text
RPS1.pdf - Published Version
Download (917kB) | Preview

Abstract

Flaky tests are tests that pass and fail on different executions of the same version of a program under test. They waste valuable developer time by making developers investigate false alerts (flaky test failures). To deal with this issue, many prediction methods have been proposed. However, the utility of these methods remains unclear since they are typically evaluated based on single-release data, ignoring that in many cases tests that fail flakily in one release also correctly fail (indicating the presence of bugs) in some other, meaning that it is possible for subsequent correctly-failing cases to pass unnoticed. In this paper, we show that this situation is prevalent and can raise significant concerns for both researchers and practitioners. In particular, we show that flaky tests, tests that exhibit flaky behaviour at some point in time, have a strong fault-revealing capability, i.e., they reveal more than 1/3 of all encountered regression faults. We also show that 76.2%, of all test executions that reveal faults in the codebase under test are made by tests that are classified as flaky by existing prediction methods. Overall, our findings motivate the need for future research to focus on predicting flaky test executions instead of flaky tests.

Type:	Proceedings paper
Title:	The Importance of Accounting for Execution Failures when Predicting Test Flakiness
Event:	ASE '24: 39th IEEE/ACM International Conference on Automated Software Engineering
Open access status:	An open access version is available from UCL Discovery
DOI:	10.1145/3691620.3695261
Publisher version:	https://doi.org/10.1145/3691620.3695261
Language:	English
Additional information:	This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification:	UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI:	https://discovery.ucl.ac.uk/id/eprint/10199773