UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

Wu, H; Changhai, N; Petke, J; Jia, Y; Harman, M; (2019) An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing. IEEE Transactions on Software Engineering , 14 (8) 10.1109/TSE.2018.2852744. (In press). Green open access

[thumbnail of Petke_08405609.pdf]
Preview
Text
Petke_08405609.pdf - Published Version

Download (2MB) | Preview

Abstract

We present an empirical comparison of three test generation techniques, namely, Combinatorial Testing (CT), Random Testing (RT) and Adaptive Random Testing (ART), under different test scenarios. This is the first study in the literature to account for the (more realistic) testing setting in which the tester may not have complete information about the parameters and constraints that pertain to the system, and to account for the challenge posed by faults (in terms of failure rate). Our study was conducted on nine real-world programs under a total of 1683 test scenarios (combinations of available parameter and constraint information and failure rate). The results show significant differences in the techniques' fault detection ability when faults are hard to detect (failure rates are relatively low). CT performs best overall; no worse than any other in 98% of scenarios studied. ART enhances RT, and is comparable to CT in 96% of scenarios, but its computational cost can be up to 3.5 times higher than CT when the program is highly constrained. Additionally, when constraint information is unavailable for a highly-constrained program, a large random test suite is as effective as CT or ART, yet its computational cost of test generation is significantly lower than that of other techniques.

Type: Article
Title: An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing
Open access status: An open access version is available from UCL Discovery
DOI: 10.1109/TSE.2018.2852744
Publisher version: https://doi.org/10.1109/TSE.2018.2852744
Language: English
Additional information: Copyright © 2018 IEEE. This is an Open Access article published under the IEEE Open Access Publishing Agreement
Keywords: Testing, Subspace constraints, Computational efficiency, Fault detection, Analytical models, Software systems
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10053044
Downloads since deposit
245Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item