Wu, H;
Changhai, N;
Petke, J;
Jia, Y;
Harman, M;
(2019)
An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing.
IEEE Transactions on Software Engineering
, 14
(8)
10.1109/TSE.2018.2852744.
(In press).
Preview |
Text
Petke_08405609.pdf - Published Version Download (2MB) | Preview |
Abstract
We present an empirical comparison of three test generation techniques, namely, Combinatorial Testing (CT), Random Testing (RT) and Adaptive Random Testing (ART), under different test scenarios. This is the first study in the literature to account for the (more realistic) testing setting in which the tester may not have complete information about the parameters and constraints that pertain to the system, and to account for the challenge posed by faults (in terms of failure rate). Our study was conducted on nine real-world programs under a total of 1683 test scenarios (combinations of available parameter and constraint information and failure rate). The results show significant differences in the techniques' fault detection ability when faults are hard to detect (failure rates are relatively low). CT performs best overall; no worse than any other in 98% of scenarios studied. ART enhances RT, and is comparable to CT in 96% of scenarios, but its computational cost can be up to 3.5 times higher than CT when the program is highly constrained. Additionally, when constraint information is unavailable for a highly-constrained program, a large random test suite is as effective as CT or ART, yet its computational cost of test generation is significantly lower than that of other techniques.
| Type: | Article |
|---|---|
| Title: | An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1109/TSE.2018.2852744 |
| Publisher version: | https://doi.org/10.1109/TSE.2018.2852744 |
| Language: | English |
| Additional information: | Copyright © 2018 IEEE. This is an Open Access article published under the IEEE Open Access Publishing Agreement |
| Keywords: | Testing, Subspace constraints, Computational efficiency, Fault detection, Analytical models, Software systems |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10053044 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
