TY  - GEN
N1  - This version is the author accepted manuscript. For information on re-use, please refer to the publisher?s terms and conditions.
SP  - 252
AV  - public
Y1  - 2019/06/06/
EP  - 263
TI  - Do Pseudo Test Suites Lead to Inflated Correlation in Measuring Test Effectiveness?
A1  - Zhang, J
A1  - Zhang, LZ
A1  - Hao, D
A1  - Wang, M
A1  - Zhang, L
KW  - test suites
KW  -  coverage criteria
KW  -  empirical study
CY  - Xi'an, China, China
UR  - http://dx.doi.org/10.1109/ICST.2019.00033
PB  - IEEE
SN  - 0960-0833
N2  - Code coverage is the most widely adopted criteria for measuring test effectiveness in software quality assurance. The performance of coverage criteria (in indicating test suites? effectiveness) has been widely studied in prior work. Most of the studies use randomly constructed pseudo test suites to facilitate data collection for correlation analysis, yet no previous work has systematically studied whether pseudo test suites would lead to inflated correlation results. This paper focuses on the potentially wide-spread threat with a study over 123 real-world Java projects. Following the typical experimental process of studying coverage criteria, we investigate the correlation between statement/assertion coverage and mutation score using both pseudo and original test suites. Except for direct correlation analysis, we control the number of assertions and the test suite size to conduct partial correlation analysis. The results reveal that 1) the correlation (between coverage criteria and mutation score) derived from pseudo test suites is much higher than from original test suites (from 0.21 to 0.39 higher in Kendall ?b value); 2) contrary to previously reported, statement coverage has a stronger correlation with mutation score than assertion coverage.
ID  - discovery10075196
ER  -