eprintid: 10075196
rev_number: 22
eprint_status: archive
userid: 608
dir: disk0/10/07/51/96
datestamp: 2019-06-05 14:00:12
lastmod: 2021-11-10 23:20:58
status_changed: 2019-06-05 14:00:12
type: proceedings_section
metadata_visibility: show
creators_name: Zhang, J
creators_name: Zhang, LZ
creators_name: Hao, D
creators_name: Wang, M
creators_name: Zhang, L
title: Do Pseudo Test Suites Lead to Inflated Correlation in Measuring Test Effectiveness?
ispublished: pub
divisions: UCL
divisions: B04
divisions: C05
divisions: F48
keywords: test suites, coverage criteria, empirical study
note: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
abstract: Code coverage is the most widely adopted criteria for measuring test effectiveness in software quality assurance. The performance of coverage criteria (in indicating test suites’ effectiveness) has been widely studied in prior work. Most of the studies use randomly constructed pseudo test suites to facilitate data collection for correlation analysis, yet no previous work has systematically studied whether pseudo test suites would lead to inflated correlation results. This paper focuses on the potentially wide-spread threat with a study over 123 real-world Java projects. Following the typical experimental process of studying coverage criteria, we investigate the correlation between statement/assertion coverage and mutation score using both pseudo and original test suites. Except for direct correlation analysis, we control the number of assertions and the test suite size to conduct partial correlation analysis. The results reveal that 1) the correlation (between coverage criteria and mutation score) derived from pseudo test suites is much higher than from original test suites (from 0.21 to 0.39 higher in Kendall τb value); 2) contrary to previously reported, statement coverage has a stronger correlation with mutation score than assertion coverage.
date: 2019-06-06
date_type: published
publisher: IEEE
official_url: http://dx.doi.org/10.1109/ICST.2019.00033
oa_status: green
full_text_type: other
language: eng
primo: open
primo_central: open_green
verified: verified_manual
elements_id: 1661369
doi: 10.1109/ICST.2019.00033
lyricists_name: Zhang, Jie
lyricists_id: JZHAE79
actors_name: Zhang, Jie
actors_id: JZHAE79
actors_role: owner
full_text_status: public
publication: International Conference on Software Testing, Verification and Reliability
place_of_pub: Xi'an, China, China
pagerange: 252-263
event_title: 12th IEEE Conference on Software Testing, Validation and Verification (ICST)
event_location: Xi'an, China
event_dates: 24 April 2019 - 27 April 2019
institution: International Conference on Software Testing, Verification and Reliability (ICST)
issn: 0960-0833
book_title: Proceedings of the 12th IEEE Conference on Software Testing, Validation and Verification (ICST) 2019
citation:        Zhang, J;    Zhang, LZ;    Hao, D;    Wang, M;    Zhang, L;      (2019)    Do Pseudo Test Suites Lead to Inflated Correlation in Measuring Test Effectiveness?                     In:  Proceedings of the 12th IEEE Conference on Software Testing, Validation and Verification (ICST) 2019.  (pp. pp. 252-263).  IEEE: Xi'an, China, China.       Green open access   
 
document_url: https://discovery.ucl.ac.uk/id/eprint/10075196/7/Zhang_Do%20Pseudo%20Test%20Suites%20Lead%20to%20Inflated%20Correlation%20in%20Measuring%20Test%20Effectiveness_AAM.pdf