Menendez, HD;
Boreale, M;
Gorla, D;
Clark, D;
(2020)
Output Sampling for Output Diversity in Automatic Unit Test Generation.
IEEE Transactions on Software Engineering
10.1109/TSE.2020.2987377.
(In press).
Preview |
Text
outgen-main.pdf - Accepted Version Download (642kB) | Preview |
Abstract
Diverse test sets are able to expose bugs that test sets generated with structural coverage techniques cannot discover. Input-diverse test set generators have been shown to be effective for this, but also have limitations: e.g., they need to be complemented with semantic information derived from the Software Under Test. We demonstrate how to drive the test set generation process with semantic information in the form of output diversity. We present the first totally automatic output sampling for output diversity unit test set generation tool, called OutGen. OutGen transforms a program into an SMT formula in bit-vector arithmetic. It then applies universal hashing in order to generate an output-based diverse set of inputs. The result offers significant diversity improvements when measured as a high output uniqueness count. It achieves this by ensuring that the test set's output probability distribution is uniform, i.e. highly diverse. The use of output sampling, as opposed to any of input sampling, CBMC, CAVM, behaviour diversity or random testing improves mutation score and bug detection by up to 4150% and 963% respectively on programs drawn from three different corpora: the R-project, SIR and CodeFlaws. OutGen test sets achieve an average mutation score of up to 92%, and 70% of the test sets detect the defect. Moreover, OutGen is the only automatic unit test generation tool that is able to detect bugs on the real number C functions from the R-project.
| Type: | Article |
|---|---|
| Title: | Output Sampling for Output Diversity in Automatic Unit Test Generation |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1109/TSE.2020.2987377 |
| Publisher version: | https://doi.org/10.1109/TSE.2020.2987377 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Unit Testing, Output Sampling, Output Diversity, SMT Solver, OutGen |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10094844 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
