Dakhama, Aidan;
Even-Mendoza, Karine;
Langdon, WB;
Menéndez, Héctor D;
Petke, Justyna;
(2025)
Enhancing search-based testing with LLMs for finding bugs in system simulators.
Automated Software Engineering
, 32
, Article 63. 10.1007/s10515-025-00531-7.
Preview |
Text
Petke_s10515-025-00531-7.pdf Download (4MB) | Preview |
Abstract
Despite the wide availability of automated testing techniques such as fuzzing, little attention has been devoted to testing computer architecture simulators. We propose a fully automated approach for this task. Our approach uses large language models (LLM) to generate input programs, including information about their parameters and types, as test cases for the simulators. The LLM’s output becomes the initial seed for an existing fuzzer, AFL++, which has been enhanced with three mutation operators, targeting both the input binary program and its parameters. We implement our approach in a tool called SearchSYS . We use it to test the gem5 system simulator. SearchSYS discovered 21 new bugs in gem5 , 14 where gem5 ’s software prediction differs from the real behaviour on actual hardware, and 7 where it crashed. New defects were uncovered with each of the 6 LLMs used.
| Type: | Article |
|---|---|
| Title: | Enhancing search-based testing with LLMs for finding bugs in system simulators |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/s10515-025-00531-7 |
| Publisher version: | https://doi.org/10.1007/s10515-025-00531-7 |
| Language: | English |
| Additional information: | This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
| Keywords: | Fuzzing; Differential back-to-back testing; Systems; ISA x86; LLM; Code LLM; Tiny LLM; ANN; SBSE; SBFT; Genetic improvement of tests; Gem5; LLM in software engineering; TinyLlama; Phi2; Llama2; Magicoder; CodeBooga; GPT-3.5-turbo |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10209206 |
Archive Staff Only
 |
View Item |
