Brownlee, Alexander EI;
Callan, James;
Even-Mendoza, Karine;
Geiger, Alina;
Hanna, Carol;
Petke, Justyna;
Sarro, Federica;
(2025)
Large Language Model Based Mutations in Genetic Improvement.
Automated Software Engineering
, 32
(15)
pp. 1-25.
10.1007/s10515-024-00473-6.
Preview |
Text
Hanna_Large Language Model Based Mutations in Genetic Improvement_VoR.pdf Download (2MB) | Preview |
Abstract
Ever since the first large language models (LLMs) have become available, both academics and practitioners have used them to aid software engineering tasks. However, little research as yet has been done in combining search-based software engineering (SBSE) and LLMs. In this paper, we evaluate the use of LLMs as mutation operators for genetic improvement (GI), an SBSE approach, to improve the GI search process. In a preliminary work, we explored the feasibility of combining the Gin Java GI toolkit with OpenAI LLMs in order to generate an edit for the JCodec tool. Here we extend this investigation involving three LLMs and three types of prompt, and five real-world software projects. We sample the edits at random, as well as using local search. We also conducted a qualitative analysis to understand why LLM-generated code edits break as part of our evaluation. Our results show that, compared with conventional statement GI edits, LLMs produce fewer unique edits, but these compile and pass tests more often, with the OpenAI model finding test-passing edits 77% of the time. The OpenAI and Mistral LLMs are roughly equal in finding the best run-time improvements. Simpler prompts are more successful than those providing more context and examples. The qualitative analysis reveals a wide variety of areas where LLMs typically fail to produce valid edits commonly including inconsistent formatting, generating non-Java syntax, or refusing to provide a solution.
| Type: | Article |
|---|---|
| Title: | Large Language Model Based Mutations in Genetic Improvement |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/s10515-024-00473-6 |
| Publisher version: | https://doi.org/10.1007/s10515-024-00473-6 |
| Language: | English |
| Additional information: | © 2025 Springer Nature. This article is licensed under a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). |
| Keywords: | Large language models, Genetic Improvement |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10198293 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
