Sobania, Dominik;
Briesch, Martin;
Hanna, Carol;
Petke, Justyna;
(2023)
An Analysis of the Automatic Bug Fixing Performance of ChatGPT.
In:
Proceedings of the 4th International Workshop on Automated Program Repair (APR 2023).
(pp. pp. 23-30).
IEEE/ACM: Melbourne, Australia.
Preview |
Text
Petke_conference_101719.pdf Download (810kB) | Preview |
Abstract
To support software developers in finding and fixing software bugs, several automated program repair techniques have been introduced. Given a test suite, standard methods usually either synthesize a repair, or navigate a search space of software edits to find test-suite passing variants. Recent program repair methods are based on deep learning approaches. One of these novel methods, which is not primarily intended for automated program repair, but is still suitable for it, is ChatGPT. The bug fixing performance of ChatGPT, however, is so far unclear. Therefore, in this paper we evaluate ChatGPT on the standard bug fixing benchmark set, QuixBugs, and compare the performance with the results of several other approaches reported in the literature. We find that ChatGPT's bug fixing performance is competitive to the common deep learning approaches CoCoNut and Codex and notably better than the results reported for the standard program repair approaches. In contrast to previous approaches, ChatGPT offers a dialogue system through which further information, e.g., the expected output for a certain input or an observed error message, can be entered. By providing such hints to ChatGPT, its success rate can be further increased, fixing 31 out of 40 bugs, outperforming state-of-the-art.
| Type: | Proceedings paper |
|---|---|
| Title: | An Analysis of the Automatic Bug Fixing Performance of ChatGPT |
| Event: | International Automated Program Repair Workshop 2023 (APR'23) co-located with the International Conference on Software Engineering (ICSE) |
| ISBN-13: | 979-8-3503-0214-1 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1109/APR59189.2023.00012 |
| Publisher version: | https://doi.org/10.1109/APR59189.2023.00012 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10165581 |
Archive Staff Only
 |
View Item |
