Jia, Haoxiang;
Morris, Robbie;
Ye, He;
Sarro, Federica;
Mechtaev, Sergey;
(2025)
Automated Repair of Ambiguous Natural Language Requirements.
In:
Proceedings of the The 40th IEEE/ACM International Conference on Automated Software Engineering.
IEEE
(In press).
Preview |
PDF
2505.07270v3.pdf - Accepted Version Download (617kB) | Preview |
Abstract
The growing use of large language models (LLMs) has increased the importance of natural language (NL) in software engineering. However, ambiguity of NL can harm software quality, as unclear problem descriptions may lead to incorrect program generation. Detecting and resolving such ambiguity is challenging, motivating our introduction of the automated repair of ambiguous NL descriptions, which we approach by reducing code generation uncertainty and better aligning NL with input–output examples. Ambiguity repair is difficult for LLMs because they must understand how their interpretation of a description changes when the text is altered. We find that directly prompting LLMs to clarify ambiguity often produces irrelevant or inconsistent edits. To address this, we decompose this task into two simpler steps: (1) analyzing and repairing the LLM’s interpretation of the description — captured by the distribution of programs it induces — using traditional testing and program repair, and (2) refining the description based on distribution changes via a method we call contrastive specification inference. We implement this approach in a tool called SPECFIX and evaluate it using four state-of-the-art LLMs (GPT-4o, GPT-4o-mini, DeepSeek-V3, and Qwen2.5-Coder-32B-Instruct) on three popular code generation benchmarks (HumanEval+, MBPP+ and LiveCodeBench). Without human intervention or external information, SPECFIX modified 43.58% of descriptions, improving Pass@1 on the modified set by 30.9%. This yields a 4.09% absolute improvement across the entire benchmark. Repairs also transfer across models: descriptions repaired for one model improve other models’ performance by 10.48%.
| Type: | Proceedings paper |
|---|---|
| Title: | Automated Repair of Ambiguous Natural Language Requirements |
| Event: | The 40th IEEE/ACM International Conference on Automated Software Engineering |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://conf.researchr.org/home/ase-2025 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10216692 |
Archive Staff Only
 |
View Item |
