Foster, Christopher;
Gulati, Abhishek;
Harman, Mark;
Harper, Inna;
Mao, Ke;
Ritchey, Jilian;
Robert, Hervé;
(2025)
Mutation-Guided LLM-based Test Generation at Meta.
In:
Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering.
(pp. pp. 180-191).
ACM (Association for Computing Machinery): New York, NY, United States.
Preview |
PDF
fse25_industry_paper_arXiv.pdf - Published Version Download (676kB) | Preview |
Abstract
This paper describes Meta's Automated Compliance Hardening (ACH) system for mutation-guided LLM-based test generation. ACH generates relatively few mutants (aka simulated faults), compared to traditional mutation testing. Instead, it focuses on generating currently undetected faults that are specific to an issue of concern. From these currently uncaught faults, ACH generates tests that can catch them, thereby 'killing' the mutants and consequently hardening the platform against regressions. We use privacy concerns to illustrate our approach, but ACH can harden code against any type of regression. In total, ACH was applied to 10,795 Android Kotlin classes in 7 software platforms deployed by Meta, from which it generated 9,095 mutants and 571 privacy-hardening test cases. ACH also deploys an LLM-based equivalent mutant detection agent that achieves a precision of 0.79 and a recall of 0.47 (rising to 0.95 and 0.96 with simple pre-processing). ACH was used in Messenger and WhatsApp test-a-thons where engineers accepted 73% of its tests, judging 36% to privacy relevant. We conclude that ACH hardens code against specific concerns and that, even when its tests do not directly tackle the specific concern, engineers find them useful for their other benefits.
| Type: | Proceedings paper |
|---|---|
| Title: | Mutation-Guided LLM-based Test Generation at Meta |
| Event: | FSE Companion '25: 33rd ACM International Conference on the Foundations of Software Engineering |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3696630.3728544 |
| Publisher version: | https://doi.org/10.1145/3696630.3728544 |
| Language: | English |
| Additional information: | This version is the author-accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Unit Testing, Automated Test Generation, Large Language Models, LLMs |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10218052 |
Archive Staff Only
 |
View Item |
