Petke, Justyna; Martinez, Matias; Kechagia, Maria; Aleti, Aldeida; Sarro, Federica; (2024) The Patch Overfitting Problem in Automated Program Repair: Practical Magnitude and a Baseline for Realistic Benchmarking. In: Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE Companion ’24). (pp. pp. 452-456). Association for Computing Machinery (ACM): New York, NY, USA.

Preview

Text Kechagia_The_Patch_Overfitting_Problem_3663529.3663776.pdf Download (164kB) | Preview

Abstract

Automated program repair techniques aim to generate patches for software bugs, mainly relying on testing to check their validity. The generation of a large number of such plausible yet incorrect patches is widely believed to hinder wider application of APR in practice, which has motivated research in automated patch assessment. We reflect on the validity of this motivation and carry out an empirical study to analyse the extent to which 10 APR tools suffer from the overfitting problem in practice. We observe that the number of plausible patches generated by any of the APR tools analysed for a given bug from the Defects4J dataset is remarkably low, a median of 2, indicating that a developer only needs to consider 2 patches in most cases to be confident to find a fix or confirming its nonexistence. This study unveils that the overfitting problem might not be as bad as previously thought. We reflect on current evaluation strategies of automated patch assessment techniques and propose a Random Selection baseline to assess whether and when using such techniques is beneficial for reducing human effort. We advocate future work should evaluate the benefit arising from patch overfitting assessment usage against the random baseline.

Download activity - last month

Export as CSVJSONXML

Download activity - last 12 months

Export as CSVJSONXML

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item