How Far Can We Go with Practical Function-Level Program Repair?
Recently, multiple Automated Program Repair (APR) techniques based on Large
Language Models (LLMs) have been proposed to enhance the repair performance.
While these techniques mainly focus on the single-line or hunk-level repair,
they face significant challenges in real-world application due to the limited
repair task scope and costly statement-level fault localization. However, the
more practical function-level APR, which broadens the scope of APR task to fix
entire buggy functions and requires only cost-efficient function-level fault
localization, remains underexplored. In this paper, we conduct the first
comprehensive study of LLM-based function-level APR including investigating the
effect of the few-shot learning mechanism and the auxiliary repair-relevant
information. Specifically, we adopt six widely-studied LLMs and construct a
benchmark in both the Defects4J 1.2 and 2.0 datasets. Our study demonstrates
that LLMs with zero-shot learning are already powerful function-level APR
techniques, while applying the few-shot learning mechanism leads to disparate
repair performance. Moreover, we find that directly applying the auxiliary
repair-relevant information to LLMs significantly increases function-level
repair performance. Inspired by our findings, we propose an LLM-based
function-level APR technique, namely SRepair, which adopts a dual-LLM framework
to leverage the power of the auxiliary repair-relevant information for
advancing the repair performance. The evaluation results demonstrate that
SRepair can correctly fix 300 single-function bugs in the Defects4J dataset,
largely surpassing all previous APR techniques by at least 85%, without the
need for the costly statement-level fault location information. Furthermore,
SRepair successfully fixes 32 multi-function bugs in the Defects4J dataset,
which is the first time achieved by any APR technique ever to our best
knowledge.
6625f4a026089aaae05d5978
Jiahong Xiang
claimed_verified
2024-04-23T07:15:24.793Z
662673feea153ffe143eb154
/avatars/416127c5fd920d692bbe862bc3344a73.svg
Jiahong Xiang
user
GhabiX
6625f4a026089aaae05d5979
Xiaoyang Xu
admin_assigned
2024-04-22T14:07:33.668Z
6501215a85a884a964dc0ded
/avatars/1ca84b3d19d4ac4da9c67ca1627bf186.svg
xiaoyang xu
user
xuxy
6625f4a026089aaae05d597a
Fanchu Kong
6625f4a026089aaae05d597b
Mingyuan Wu
6625f4a026089aaae05d597c
Haotian Zhang
6625f4a026089aaae05d597d
Yuqun Zhang
How Far Can We Go with Practical Function-Level Program Repair?
Recently, multiple Automated Program Repair (APR) techniques based on Large
Language Models (LLMs) have been proposed to enhance the repair performance.
While these techniques mainly focus on the single-line or hunk-level repair,
they face significant challenges in real-world application due to the limited
repair task scope and costly statement-level fault localization. However, the
more practical function-level APR, which broadens the scope of APR task to fix
entire buggy functions and requires only cost-efficient function-level fault
localization, remains underexplored. In this paper, we conduct the first
comprehensive study of LLM-based function-level APR including investigating the
effect of the few-shot learning mechanism and the auxiliary repair-relevant
information. Specifically, we adopt six widely-studied LLMs and construct a
benchmark in both the Defects4J 1.2 and 2.0 datasets. Our study demonstrates
that LLMs with zero-shot learning are already powerful function-level APR
techniques, while applying the few-shot learning mechanism leads to disparate
repair performance. Moreover, we find that directly applying the auxiliary
repair-relevant information to LLMs significantly increases function-level
repair performance. Inspired by our findings, we propose an LLM-based
function-level APR technique, namely SRepair, which adopts a dual-LLM framework
to leverage the power of the auxiliary repair-relevant information for
advancing the repair performance. The evaluation results demonstrate that
SRepair can correctly fix 300 single-function bugs in the Defects4J dataset,
largely surpassing all previous APR techniques by at least 85%, without the
need for the costly statement-level fault location information. Furthermore,
SRepair successfully fixes 32 multi-function bugs in the Defects4J dataset,
which is the first time achieved by any APR technique ever to our best
knowledge.
A study reveals that LLMs, particularly those using zero-shot learning and auxiliary repair information, outperform traditional APR techniques in function-level repairs, especially when using a dual-LLM framework.
Large Language Models Automated Program Repair few-shot learning zero-shot learning Defects4J auxiliary repair-relevant information dual-LLM framework