When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches
  Generated by Code Agents?
Code agents are increasingly trusted to autonomously fix bugs on platforms
such as GitHub, yet their security evaluation focuses almost exclusively on
functional correctness. In this paper, we reveal a novel type of threat to
real-world code agents: Functionally Correct yet Vulnerable (FCV) patches,
which pass all test cases but contain vulnerable code. With our proposed
FCV-Attack, which can be deliberately crafted by malicious attackers or
implicitly introduced by benign developers, we show that SOTA LLMs (e.g.,
ChatGPT and Claude) and agent scaffolds (e.g., SWE-agent and OpenHands) are all
vulnerable to this FCV threat; across 12 agent-model combinations on SWE-Bench,
the attack only requires black-box access and a single query to the code agent
to perform the attack. For example, for CWE-538 (information exposure
vulnerability), the FCV-Attack attains an attack success rate of 40.7% on
GPT-5 Mini + OpenHands. Our results reveal an important security threat
overlooked by current evaluation paradigms and urge the development of
security-aware defenses for code agents.
68f94e42b9b2e4ae046736b1
Yibo Peng
68f94e42b9b2e4ae046736b2
James Song
68f94e42b9b2e4ae046736b3
Lei Li
68f94e42b9b2e4ae046736b4
Xinyu Yang
68f94e42b9b2e4ae046736b5
Mihai Christodorescu
68f94e42b9b2e4ae046736b6
Ravi Mangal
68f94e42b9b2e4ae046736b7
Corina Pasareanu
68f94e42b9b2e4ae046736b8
Haizhong Zheng
68f94e42b9b2e4ae046736b9
Beidi Chen
When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches
  Generated by Code Agents?
Code agents are increasingly trusted to autonomously fix bugs on platforms
such as GitHub, yet their security evaluation focuses almost exclusively on
functional correctness. In this paper, we reveal a novel type of threat to
real-world code agents: Functionally Correct yet Vulnerable (FCV) patches,
which pass all test cases but contain vulnerable code. With our proposed
FCV-Attack, which can be deliberately crafted by malicious attackers or
implicitly introduced by benign developers, we show that SOTA LLMs (e.g.,
ChatGPT and Claude) and agent scaffolds (e.g., SWE-agent and OpenHands) are all
vulnerable to this FCV threat; across 12 agent-model combinations on SWE-Bench,
the attack only requires black-box access and a single query to the code agent
to perform the attack. For example, for CWE-538 (information exposure
vulnerability), the FCV-Attack attains an attack success rate of 40.7% on
GPT-5 Mini + OpenHands. Our results reveal an important security threat
overlooked by current evaluation paradigms and urge the development of
security-aware defenses for code agents.
FCV-Attack exploits functionally correct yet vulnerable patches in code agents, demonstrating a significant security threat to current evaluation methods.
Functionally Correct yet Vulnerable (FCV) patches FCV-Attack SOTA LLMs agent scaffolds SWE-Bench CWE-538 information exposure vulnerability