Code Agent can be an End-to-end System Hacker: Benchmarking Real-world
  Threats of Computer-use Agent
Computer-use agent (CUA) frameworks, powered by large language models (LLMs)
or multimodal LLMs (MLLMs), are rapidly maturing as assistants that can
perceive context, reason, and act directly within software environments. Among
their most critical applications is operating system (OS) control. As CUAs in
the OS domain become increasingly embedded in daily operations, it is
imperative to examine their real-world security implications, specifically
whether CUAs can be misused to perform realistic, security-relevant attacks.
Existing works exhibit four major limitations: Missing attacker-knowledge model
on tactics, techniques, and procedures (TTP), Incomplete coverage for
end-to-end kill chains, unrealistic environment without multi-host and
encrypted user credentials, and unreliable judgment dependent on
LLM-as-a-Judge. To address these gaps, we propose AdvCUA, the first benchmark
aligned with real-world TTPs in MITRE ATT&CK Enterprise Matrix, which comprises
140 tasks, including 40 direct malicious tasks, 74 TTP-based malicious tasks,
and 26 end-to-end kill chains, systematically evaluates CUAs under a realistic
enterprise OS security threat in a multi-host environment sandbox by hard-coded
evaluation. We evaluate the existing five mainstream CUAs, including ReAct,
AutoGPT, Gemini CLI, Cursor CLI, and Cursor IDE based on 8 foundation LLMs. The
results demonstrate that current frontier CUAs do not adequately cover OS
security-centric threats. These capabilities of CUAs reduce dependence on
custom malware and deep domain expertise, enabling even inexperienced attackers
to mount complex enterprise intrusions, which raises social concern about the
responsibility and security of CUAs.
68e717967ae125f9582e695b
Weidi Luo
68e717967ae125f9582e695c
Qiming Zhang
claimed_verified
2025-10-10T09:32:21.014Z
66a8ef0c7cda19fabead1340
/avatars/1a7b776c327cda5190170c32a9b23694.svg
Qiming Zhang
user
TerryZhang08
68e717967ae125f9582e695d
Tianyu Lu
68e717967ae125f9582e695e
Xiaogeng Liu
68e717967ae125f9582e695f
Bin Hu
claimed_verified
2025-10-11T13:49:03.747Z
63d73c646f49aa82306aed0f
/avatars/f6d755d1e304368619c277af2ada937c.svg
Bin Hu
user
binhu
68e717967ae125f9582e6960
Hung-Chun Chiu
68e717967ae125f9582e6961
Siyuan Ma
claimed_verified
2025-10-14T07:32:38.660Z
6552ec05ab7c20ac6fe5d789
/avatars/baab891b051a5ef64334ef13e3fab67b.svg
Siyuan Ma
user
HuggingFriends
68e717967ae125f9582e6962
Yizhe Zhang
68e717967ae125f9582e6963
Xusheng Xiao
68e717967ae125f9582e6964
Yinzhi Cao
68e717967ae125f9582e6965
Zhen Xiang
68e717967ae125f9582e6966
Chaowei Xiao
Code Agent can be an End-to-end System Hacker: Benchmarking Real-world
  Threats of Computer-use Agent
Computer-use agent (CUA) frameworks, powered by large language models (LLMs)
or multimodal LLMs (MLLMs), are rapidly maturing as assistants that can
perceive context, reason, and act directly within software environments. Among
their most critical applications is operating system (OS) control. As CUAs in
the OS domain become increasingly embedded in daily operations, it is
imperative to examine their real-world security implications, specifically
whether CUAs can be misused to perform realistic, security-relevant attacks.
Existing works exhibit four major limitations: Missing attacker-knowledge model
on tactics, techniques, and procedures (TTP), Incomplete coverage for
end-to-end kill chains, unrealistic environment without multi-host and
encrypted user credentials, and unreliable judgment dependent on
LLM-as-a-Judge. To address these gaps, we propose AdvCUA, the first benchmark
aligned with real-world TTPs in MITRE ATT&CK Enterprise Matrix, which comprises
140 tasks, including 40 direct malicious tasks, 74 TTP-based malicious tasks,
and 26 end-to-end kill chains, systematically evaluates CUAs under a realistic
enterprise OS security threat in a multi-host environment sandbox by hard-coded
evaluation. We evaluate the existing five mainstream CUAs, including ReAct,
AutoGPT, Gemini CLI, Cursor CLI, and Cursor IDE based on 8 foundation LLMs. The
results demonstrate that current frontier CUAs do not adequately cover OS
security-centric threats. These capabilities of CUAs reduce dependence on
custom malware and deep domain expertise, enabling even inexperienced attackers
to mount complex enterprise intrusions, which raises social concern about the
responsibility and security of CUAs.
AdvCUA benchmarks the security implications of computer-use agents in operating systems by evaluating their capabilities against real-world attack tactics and techniques in a multi-host environment.
large language models multimodal LLMs computer-use agents operating system control security implications tactics techniques procedures MITRE ATT&CK Enterprise Matrix end-to-end kill chains multi-host environment enterprise OS security ReAct AutoGPT Gemini CLI Cursor CLI Cursor IDE