Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered
  Smartphone Agents
Smartphones bring significant convenience to users but also enable devices to
extensively record various types of personal information. Existing smartphone
agents powered by Multimodal Large Language Models (MLLMs) have achieved
remarkable performance in automating different tasks. However, as the cost,
these agents are granted substantial access to sensitive users' personal
information during this operation. To gain a thorough understanding of the
privacy awareness of these agents, we present the first large-scale benchmark
encompassing 7,138 scenarios to the best of our knowledge. In addition, for
privacy context in scenarios, we annotate its type (e.g., Account Credentials),
sensitivity level, and location. We then carefully benchmark seven available
mainstream smartphone agents. Our results demonstrate that almost all
benchmarked agents show unsatisfying privacy awareness (RA), with performance
remaining below 60% even with explicit hints. Overall, closed-source agents
show better privacy ability than open-source ones, and Gemini 2.0-flash
achieves the best, achieving an RA of 67%. We also find that the agents'
privacy detection capability is highly related to scenario sensitivity level,
i.e., the scenario with a higher sensitivity level is typically more
identifiable. We hope the findings enlighten the research community to rethink
the unbalanced utility-privacy tradeoff about smartphone agents. Our code and
benchmark are available at https://zhixin-l.github.io/SAPA-Bench.
68afb4c9245176306494cbbd
Zhixin Lin
68afb4c9245176306494cbbe
Jungang Li
claimed_verified
2025-08-28T08:55:04.751Z
64b76528fdb702b3d8641514
/avatars/dcf7987e4c54f11fef0459c46f05ed62.svg
Jungang Li
user
Jungang
68afb4c9245176306494cbbf
Shidong Pan
68afb4c9245176306494cbc0
Yibo Shi
68afb4c9245176306494cbc1
Yue Yao
68afb4c9245176306494cbc2
Dongliang Xu
Mind the Third Eye! Benchmarking Privacy Awareness in MLLM-powered
  Smartphone Agents
Smartphones bring significant convenience to users but also enable devices to
extensively record various types of personal information. Existing smartphone
agents powered by Multimodal Large Language Models (MLLMs) have achieved
remarkable performance in automating different tasks. However, as the cost,
these agents are granted substantial access to sensitive users' personal
information during this operation. To gain a thorough understanding of the
privacy awareness of these agents, we present the first large-scale benchmark
encompassing 7,138 scenarios to the best of our knowledge. In addition, for
privacy context in scenarios, we annotate its type (e.g., Account Credentials),
sensitivity level, and location. We then carefully benchmark seven available
mainstream smartphone agents. Our results demonstrate that almost all
benchmarked agents show unsatisfying privacy awareness (RA), with performance
remaining below 60% even with explicit hints. Overall, closed-source agents
show better privacy ability than open-source ones, and Gemini 2.0-flash
achieves the best, achieving an RA of 67%. We also find that the agents'
privacy detection capability is highly related to scenario sensitivity level,
i.e., the scenario with a higher sensitivity level is typically more
identifiable. We hope the findings enlighten the research community to rethink
the unbalanced utility-privacy tradeoff about smartphone agents. Our code and
benchmark are available at https://zhixin-l.github.io/SAPA-Bench.
A large-scale benchmark evaluates the privacy awareness of smartphone agents powered by Multimodal Large Language Models, revealing significant gaps in their ability to protect sensitive user information.
Multimodal Large Language Models privacy awareness benchmark privacy context scenario sensitivity level privacy detection capability utility-privacy tradeoff