Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks
Smartphones have become indispensable in modern life, yet navigating complex
tasks on mobile devices often remains frustrating. Recent advancements in large
multimodal model (LMM)-based mobile agents have demonstrated the ability to
perceive and act in mobile environments. However, current approaches face
significant limitations: they fall short in addressing real-world human needs,
struggle with reasoning-intensive and long-horizon tasks, and lack mechanisms
to learn and improve from prior experiences. To overcome these challenges, we
introduce Mobile-Agent-E, a hierarchical multi-agent framework capable of
self-evolution through past experience. By hierarchical, we mean an explicit
separation of high-level planning and low-level action execution. The framework
comprises a Manager, responsible for devising overall plans by breaking down
complex tasks into subgoals, and four subordinate agents--Perceptor, Operator,
Action Reflector, and Notetaker--which handle fine-grained visual perception,
immediate action execution, error verification, and information aggregation,
respectively. Mobile-Agent-E also features a novel self-evolution module which
maintains a persistent long-term memory comprising Tips and Shortcuts. Tips are
general guidance and lessons learned from prior tasks on how to effectively
interact with the environment. Shortcuts are reusable, executable sequences of
atomic operations tailored for specific subroutines. The inclusion of Tips and
Shortcuts facilitates continuous refinement in performance and efficiency.
Alongside this framework, we introduce Mobile-Eval-E, a new benchmark featuring
complex mobile tasks requiring long-horizon, multi-app interactions. Empirical
results show that Mobile-Agent-E achieves a 22% absolute improvement over
previous state-of-the-art approaches across three foundation model backbones.
Project page: https://x-plug.github.io/MobileAgent.
6790791b203b95acf96ebf45
Zhenhailong Wang
extracted_pending
2025-01-22T04:50:40.468Z
628d7265db4cd1d1717c884f
/avatars/dff2a3dd10d84b4a73fa486402de7219.svg
Zhenhailong Wang
user
mikewang
6790791b203b95acf96ebf46
Haiyang Xu
claimed_verified
2025-01-22T10:08:13.360Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
6790791b203b95acf96ebf47
Junyang Wang
admin_assigned
2025-01-22T15:47:37.467Z
6438f6415aa69077ffb16942
/avatars/c83dbd3e10e88db97c2a86092bad5917.svg
Junyang Wang
user
junyangwang0410
6790791b203b95acf96ebf48
Xi Zhang
6790791b203b95acf96ebf49
Ming Yan
6790791b203b95acf96ebf4a
Ji Zhang
6790791b203b95acf96ebf4b
Fei Huang
6790791b203b95acf96ebf4c
Heng Ji
Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks
Smartphones have become indispensable in modern life, yet navigating complex
tasks on mobile devices often remains frustrating. Recent advancements in large
multimodal model (LMM)-based mobile agents have demonstrated the ability to
perceive and act in mobile environments. However, current approaches face
significant limitations: they fall short in addressing real-world human needs,
struggle with reasoning-intensive and long-horizon tasks, and lack mechanisms
to learn and improve from prior experiences. To overcome these challenges, we
introduce Mobile-Agent-E, a hierarchical multi-agent framework capable of
self-evolution through past experience. By hierarchical, we mean an explicit
separation of high-level planning and low-level action execution. The framework
comprises a Manager, responsible for devising overall plans by breaking down
complex tasks into subgoals, and four subordinate agents--Perceptor, Operator,
Action Reflector, and Notetaker--which handle fine-grained visual perception,
immediate action execution, error verification, and information aggregation,
respectively. Mobile-Agent-E also features a novel self-evolution module which
maintains a persistent long-term memory comprising Tips and Shortcuts. Tips are
general guidance and lessons learned from prior tasks on how to effectively
interact with the environment. Shortcuts are reusable, executable sequences of
atomic operations tailored for specific subroutines. The inclusion of Tips and
Shortcuts facilitates continuous refinement in performance and efficiency.
Alongside this framework, we introduce Mobile-Eval-E, a new benchmark featuring
complex mobile tasks requiring long-horizon, multi-app interactions. Empirical
results show that Mobile-Agent-E achieves a 22% absolute improvement over
previous state-of-the-art approaches across three foundation model backbones.
Project page: https://x-plug.github.io/MobileAgent.
A hierarchical multi-agent framework called Mobile-Agent-E enhances mobile task navigation through self-evolution, improving performance and efficiency on complex long-horizon tasks.
large multimodal model (LMM) mobile agents hierarchical multi-agent framework self-evolution Manager Perceptor Operator Action Reflector Notetaker long-term memory Tips Shortcuts Mobile-Eval-E long-horizon multi-app interactions