D-Artemis: A Deliberative Cognitive Framework for Mobile GUI
  Multi-Agents
Graphical User Interface (GUI) agents aim to automate a wide spectrum of
human tasks by emulating user interaction. Despite rapid advancements, current
approaches are hindered by several critical challenges: data bottleneck in
end-to-end training, high cost of delayed error detection, and risk of
contradictory guidance. Inspired by the human cognitive loop of Thinking,
Alignment, and Reflection, we present D-Artemis -- a novel deliberative
framework in this paper. D-Artemis leverages a fine-grained, app-specific tip
retrieval mechanism to inform its decision-making process. It also employs a
proactive Pre-execution Alignment stage, where Thought-Action Consistency (TAC)
Check module and Action Correction Agent (ACA) work in concert to mitigate the
risk of execution failures. A post-execution Status Reflection Agent (SRA)
completes the cognitive loop, enabling strategic learning from experience.
Crucially, D-Artemis enhances the capabilities of general-purpose Multimodal
large language models (MLLMs) for GUI tasks without the need for training on
complex trajectory datasets, demonstrating strong generalization. D-Artemis
establishes new state-of-the-art (SOTA) results across both major benchmarks,
achieving a 75.8% success rate on AndroidWorld and 96.8% on ScreenSpot-V2.
Extensive ablation studies further demonstrate the significant contribution of
each component to the framework.
68d9f61a0177a6054b013b2f
Hongze Mi
68d9f61a0177a6054b013b30
Yibo Feng
68d9f61a0177a6054b013b31
Wenjie Lu
68d9f61a0177a6054b013b32
Yuqi Wang
68d9f61a0177a6054b013b33
Jinyuan Li
68d9f61a0177a6054b013b34
Song Cao
68d9f61a0177a6054b013b35
He Cui
68d9f61a0177a6054b013b36
Tengfei Tian
68d9f61a0177a6054b013b37
Xuelin Zhang
68d9f61a0177a6054b013b38
Haotian Luo
68d9f61a0177a6054b013b39
Di Sun
68d9f61a0177a6054b013b3a
Naiqiang Tan
68d9f61a0177a6054b013b3b
Gang Pan
D-Artemis: A Deliberative Cognitive Framework for Mobile GUI
  Multi-Agents
Graphical User Interface (GUI) agents aim to automate a wide spectrum of
human tasks by emulating user interaction. Despite rapid advancements, current
approaches are hindered by several critical challenges: data bottleneck in
end-to-end training, high cost of delayed error detection, and risk of
contradictory guidance. Inspired by the human cognitive loop of Thinking,
Alignment, and Reflection, we present D-Artemis -- a novel deliberative
framework in this paper. D-Artemis leverages a fine-grained, app-specific tip
retrieval mechanism to inform its decision-making process. It also employs a
proactive Pre-execution Alignment stage, where Thought-Action Consistency (TAC)
Check module and Action Correction Agent (ACA) work in concert to mitigate the
risk of execution failures. A post-execution Status Reflection Agent (SRA)
completes the cognitive loop, enabling strategic learning from experience.
Crucially, D-Artemis enhances the capabilities of general-purpose Multimodal
large language models (MLLMs) for GUI tasks without the need for training on
complex trajectory datasets, demonstrating strong generalization. D-Artemis
establishes new state-of-the-art (SOTA) results across both major benchmarks,
achieving a 75.8% success rate on AndroidWorld and 96.8% on ScreenSpot-V2.
Extensive ablation studies further demonstrate the significant contribution of
each component to the framework.
D-Artemis, a novel deliberative framework, enhances GUI automation by leveraging app-specific tips, proactive alignment, and reflection, achieving state-of-the-art results with general-purpose multimodal large language models.
deliberative framework fine-grained app-specific tip retrieval Thought-Action Consistency (TAC) Check Action Correction Agent (ACA) Status Reflection Agent (SRA) Multimodal large language models (MLLMs) GUI tasks AndroidWorld ScreenSpot-V2