LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration
  Benchmark
Mobile GUI agents show promise in automating tasks but face generalization
challenges in diverse real-world scenarios. Traditional approaches using
pre-training or fine-tuning with massive datasets struggle with the diversity
of mobile applications and user-specific tasks. We propose enhancing mobile GUI
agent capabilities through human demonstrations, focusing on improving
performance in unseen scenarios rather than pursuing universal generalization
through larger datasets. To realize this paradigm, we introduce LearnGUI, the
first comprehensive dataset specifically designed for studying
demonstration-based learning in mobile GUI agents, comprising 2,252 offline
tasks and 101 online tasks with high-quality human demonstrations. We further
develop LearnAct, a sophisticated multi-agent framework that automatically
extracts knowledge from demonstrations to enhance task completion. This
framework integrates three specialized agents: DemoParser for knowledge
extraction, KnowSeeker for relevant knowledge retrieval, and ActExecutor for
demonstration-enhanced task execution. Our experimental results show
significant performance gains in both offline and online evaluations. In
offline assessments, a single demonstration improves model performance,
increasing Gemini-1.5-Pro's accuracy from 19.3% to 51.7%. In online
evaluations, our framework enhances UI-TARS-7B-SFT's task success rate from
18.1% to 32.8%. LearnAct framework and LearnGUI benchmark establish
demonstration-based learning as a promising direction for more adaptable,
personalized, and deployable mobile GUI agents.
680662b97415e191e3579b9e
Guangyi Liu
claimed_verified
2025-04-22T09:52:04.194Z
64d761b98ebc40443831f82a
https://cdn-avatars.huggingface.co/v1/production/uploads/64d761b98ebc40443831f82a/DHBOtOstiFp2-lDY6b9gb.png
Guangyi Liu
user
lgy0404
680662b97415e191e3579b9f
Pengxiang Zhao
claimed_verified
2025-04-22T09:52:01.390Z
65a088f4300957620ba45c70
/avatars/56ed45e10d3455531979f30881b2d3f9.svg
pengxiang zhao
user
Pengxiangzhao
680662b97415e191e3579ba0
Liang Liu
680662b97415e191e3579ba1
Zhiming Chen
admin_assigned
2025-04-22T11:01:08.419Z
67c26500918bb8d89b01d6a3
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/OJROx6Q4Z2rnW5eOiqvm3.png
Zhiming Chen
user
zhimingc
680662b97415e191e3579ba2
Yuxiang Chai
admin_assigned
2025-04-22T11:01:01.122Z
6458ce236fa580137af5aa95
/avatars/db65a7332e375eb5daad5c1b076b1e3b.svg
Yuxiang Chai
user
Yuxiang007
680662b97415e191e3579ba3
Shuai Ren
680662b97415e191e3579ba4
Hao Wang
680662b97415e191e3579ba5
Shibo He
admin_assigned
2025-04-22T11:00:42.823Z
66604211253289136b6c42d6
/avatars/b2ea4870d45efea01d4c2f34a2cd96bd.svg
SHIBO HE
user
dkeeeee
680662b97415e191e3579ba6
Wenchao Meng
LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration
  Benchmark
Mobile GUI agents show promise in automating tasks but face generalization
challenges in diverse real-world scenarios. Traditional approaches using
pre-training or fine-tuning with massive datasets struggle with the diversity
of mobile applications and user-specific tasks. We propose enhancing mobile GUI
agent capabilities through human demonstrations, focusing on improving
performance in unseen scenarios rather than pursuing universal generalization
through larger datasets. To realize this paradigm, we introduce LearnGUI, the
first comprehensive dataset specifically designed for studying
demonstration-based learning in mobile GUI agents, comprising 2,252 offline
tasks and 101 online tasks with high-quality human demonstrations. We further
develop LearnAct, a sophisticated multi-agent framework that automatically
extracts knowledge from demonstrations to enhance task completion. This
framework integrates three specialized agents: DemoParser for knowledge
extraction, KnowSeeker for relevant knowledge retrieval, and ActExecutor for
demonstration-enhanced task execution. Our experimental results show
significant performance gains in both offline and online evaluations. In
offline assessments, a single demonstration improves model performance,
increasing Gemini-1.5-Pro's accuracy from 19.3% to 51.7%. In online
evaluations, our framework enhances UI-TARS-7B-SFT's task success rate from
18.1% to 32.8%. LearnAct framework and LearnGUI benchmark establish
demonstration-based learning as a promising direction for more adaptable,
personalized, and deployable mobile GUI agents.
The LearnAct framework and LearnGUI dataset improve mobile GUI agent performance through demonstration-based learning, enhancing task success rates in both offline and online evaluations.
LearnGUI LearnAct multi-agent framework DemoParser KnowSeeker ActExecutor demonstration-based learning