AppAgent: Multimodal Agents as Smartphone Users
Recent advancements in large language models (LLMs) have led to the creation
of intelligent agents capable of performing complex tasks. This paper
introduces a novel LLM-based multimodal agent framework designed to operate
smartphone applications. Our framework enables the agent to operate smartphone
applications through a simplified action space, mimicking human-like
interactions such as tapping and swiping. This novel approach bypasses the need
for system back-end access, thereby broadening its applicability across diverse
apps. Central to our agent's functionality is its innovative learning method.
The agent learns to navigate and use new apps either through autonomous
exploration or by observing human demonstrations. This process generates a
knowledge base that the agent refers to for executing complex tasks across
different applications. To demonstrate the practicality of our agent, we
conducted extensive testing over 50 tasks in 10 different applications,
including social media, email, maps, shopping, and sophisticated image editing
tools. The results affirm our agent's proficiency in handling a diverse array
of high-level tasks.
6585cc0d9747b6c745a14ef1
Chi Zhang
claimed_verified
2023-12-26T10:37:33.698Z
64196320ed725fef64419c2a
/avatars/96feb22fb5e8931d6c9e0ea06148266f.svg
Chi Zhang
user
DrChiZhang
65850d0069dfb7bfc5ef9dcc
Zhao Yang
65850d0069dfb7bfc5ef9dcd
Jiaxuan Liu
65850d0069dfb7bfc5ef9dce
Yucheng Han
admin_assigned
2023-12-22T10:58:07.217Z
655c65c7ae75c25ae7e9f183
https://cdn-avatars.huggingface.co/v1/production/uploads/655c65c7ae75c25ae7e9f183/DuodzOHTig1ui_TJ6Sjfd.jpeg
Yucheng Han
user
listen2you002
65850d0069dfb7bfc5ef9dcf
Xin Chen
claimed_verified
2023-12-26T10:37:39.494Z
63b543b2bda8d44adf2d7d0a
https://cdn-avatars.huggingface.co/v1/production/uploads/63b543b2bda8d44adf2d7d0a/G_AzO8iA-WwEcxvJEKVQ0.jpeg
Chen
user
ChrisChenXin
65850d0069dfb7bfc5ef9dd0
Zebiao Huang
65850d0069dfb7bfc5ef9dd1
Bin Fu
65850d0069dfb7bfc5ef9dd2
Gang Yu
admin_assigned
2023-12-22T10:59:08.698Z
63417332c5565a4b8d43a0d8
https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
Gang Yu
user
skicy
AppAgent: Multimodal Agents as Smartphone Users
Recent advancements in large language models (LLMs) have led to the creation
of intelligent agents capable of performing complex tasks. This paper
introduces a novel LLM-based multimodal agent framework designed to operate
smartphone applications. Our framework enables the agent to operate smartphone
applications through a simplified action space, mimicking human-like
interactions such as tapping and swiping. This novel approach bypasses the need
for system back-end access, thereby broadening its applicability across diverse
apps. Central to our agent's functionality is its innovative learning method.
The agent learns to navigate and use new apps either through autonomous
exploration or by observing human demonstrations. This process generates a
knowledge base that the agent refers to for executing complex tasks across
different applications. To demonstrate the practicality of our agent, we
conducted extensive testing over 50 tasks in 10 different applications,
including social media, email, maps, shopping, and sophisticated image editing
tools. The results affirm our agent's proficiency in handling a diverse array
of high-level tasks.
A novel LLM-based multimodal agent learns to operate smartphone apps through autonomous exploration or imitation, demonstrating proficiency across diverse tasks.
large language models multimodal agent autonomous exploration human demonstrations knowledge base high-level tasks