AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
AI agents have drawn increasing attention mostly on their ability to perceive
environments, understand tasks, and autonomously achieve goals. To advance
research on AI agents in mobile scenarios, we introduce the Android
Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for
generalist mobile GUI-control agents. Their capabilities of completing complex
tasks by directly interacting with the graphical user interface (GUI) on mobile
devices are trained and evaluated with the proposed dataset. AMEX comprises
over 104K high-resolution screenshots from 110 popular mobile applications,
which are annotated at multiple levels. Unlike existing mobile device-control
datasets, e.g., MoTIF, AitW, etc., AMEX includes three levels of annotations:
GUI interactive element grounding, GUI screen and element functionality
descriptions, and complex natural language instructions, each averaging 13
steps with stepwise GUI-action chains. We develop this dataset from a more
instructive and detailed perspective, complementing the general settings of
existing datasets. Additionally, we develop a baseline model SPHINX Agent and
compare its performance across state-of-the-art agents trained on other
datasets. To facilitate further research, we open-source our dataset, models,
and relevant evaluation tools. The project is available at
https://yuxiangchai.github.io/AMEX/
66a30b583c7f0decff91b8a6
Yuxiang Chai
admin_assigned
2024-07-26T07:40:16.403Z
6458ce236fa580137af5aa95
/avatars/db65a7332e375eb5daad5c1b076b1e3b.svg
Yuxiang Chai
user
Yuxiang007
66a30b583c7f0decff91b8a7
Siyuan Huang
claimed_verified
2024-09-17T09:31:20.873Z
634e4120038b5879133552f5
/avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
Siyuan
user
SiyuanH
66a30b583c7f0decff91b8a8
Yazhe Niu
66a30b583c7f0decff91b8a9
Han Xiao
claimed_verified
2025-05-28T09:02:41.235Z
666aa99cd1652853e4f9a8b9
/avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
HanXiao
user
HanXiao1999
66a30b583c7f0decff91b8aa
Liang Liu
66a30b583c7f0decff91b8ab
Dingyu Zhang
66a30b583c7f0decff91b8ac
Peng Gao
66a30b583c7f0decff91b8ad
Shuai Ren
66a30b583c7f0decff91b8ae
Hongsheng Li
admin_assigned
2024-07-26T07:48:40.057Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
AI agents have drawn increasing attention mostly on their ability to perceive
environments, understand tasks, and autonomously achieve goals. To advance
research on AI agents in mobile scenarios, we introduce the Android
Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for
generalist mobile GUI-control agents. Their capabilities of completing complex
tasks by directly interacting with the graphical user interface (GUI) on mobile
devices are trained and evaluated with the proposed dataset. AMEX comprises
over 104K high-resolution screenshots from 110 popular mobile applications,
which are annotated at multiple levels. Unlike existing mobile device-control
datasets, e.g., MoTIF, AitW, etc., AMEX includes three levels of annotations:
GUI interactive element grounding, GUI screen and element functionality
descriptions, and complex natural language instructions, each averaging 13
steps with stepwise GUI-action chains. We develop this dataset from a more
instructive and detailed perspective, complementing the general settings of
existing datasets. Additionally, we develop a baseline model SPHINX Agent and
compare its performance across state-of-the-art agents trained on other
datasets. To facilitate further research, we open-source our dataset, models,
and relevant evaluation tools. The project is available at
https://yuxiangchai.github.io/AMEX/
The Android Multi-annotation EXpo (AMEX) dataset supports training and evaluating generalist mobile GUI-control agents with detailed multi-level annotations for complex task completion.
Android Multi-annotation EXpo AMEX GUI-control agents graphical user interface mobile applications GUI interactive element grounding GUI screen and element functionality descriptions complex natural language instructions GUI-action chains SPHINX Agent