AndroidLab: Training and Systematic Benchmarking of Android Autonomous
  Agents
Autonomous agents have become increasingly important for interacting with the
real world. Android agents, in particular, have been recently a
frequently-mentioned interaction method. However, existing studies for training
and evaluating Android agents lack systematic research on both open-source and
closed-source models. In this work, we propose AndroidLab as a systematic
Android agent framework. It includes an operation environment with different
modalities, action space, and a reproducible benchmark. It supports both large
language models (LLMs) and multimodal models (LMMs) in the same action space.
AndroidLab benchmark includes predefined Android virtual devices and 138 tasks
across nine apps built on these devices. By using the AndroidLab environment,
we develop an Android Instruction dataset and train six open-source LLMs and
LMMs, lifting the average success rates from 4.59% to 21.50% for LLMs and from
1.93% to 13.28% for LMMs. AndroidLab is open-sourced and publicly available at
https://github.com/THUDM/Android-Lab.
67299eee6cc63cc1ba362e9b
Yifan Xu
67299eee6cc63cc1ba362e9c
Xiao Liu
claimed_verified
2024-11-05T16:09:19.323Z
62f098602ca4d32a7cd87aba
https://cdn-avatars.huggingface.co/v1/production/uploads/1659934844914-62f098602ca4d32a7cd87aba.jpeg
Xiao Liu
user
ShawLiu
67299eee6cc63cc1ba362e9d
Xueqiao Sun
admin_assigned
2024-11-05T13:38:12.414Z
6729afc3c519b000f45d053b
/avatars/1d47cfd495e72316420fc16f7286df2b.svg
Xueqiao Sun
user
snow1007
67299eee6cc63cc1ba362e9e
Siyi Cheng
admin_assigned
2024-11-05T13:38:51.528Z
654cf05258c9d74272181118
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Dfx-hp3RLkP95gQtOM7T0.png
Siyi Cheng
user
LeakyKayO
67299eee6cc63cc1ba362e9f
Hao Yu
67299eee6cc63cc1ba362ea0
Hanyu Lai
admin_assigned
2024-11-05T13:38:57.325Z
62ea2d48b493272c269e8f34
/avatars/89b215dafa503b51ab212a9b63c82aca.svg
Hanyu Lai
user
hanyullai
67299eee6cc63cc1ba362ea1
Shudan Zhang
admin_assigned
2024-11-05T13:39:04.042Z
637883f3736e2989331da910
/avatars/f72b579724d2d43889907e8b732e79e6.svg
Shudan Zhang
user
zdaniel0222
67299eee6cc63cc1ba362ea2
Dan Zhang
67299eee6cc63cc1ba362ea3
Jie Tang
admin_assigned
2024-11-05T13:37:40.424Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
67299eee6cc63cc1ba362ea4
Yuxiao Dong
admin_assigned
2024-11-05T13:36:14.681Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
AndroidLab: Training and Systematic Benchmarking of Android Autonomous
  Agents
Autonomous agents have become increasingly important for interacting with the
real world. Android agents, in particular, have been recently a
frequently-mentioned interaction method. However, existing studies for training
and evaluating Android agents lack systematic research on both open-source and
closed-source models. In this work, we propose AndroidLab as a systematic
Android agent framework. It includes an operation environment with different
modalities, action space, and a reproducible benchmark. It supports both large
language models (LLMs) and multimodal models (LMMs) in the same action space.
AndroidLab benchmark includes predefined Android virtual devices and 138 tasks
across nine apps built on these devices. By using the AndroidLab environment,
we develop an Android Instruction dataset and train six open-source LLMs and
LMMs, lifting the average success rates from 4.59% to 21.50% for LLMs and from
1.93% to 13.28% for LMMs. AndroidLab is open-sourced and publicly available at
https://github.com/THUDM/Android-Lab.
AndroidLab provides a systematic framework for training and evaluating Android agents, supporting both large language models and multimodal models, and improves their task success rates.
Android agent framework operation environment action space reproducible benchmark large language models multimodal models Android Instruction dataset Android virtual devices