GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on
  Mobile Devices
Smartphone users often navigate across multiple applications (apps) to
complete tasks such as sharing content between social media platforms.
Autonomous Graphical User Interface (GUI) navigation agents can enhance user
experience in communication, entertainment, and productivity by streamlining
workflows and reducing manual intervention. However, prior GUI agents often
trained with datasets comprising simple tasks that can be completed within a
single app, leading to poor performance in cross-app navigation. To address
this problem, we introduce GUI Odyssey, a comprehensive dataset for training
and evaluating cross-app navigation agents. GUI Odyssey consists of 7,735
episodes from 6 mobile devices, spanning 6 types of cross-app tasks, 201 apps,
and 1.4K app combos. Leveraging GUI Odyssey, we developed OdysseyAgent, a
multimodal cross-app navigation agent by fine-tuning the Qwen-VL model with a
history resampling module. Extensive experiments demonstrate OdysseyAgent's
superior accuracy compared to existing models. For instance, OdysseyAgent
surpasses fine-tuned Qwen-VL and zero-shot GPT-4V by 1.44\% and 55.49\%
in-domain accuracy, and 2.29\% and 48.14\% out-of-domain accuracy on average.
The dataset and code will be released in
https://github.com/OpenGVLab/GUI-Odyssey.
666fd4c3cc1f54dfa9132919
Quanfeng Lu
admin_assigned
2024-06-17T14:14:28.922Z
653a483dacdeea08424ef55d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tYG2isIZBLCaBbWABatgK.png
Quanfeng Lu
user
hflqf88888
666fd4c3cc1f54dfa913291a
Wenqi Shao
admin_assigned
2024-06-17T14:14:35.766Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
666fd4c3cc1f54dfa913291b
Zitao Liu
admin_assigned
2024-06-17T14:14:41.384Z
666fefc983571a7a055a9a28
/avatars/a19a0cf6c3315f16f6adc8cbae8238be.svg
Zitao Liu
user
windlao
666fd4c3cc1f54dfa913291c
Fanqing Meng
admin_assigned
2024-06-17T14:14:47.622Z
640b37b2bab5ca8fbe7df8f2
/avatars/c7bef45efad6a0d911a720e2236fcba5.svg
fanqing meng
user
FanqingM
666fd4c3cc1f54dfa913291d
Boxuan Li
admin_assigned
2024-06-17T14:14:53.006Z
664aefb990135abe9ba9b8c9
/avatars/79658c23555deadcfb097b6ebccbe4a6.svg
Boxuan Li
user
liboxuanhk
666fd4c3cc1f54dfa913291e
Botong Chen
666fd4c3cc1f54dfa913291f
Siyuan Huang
claimed_verified
2024-09-16T07:03:32.762Z
634e4120038b5879133552f5
/avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
Siyuan
user
SiyuanH
666fd4c3cc1f54dfa9132920
Kaipeng Zhang
claimed_verified
2024-06-17T07:31:41.126Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
666fd4c3cc1f54dfa9132921
Yu Qiao
666fd4c3cc1f54dfa9132922
Ping Luo
GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on
  Mobile Devices
Smartphone users often navigate across multiple applications (apps) to
complete tasks such as sharing content between social media platforms.
Autonomous Graphical User Interface (GUI) navigation agents can enhance user
experience in communication, entertainment, and productivity by streamlining
workflows and reducing manual intervention. However, prior GUI agents often
trained with datasets comprising simple tasks that can be completed within a
single app, leading to poor performance in cross-app navigation. To address
this problem, we introduce GUI Odyssey, a comprehensive dataset for training
and evaluating cross-app navigation agents. GUI Odyssey consists of 7,735
episodes from 6 mobile devices, spanning 6 types of cross-app tasks, 201 apps,
and 1.4K app combos. Leveraging GUI Odyssey, we developed OdysseyAgent, a
multimodal cross-app navigation agent by fine-tuning the Qwen-VL model with a
history resampling module. Extensive experiments demonstrate OdysseyAgent's
superior accuracy compared to existing models. For instance, OdysseyAgent
surpasses fine-tuned Qwen-VL and zero-shot GPT-4V by 1.44\% and 55.49\%
in-domain accuracy, and 2.29\% and 48.14\% out-of-domain accuracy on average.
The dataset and code will be released in
https://github.com/OpenGVLab/GUI-Odyssey.
GUI Odyssey is a dataset for training and evaluating cross-app navigation agents, and OdysseyAgent, a multimodal agent based on Qwen-VL, demonstrates superior accuracy compared to existing models.
graphical user interface GUI agents cross-app navigation GUI Odyssey multimodal navigation Qwen-VL history resampling module in-domain accuracy out-of-domain accuracy