UItron: Foundational GUI Agent with Advanced Perception and Planning
GUI agent aims to enable automated operations on Mobile/PC devices, which is
an important task toward achieving artificial general intelligence. The rapid
advancement of VLMs accelerates the development of GUI agents, owing to their
powerful capabilities in visual understanding and task planning. However,
building a GUI agent remains a challenging task due to the scarcity of
operation trajectories, the availability of interactive infrastructure, and the
limitation of initial capabilities in foundation models. In this work, we
introduce UItron, an open-source foundational model for automatic GUI agents,
featuring advanced GUI perception, grounding, and planning capabilities. UItron
highlights the necessity of systemic data engineering and interactive
infrastructure as foundational components for advancing GUI agent development.
It not only systematically studies a series of data engineering strategies to
enhance training effects, but also establishes an interactive environment
connecting both Mobile and PC devices. In training, UItron adopts supervised
finetuning over perception and planning tasks in various GUI scenarios, and
then develop a curriculum reinforcement learning framework to enable complex
reasoning and exploration for online environments. As a result, UItron achieves
superior performance in benchmarks of GUI perception, grounding, and planning.
In particular, UItron highlights the interaction proficiency with top-tier
Chinese mobile APPs, as we identified a general lack of Chinese capabilities
even in state-of-the-art solutions. To this end, we manually collect over one
million steps of operation trajectories across the top 100 most popular apps,
and build the offline and online agent evaluation environments. Experimental
results demonstrate that UItron achieves significant progress in Chinese app
scenarios, propelling GUI agents one step closer to real-world application.
68b5103a851c6e7b001eca4e
Zhixiong Zeng
68b5103a851c6e7b001eca4f
Jing Huang
68b5103a851c6e7b001eca50
Liming Zheng
68b5103a851c6e7b001eca51
Wenkang Han
68b5103a851c6e7b001eca52
Yufeng Zhong
68b5103a851c6e7b001eca53
Lei Chen
68b5103a851c6e7b001eca54
Longrong Yang
68b5103a851c6e7b001eca55
Yingjie Chu
68b5103a851c6e7b001eca56
Yuzhi He
68b5103a851c6e7b001eca57
Lin Ma
UItron: Foundational GUI Agent with Advanced Perception and Planning
GUI agent aims to enable automated operations on Mobile/PC devices, which is
an important task toward achieving artificial general intelligence. The rapid
advancement of VLMs accelerates the development of GUI agents, owing to their
powerful capabilities in visual understanding and task planning. However,
building a GUI agent remains a challenging task due to the scarcity of
operation trajectories, the availability of interactive infrastructure, and the
limitation of initial capabilities in foundation models. In this work, we
introduce UItron, an open-source foundational model for automatic GUI agents,
featuring advanced GUI perception, grounding, and planning capabilities. UItron
highlights the necessity of systemic data engineering and interactive
infrastructure as foundational components for advancing GUI agent development.
It not only systematically studies a series of data engineering strategies to
enhance training effects, but also establishes an interactive environment
connecting both Mobile and PC devices. In training, UItron adopts supervised
finetuning over perception and planning tasks in various GUI scenarios, and
then develop a curriculum reinforcement learning framework to enable complex
reasoning and exploration for online environments. As a result, UItron achieves
superior performance in benchmarks of GUI perception, grounding, and planning.
In particular, UItron highlights the interaction proficiency with top-tier
Chinese mobile APPs, as we identified a general lack of Chinese capabilities
even in state-of-the-art solutions. To this end, we manually collect over one
million steps of operation trajectories across the top 100 most popular apps,
and build the offline and online agent evaluation environments. Experimental
results demonstrate that UItron achieves significant progress in Chinese app
scenarios, propelling GUI agents one step closer to real-world application.
UItron, an open-source foundational model for GUI agents, enhances visual understanding and task planning through advanced perception, grounding, and planning capabilities, achieving superior performance in Chinese app scenarios.
GUI agents VLMs visual understanding task planning UItron GUI perception grounding planning supervised finetuning curriculum reinforcement learning offline and online agent evaluation environments