ColorAgent: Building A Robust, Personalized, and Interactive OS Agent
With the advancements in hardware, software, and large language model
technologies, the interaction between humans and operating systems has evolved
from the command-line interface to the rapidly emerging AI agent interactions.
Building an operating system (OS) agent capable of executing user instructions
and faithfully following user desires is becoming a reality. In this technical
report, we present ColorAgent, an OS agent designed to engage in long-horizon,
robust interactions with the environment while also enabling personalized and
proactive user interaction. To enable long-horizon interactions with the
environment, we enhance the model's capabilities through step-wise
reinforcement learning and self-evolving training, while also developing a
tailored multi-agent framework that ensures generality, consistency, and
robustness. In terms of user interaction, we explore personalized user intent
recognition and proactive engagement, positioning the OS agent not merely as an
automation tool but as a warm, collaborative partner. We evaluate ColorAgent on
the AndroidWorld and AndroidLab benchmarks, achieving success rates of 77.2%
and 50.7%, respectively, establishing a new state of the art. Nonetheless, we
note that current benchmarks are insufficient for a comprehensive evaluation of
OS agents and propose further exploring directions in future work, particularly
in the areas of evaluation paradigms, agent collaboration, and security. Our
code is available at https://github.com/MadeAgents/mobile-use.
68f98cceb9b2e4ae046737c3
Ning Li
68f98cceb9b2e4ae046737c4
Qiqiang Lin
68f98cceb9b2e4ae046737c5
Zheng Wu
68f98cceb9b2e4ae046737c6
Xiaoyun Mo
68f98cceb9b2e4ae046737c7
Weiming Zhang
68f98cceb9b2e4ae046737c8
Yin Zhao
68f98cceb9b2e4ae046737c9
Xiangmou Qu
68f98cceb9b2e4ae046737ca
Jiamu Zhou
68f98cceb9b2e4ae046737cb
Jun Wang
68f98cceb9b2e4ae046737cc
Congmin Zheng
68f98cceb9b2e4ae046737cd
Yuanyi Song
68f98cceb9b2e4ae046737ce
Hongjiang Chen
68f98cceb9b2e4ae046737cf
Heyuan Huang
68f98cceb9b2e4ae046737d0
Jihong Wang
68f98cceb9b2e4ae046737d1
Jiaxin Yin
68f98cceb9b2e4ae046737d2
Jingwei Yu
68f98cceb9b2e4ae046737d3
Junwei Liao
68f98cceb9b2e4ae046737d4
Qiuying Peng
68f98cceb9b2e4ae046737d5
Xingyu Lou
68f98cceb9b2e4ae046737d6
Jun Wang
68f98cceb9b2e4ae046737d7
Weiwen Liu
68f98cceb9b2e4ae046737d8
Zhuosheng Zhang
68f98cceb9b2e4ae046737d9
Weinan Zhang
ColorAgent: Building A Robust, Personalized, and Interactive OS Agent
With the advancements in hardware, software, and large language model
technologies, the interaction between humans and operating systems has evolved
from the command-line interface to the rapidly emerging AI agent interactions.
Building an operating system (OS) agent capable of executing user instructions
and faithfully following user desires is becoming a reality. In this technical
report, we present ColorAgent, an OS agent designed to engage in long-horizon,
robust interactions with the environment while also enabling personalized and
proactive user interaction. To enable long-horizon interactions with the
environment, we enhance the model's capabilities through step-wise
reinforcement learning and self-evolving training, while also developing a
tailored multi-agent framework that ensures generality, consistency, and
robustness. In terms of user interaction, we explore personalized user intent
recognition and proactive engagement, positioning the OS agent not merely as an
automation tool but as a warm, collaborative partner. We evaluate ColorAgent on
the AndroidWorld and AndroidLab benchmarks, achieving success rates of 77.2%
and 50.7%, respectively, establishing a new state of the art. Nonetheless, we
note that current benchmarks are insufficient for a comprehensive evaluation of
OS agents and propose further exploring directions in future work, particularly
in the areas of evaluation paradigms, agent collaboration, and security. Our
code is available at https://github.com/MadeAgents/mobile-use.
ColorAgent, an OS agent using step-wise reinforcement learning and a multi-agent framework, achieves high success rates in long-horizon interactions and personalized user engagement on Android benchmarks.
step-wise reinforcement learning self-evolving training multi-agent framework personalized user intent recognition proactive engagement AndroidWorld AndroidLab