Mano Report
Graphical user interfaces (GUIs) are the primary medium for human-computer
interaction, yet automating GUI interactions remains challenging due to the
complexity of visual elements, dynamic environments, and the need for
multi-step reasoning. Existing methods based on vision-language models (VLMs)
often suffer from limited resolution, domain mismatch, and insufficient
sequential decisionmaking capability. To address these issues, we propose Mano,
a robust GUI agent built upon a multi-modal foundation model pre-trained on
extensive web and computer system data. Our approach integrates a novel
simulated environment for high-fidelity data generation, a three-stage training
pipeline (supervised fine-tuning, offline reinforcement learning, and online
reinforcement learning), and a verification module for error recovery. Mano
demonstrates state-of-the-art performance on multiple GUI benchmarks, including
Mind2Web and OSWorld, achieving significant improvements in success rate and
operational accuracy. Our work provides new insights into the effective
integration of reinforcement learning with VLMs for practical GUI agent
deployment, highlighting the importance of domain-specific data, iterative
training, and holistic reward design.
68d20f911ca7156988a8ed61
Tianyu Fu
68d20f911ca7156988a8ed62
Anyang Su
68d20f911ca7156988a8ed63
Chenxu Zhao
68d20f911ca7156988a8ed64
Hanning Wang
68d20f911ca7156988a8ed65
Minghui Wu
68d20f911ca7156988a8ed66
Zhe Yu
68d20f911ca7156988a8ed67
Fei Hu
68d20f911ca7156988a8ed68
Mingjia Shi
68d20f911ca7156988a8ed69
Wei Dong
68d20f911ca7156988a8ed6a
Jiayao Wang
68d20f911ca7156988a8ed6b
Yuyang Chen
68d20f911ca7156988a8ed6c
Ruiyang Yu
68d20f911ca7156988a8ed6d
Siran Peng
68d20f911ca7156988a8ed6e
Menglin Li
68d20f911ca7156988a8ed6f
Nan Huang
68d20f911ca7156988a8ed70
Haitian Wei
68d20f911ca7156988a8ed71
Jiawei Yu
68d20f911ca7156988a8ed72
Yi Xin
68d20f911ca7156988a8ed73
Xilin Zhao
68d20f911ca7156988a8ed74
Kai Gu
68d20f911ca7156988a8ed75
Ping Jiang
68d20f911ca7156988a8ed76
Sifan Zhou
68d20f911ca7156988a8ed77
Shuo Wang
Mano Report
Graphical user interfaces (GUIs) are the primary medium for human-computer
interaction, yet automating GUI interactions remains challenging due to the
complexity of visual elements, dynamic environments, and the need for
multi-step reasoning. Existing methods based on vision-language models (VLMs)
often suffer from limited resolution, domain mismatch, and insufficient
sequential decisionmaking capability. To address these issues, we propose Mano,
a robust GUI agent built upon a multi-modal foundation model pre-trained on
extensive web and computer system data. Our approach integrates a novel
simulated environment for high-fidelity data generation, a three-stage training
pipeline (supervised fine-tuning, offline reinforcement learning, and online
reinforcement learning), and a verification module for error recovery. Mano
demonstrates state-of-the-art performance on multiple GUI benchmarks, including
Mind2Web and OSWorld, achieving significant improvements in success rate and
operational accuracy. Our work provides new insights into the effective
integration of reinforcement learning with VLMs for practical GUI agent
deployment, highlighting the importance of domain-specific data, iterative
training, and holistic reward design.
A robust GUI agent, Mano, integrates reinforcement learning with vision-language models for high-fidelity data generation and improved performance on GUI benchmarks.
vision-language models multi-modal foundation model simulated environment three-stage training pipeline supervised fine-tuning offline reinforcement learning online reinforcement learning verification module Mind2Web OSWorld domain-specific data iterative training holistic reward design