Reinforcing Visual State Reasoning for Multi-Turn VLM Agents
Kangrui Wang, Pingyue Zhang, Zihan Wang, Yaning Gao, Linjie Li, Qineng Wang, Hanyang Chen, Yiping Lu, Zhengyuan Yang, Lijuan Wang, Ranjay Krishna, Jiajun Wu, Fei-Fei Li, Yejin Choi, Manling Li
Reinforcement Learning (RL) has demonstrated effectiveness in enhancing Large Language Model (LLM) in performing multi-turn agentic tasks, while Vision-Language Models (VLMs) has not yet shown promise in completing multi-turn agentic tasks, where a significant challenge is reasoning about visual states. Our empirical investigation reveals that incorporating explicit visual state reasoning, such as Grounding (describing current visual state) and WorldModeling (predicting next state), into the VLM's thinking process during RL training significantly enhances task performance. We further explore optimal visual state representations, finding natural language effective generally, while structured formats prove crucial for tasks demanding high precision or understanding of low-level visual details. To specifically reinforce visual state reasoning, we introduce Visual Reasoning RL, which incorporates two key techniques: a turn-level visual reasoning reward to supervise reasoning accuracy, and Bi-Level General Advantage Estimation (GAE) that estimates advantages at both turn and token levels. This comprehensive approach consistently improves task performance and reasoning quality. These experiments are facilitated by VAGEN, a scalable training framework for multi-turn VLM agents across diverse visual environments. Our findings offer pathways to more robust visual state reasoning in VLM agents.