GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based
  VLM Agent Training
Reinforcement learning with verifiable outcome rewards (RLVR) has effectively
scaled up chain-of-thought (CoT) reasoning in large language models (LLMs).
Yet, its efficacy in training vision-language model (VLM) agents for
goal-directed action reasoning in visual environments is less established. This
work investigates this problem through extensive experiments on complex card
games, such as 24 points, and embodied tasks from ALFWorld. We find that when
rewards are based solely on action outcomes, RL fails to incentivize CoT
reasoning in VLMs, instead leading to a phenomenon we termed thought collapse,
characterized by a rapid loss of diversity in the agent's thoughts,
state-irrelevant and incomplete reasoning, and subsequent invalid actions,
resulting in negative rewards. To counteract thought collapse, we highlight the
necessity of process guidance and propose an automated corrector that evaluates
and refines the agent's reasoning at each RL step. This simple and scalable GTR
(Guided Thought Reinforcement) framework trains reasoning and action
simultaneously without the need for dense, per-step human labeling. Our
experiments demonstrate that GTR significantly enhances the performance and
generalization of the LLaVA-7b model across various visual environments,
achieving 3-5 times higher task success rates compared to SoTA models with
notably smaller model sizes.
67d280f20a6a6dd4a0ffe9e8
Tong Wei
claimed_verified
2025-03-14T08:58:47.531Z
66c2e8b4a03b764ca9057e65
/avatars/ce17d68f02a08c148ebb9df3170812c8.svg
Tong Wei
user
weit123
67d280f20a6a6dd4a0ffe9e9
Yijun Yang
admin_assigned
2025-03-14T11:23:38.382Z
645e553c3b6d85c65e8b0e54
/avatars/1fffc6499b9d65b21a895ca96f03b781.svg
Steven
user
yijunyang
67d280f20a6a6dd4a0ffe9ea
Junliang Xing
67d280f20a6a6dd4a0ffe9eb
Yuanchun Shi
67d280f20a6a6dd4a0ffe9ec
Zongqing Lu
67d280f20a6a6dd4a0ffe9ed
Deheng Ye
GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based
  VLM Agent Training
Reinforcement learning with verifiable outcome rewards (RLVR) has effectively
scaled up chain-of-thought (CoT) reasoning in large language models (LLMs).
Yet, its efficacy in training vision-language model (VLM) agents for
goal-directed action reasoning in visual environments is less established. This
work investigates this problem through extensive experiments on complex card
games, such as 24 points, and embodied tasks from ALFWorld. We find that when
rewards are based solely on action outcomes, RL fails to incentivize CoT
reasoning in VLMs, instead leading to a phenomenon we termed thought collapse,
characterized by a rapid loss of diversity in the agent's thoughts,
state-irrelevant and incomplete reasoning, and subsequent invalid actions,
resulting in negative rewards. To counteract thought collapse, we highlight the
necessity of process guidance and propose an automated corrector that evaluates
and refines the agent's reasoning at each RL step. This simple and scalable GTR
(Guided Thought Reinforcement) framework trains reasoning and action
simultaneously without the need for dense, per-step human labeling. Our
experiments demonstrate that GTR significantly enhances the performance and
generalization of the LLaVA-7b model across various visual environments,
achieving 3-5 times higher task success rates compared to SoTA models with
notably smaller model sizes.
GTR framework enhances VLM agent performance in visual environments by addressing thought collapse through process guidance and automated reasoning refinement.
Reinforcement learning verifiable outcome rewards RLVR chain-of-thought reasoning large language models vision-language models goal-directed action reasoning thought collapse state-irrelevant reasoning invalid actions negative rewards process guidance Guided Thought Reinforcement GTR LLaVA-7b task success rates