Spotlight on Token Perception for Multimodal Reinforcement Learning
While Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the
reasoning capabilities of Large Vision-Language Models (LVLMs), most existing
methods in multimodal reasoning neglect the critical role of visual perception
within the RLVR optimization process. In this paper, we undertake a pioneering
exploration of multimodal RLVR through the novel perspective of token
perception, which measures the visual dependency of each generated token. With
a granular analysis of Chain-of-Thought (CoT) processes, we uncover two key
insights: first, token perception in a rollout trajectory is sparsely
distributed, where only a small fraction of tokens have high visual dependency
for visually-grounded reasoning; second, different trajectories exhibit
significant divergence in their overall visual dependency. Based on these
observations, we propose Visually-Perceptive Policy Optimization (VPPO), a
novel policy gradient algorithm that explicitly leverages token perception to
refine the learning signal. Specifically, VPPO achieves this through a dual
mechanism: it reweights a trajectory's advantage by its overall visual
dependency, and focuses policy updates exclusively on perceptually pivotal
tokens. On a comprehensive suite of eight perception and reasoning benchmarks,
VPPO demonstrates substantial gains over leading open-source RL-tuned models,
with its effectiveness consistently validated across 7B and 32B model scales.
Our findings not only establish a new token-level perceptual perspective for
analyzing multimodal RLVR but also present a novel and effective optimization
strategy to significantly enhance the multimodal reasoning capabilities of
LVLMs.
68ec8ab6cd07fb414898ca7e
Siyuan Huang
admin_assigned
2025-10-14T19:21:44.979Z
66f79b323fe089b75e9e0c04
/avatars/92918bf8913012a3f005f09e03b381c2.svg
Siyuan Huang
user
chamber111
68ec8ab6cd07fb414898ca7f
Xiaoye Qu
68ec8ab6cd07fb414898ca80
Yafu Li
68ec8ab6cd07fb414898ca81
Yun Luo
68ec8ab6cd07fb414898ca82
Zefeng He
68ec8ab6cd07fb414898ca83
Daizong Liu
68ec8ab6cd07fb414898ca84
Yu Cheng
Spotlight on Token Perception for Multimodal Reinforcement Learning
While Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the
reasoning capabilities of Large Vision-Language Models (LVLMs), most existing
methods in multimodal reasoning neglect the critical role of visual perception
within the RLVR optimization process. In this paper, we undertake a pioneering
exploration of multimodal RLVR through the novel perspective of token
perception, which measures the visual dependency of each generated token. With
a granular analysis of Chain-of-Thought (CoT) processes, we uncover two key
insights: first, token perception in a rollout trajectory is sparsely
distributed, where only a small fraction of tokens have high visual dependency
for visually-grounded reasoning; second, different trajectories exhibit
significant divergence in their overall visual dependency. Based on these
observations, we propose Visually-Perceptive Policy Optimization (VPPO), a
novel policy gradient algorithm that explicitly leverages token perception to
refine the learning signal. Specifically, VPPO achieves this through a dual
mechanism: it reweights a trajectory's advantage by its overall visual
dependency, and focuses policy updates exclusively on perceptually pivotal
tokens. On a comprehensive suite of eight perception and reasoning benchmarks,
VPPO demonstrates substantial gains over leading open-source RL-tuned models,
with its effectiveness consistently validated across 7B and 32B model scales.
Our findings not only establish a new token-level perceptual perspective for
analyzing multimodal RLVR but also present a novel and effective optimization
strategy to significantly enhance the multimodal reasoning capabilities of
LVLMs.
VPPO, a novel policy gradient algorithm, enhances multimodal RLVR by leveraging token perception to refine learning signals and improve reasoning capabilities in Large Vision-Language Models.
Reinforcement Learning with Verifiable Rewards Large Vision-Language Models multimodal reasoning token perception Chain-of-Thought rollout trajectory visual dependency Visually-Perceptive Policy Optimization policy gradient algorithm perceptually pivotal tokens