VOGUE: Guiding Exploration with Visual Uncertainty Improves Multimodal
  Reasoning
Reinforcement learning with verifiable rewards (RLVR) improves reasoning in
large language models (LLMs) but struggles with exploration, an issue that
still persists for multimodal LLMs (MLLMs). Current methods treat the visual
input as a fixed, deterministic condition, overlooking a critical source of
ambiguity and struggling to build policies robust to plausible visual
variations. We introduce VOGUE (Visual Uncertainty Guided
Exploration), a novel method that shifts exploration from the output (text)
to the input (visual) space. By treating the image as a stochastic context,
VOGUE quantifies the policy's sensitivity to visual perturbations using the
symmetric KL divergence between a "raw" and "noisy" branch, creating a direct
signal for uncertainty-aware exploration. This signal shapes the learning
objective via an uncertainty-proportional bonus, which, combined with a
token-entropy bonus and an annealed sampling schedule, effectively balances
exploration and exploitation. Implemented within GRPO on two model scales
(Qwen2.5-VL-3B/7B), VOGUE boosts pass@1 accuracy by an average of 2.6% on three
visual math benchmarks and 3.7% on three general-domain reasoning benchmarks,
while simultaneously increasing pass@4 performance and mitigating the
exploration decay commonly observed in RL fine-tuning. Our work shows that
grounding exploration in the inherent uncertainty of visual inputs is an
effective strategy for improving multimodal reasoning.
68df2e3edf49fb0df1e03c19
Rui Liu
claimed_verified
2025-10-05T12:47:20.250Z
6656bf615b203a05a1f0968c
/avatars/1ee0b0099c10dd76c8e3b7d312221b15.svg
Rui Liu
user
lr10260
68df2e3edf49fb0df1e03c1a
Dian Yu
68df2e3edf49fb0df1e03c1b
Tong Zheng
claimed_verified
2025-10-03T12:30:01.705Z
6623ea65b642e29cdf90a1b4
/avatars/e32e90574c1162b2be87ed78604e3e4d.svg
TongZheng
user
TongZheng1999
68df2e3edf49fb0df1e03c1c
Runpeng Dai
claimed_verified
2025-10-03T12:30:04.017Z
65037565da2d88e201f63b7a
/avatars/d1b6ce17236360e9583b8bb4cb87e506.svg
Runpeng Dai
user
Leo-Dai
68df2e3edf49fb0df1e03c1d
Zongxia Li
68df2e3edf49fb0df1e03c1e
Wenhao Yu
68df2e3edf49fb0df1e03c1f
Zhenwen Liang
68df2e3edf49fb0df1e03c20
Linfeng Song
68df2e3edf49fb0df1e03c21
Haitao Mi
68df2e3edf49fb0df1e03c22
Pratap Tokekar
68df2e3edf49fb0df1e03c23
Dong Yu
VOGUE: Guiding Exploration with Visual Uncertainty Improves Multimodal
  Reasoning
Reinforcement learning with verifiable rewards (RLVR) improves reasoning in
large language models (LLMs) but struggles with exploration, an issue that
still persists for multimodal LLMs (MLLMs). Current methods treat the visual
input as a fixed, deterministic condition, overlooking a critical source of
ambiguity and struggling to build policies robust to plausible visual
variations. We introduce VOGUE (Visual Uncertainty Guided
Exploration), a novel method that shifts exploration from the output (text)
to the input (visual) space. By treating the image as a stochastic context,
VOGUE quantifies the policy's sensitivity to visual perturbations using the
symmetric KL divergence between a "raw" and "noisy" branch, creating a direct
signal for uncertainty-aware exploration. This signal shapes the learning
objective via an uncertainty-proportional bonus, which, combined with a
token-entropy bonus and an annealed sampling schedule, effectively balances
exploration and exploitation. Implemented within GRPO on two model scales
(Qwen2.5-VL-3B/7B), VOGUE boosts pass@1 accuracy by an average of 2.6% on three
visual math benchmarks and 3.7% on three general-domain reasoning benchmarks,
while simultaneously increasing pass@4 performance and mitigating the
exploration decay commonly observed in RL fine-tuning. Our work shows that
grounding exploration in the inherent uncertainty of visual inputs is an
effective strategy for improving multimodal reasoning.
VOGUE, a method that shifts exploration to the visual input space by quantifying policy sensitivity to visual perturbations, enhances multimodal reasoning in large language models.
reinforcement learning with verifiable rewards multimodal LLMs visual uncertainty stochastic context symmetric KL divergence uncertainty-aware exploration token-entropy bonus annealed sampling schedule GRPO pass@1 accuracy pass@4 performance exploration decay