Visionary-R1: Mitigating Shortcuts in Visual Reasoning with
  Reinforcement Learning
Learning general-purpose reasoning capabilities has long been a challenging
problem in AI. Recent research in large language models (LLMs), such as
DeepSeek-R1, has shown that reinforcement learning techniques like GRPO can
enable pre-trained LLMs to develop reasoning capabilities using simple
question-answer pairs. In this paper, we aim to train visual language models
(VLMs) to perform reasoning on image data through reinforcement learning and
visual question-answer pairs, without any explicit chain-of-thought (CoT)
supervision. Our findings indicate that simply applying reinforcement learning
to a VLM -- by prompting the model to produce a reasoning chain before
providing an answer -- can lead the model to develop shortcuts from easy
questions, thereby reducing its ability to generalize across unseen data
distributions. We argue that the key to mitigating shortcut learning is to
encourage the model to interpret images prior to reasoning. Therefore, we train
the model to adhere to a caption-reason-answer output format: initially
generating a detailed caption for an image, followed by constructing an
extensive reasoning chain. When trained on 273K CoT-free visual question-answer
pairs and using only reinforcement learning, our model, named Visionary-R1,
outperforms strong multimodal models, such as GPT-4o, Claude3.5-Sonnet, and
Gemini-1.5-Pro, on multiple visual reasoning benchmarks.
682d718306291bf11fcf69a3
Jiaer Xia
admin_assigned
2025-05-21T10:14:20.493Z
664da76e4eb4c91c8c32cc06
/avatars/050bb6e4136f8a1645fef277ad08c7fc.svg
Jiaer Xia
user
Jiaer-Xia
682d718306291bf11fcf69a4
Yuhang Zang
claimed_verified
2025-05-21T08:39:56.735Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
682d718306291bf11fcf69a5
Peng Gao
682d718306291bf11fcf69a6
Yixuan Li
claimed_verified
2025-06-03T08:51:33.410Z
654c865e17d83697c75f3f90
https://cdn-avatars.huggingface.co/v1/production/uploads/654c865e17d83697c75f3f90/JW79nXWaCKmw2HmELshEw.png
JiaerXia
user
JiaerX
682d718306291bf11fcf69a7
Kaiyang Zhou
admin_assigned
2025-05-21T10:14:28.433Z
62ac6656de8bfbb93094b8fd
/avatars/1d8f445b6d5f9d43ddab81056bcb141e.svg
Kaiyang Zhou
user
kaiyangzhou
Visionary-R1: Mitigating Shortcuts in Visual Reasoning with
  Reinforcement Learning
Learning general-purpose reasoning capabilities has long been a challenging
problem in AI. Recent research in large language models (LLMs), such as
DeepSeek-R1, has shown that reinforcement learning techniques like GRPO can
enable pre-trained LLMs to develop reasoning capabilities using simple
question-answer pairs. In this paper, we aim to train visual language models
(VLMs) to perform reasoning on image data through reinforcement learning and
visual question-answer pairs, without any explicit chain-of-thought (CoT)
supervision. Our findings indicate that simply applying reinforcement learning
to a VLM -- by prompting the model to produce a reasoning chain before
providing an answer -- can lead the model to develop shortcuts from easy
questions, thereby reducing its ability to generalize across unseen data
distributions. We argue that the key to mitigating shortcut learning is to
encourage the model to interpret images prior to reasoning. Therefore, we train
the model to adhere to a caption-reason-answer output format: initially
generating a detailed caption for an image, followed by constructing an
extensive reasoning chain. When trained on 273K CoT-free visual question-answer
pairs and using only reinforcement learning, our model, named Visionary-R1,
outperforms strong multimodal models, such as GPT-4o, Claude3.5-Sonnet, and
Gemini-1.5-Pro, on multiple visual reasoning benchmarks.
Reinforcement learning applied to visual language models with image captions and reasoning chains leads to improved performance on visual reasoning benchmarks compared to multimodal models.
reinforcement learning visual language models visual question-answer pairs chain-of-thought caption-reason-answer visual reasoning benchmarks