VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
Recently DeepSeek R1 has shown that reinforcement learning (RL) can
substantially improve the reasoning capabilities of Large Language Models
(LLMs) through a simple yet effective design. The core of R1 lies in its
rule-based reward formulation, which leverages tasks with deterministic
ground-truth answers to enable precise and stable reward computation. In the
visual domain, we similarly observe that a wide range of visual understanding
tasks are inherently equipped with well-defined ground-truth annotations. This
property makes them naturally compatible with rule-based reward mechanisms.
Motivated by this observation, we investigate the extension of R1-style
reinforcement learning to Vision-Language Models (VLMs), aiming to enhance
their visual reasoning capabilities. To this end, we develop VLM-R1, a
dedicated framework designed to harness RL for improving VLMs' performance on
general vision-language tasks. Using this framework, we further explore the
feasibility of applying RL to visual domain. Experimental results indicate that
the RL-based model not only delivers competitive performance on visual
understanding tasks but also surpasses Supervised Fine-Tuning (SFT) in
generalization ability. Furthermore, we conduct comprehensive ablation studies
that uncover a series of noteworthy insights, including the presence of reward
hacking in object detection, the emergence of the "OD aha moment", the impact
of training data quality, and the scaling behavior of RL across different model
sizes. Through these analyses, we aim to deepen the understanding of how
reinforcement learning enhances the capabilities of vision-language models, and
we hope our findings and open-source contributions will support continued
progress in the vision-language RL community. Our code and model are available
at https://github.com/om-ai-lab/VLM-R1
67fc52b190075b82590d75db
Haozhan Shen
67fc52b190075b82590d75dc
Peng Liu
67fc52b190075b82590d75dd
Jingcheng Li
67fc52b190075b82590d75de
Chunxin Fang
67fc52b190075b82590d75df
Yibo Ma
67fc52b190075b82590d75e0
Jiajia Liao
67fc52b190075b82590d75e1
Qiaoli Shen
67fc52b190075b82590d75e2
Zilun Zhang
67fc52b190075b82590d75e3
Kangjia Zhao
67fc52b190075b82590d75e4
Qianqian Zhang
67fc52b190075b82590d75e5
Ruochen Xu
67fc52b190075b82590d75e6
Tiancheng Zhao
VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
Recently DeepSeek R1 has shown that reinforcement learning (RL) can
substantially improve the reasoning capabilities of Large Language Models
(LLMs) through a simple yet effective design. The core of R1 lies in its
rule-based reward formulation, which leverages tasks with deterministic
ground-truth answers to enable precise and stable reward computation. In the
visual domain, we similarly observe that a wide range of visual understanding
tasks are inherently equipped with well-defined ground-truth annotations. This
property makes them naturally compatible with rule-based reward mechanisms.
Motivated by this observation, we investigate the extension of R1-style
reinforcement learning to Vision-Language Models (VLMs), aiming to enhance
their visual reasoning capabilities. To this end, we develop VLM-R1, a
dedicated framework designed to harness RL for improving VLMs' performance on
general vision-language tasks. Using this framework, we further explore the
feasibility of applying RL to visual domain. Experimental results indicate that
the RL-based model not only delivers competitive performance on visual
understanding tasks but also surpasses Supervised Fine-Tuning (SFT) in
generalization ability. Furthermore, we conduct comprehensive ablation studies
that uncover a series of noteworthy insights, including the presence of reward
hacking in object detection, the emergence of the "OD aha moment", the impact
of training data quality, and the scaling behavior of RL across different model
sizes. Through these analyses, we aim to deepen the understanding of how
reinforcement learning enhances the capabilities of vision-language models, and
we hope our findings and open-source contributions will support continued
progress in the vision-language RL community. Our code and model are available
at https://github.com/om-ai-lab/VLM-R1
A reinforcement learning framework extends the capabilities of Vision-Language Models (VLMs) in visual reasoning by leveraging rule-based reward formulations, achieving competitive performance and superior generalization compared to supervised fine-tuning.
reinforcement learning Large Language Models (LLMs) rule-based reward formulation deterministic ground-truth answers Vision-Language Models (VLMs) VLM-R1 visual understanding tasks reward hacking OD aha moment training data quality scaling behavior