SFT or RL? An Early Investigation into Training R1-Like Reasoning Large
  Vision-Language Models
This work revisits the dominant supervised fine-tuning (SFT) then
reinforcement learning (RL) paradigm for training Large Vision-Language Models
(LVLMs), and reveals a key finding: SFT can significantly undermine subsequent
RL by inducing ``pseudo reasoning paths'' imitated from expert models. While
these paths may resemble the native reasoning paths of RL models, they often
involve prolonged, hesitant, less informative steps, and incorrect reasoning.
To systematically study this effect, we introduce VLAA-Thinking, a new
multimodal dataset designed to support reasoning in LVLMs. Constructed via a
six-step pipeline involving captioning, reasoning distillation, answer rewrite
and verification, VLAA-Thinking comprises high-quality, step-by-step visual
reasoning traces for SFT, along with a more challenging RL split from the same
data source. Using this dataset, we conduct extensive experiments comparing
SFT, RL and their combinations. Results show that while SFT helps models learn
reasoning formats, it often locks aligned models into imitative, rigid
reasoning modes that impede further learning. In contrast, building on the
Group Relative Policy Optimization (GRPO) with a novel mixed reward module
integrating both perception and cognition signals, our RL approach fosters more
genuine, adaptive reasoning behavior. Notably, our model VLAA-Thinker, based on
Qwen2.5VL 3B, achieves top-1 performance on Open LMM Reasoning Leaderboard
(https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard)
among 4B scale LVLMs, surpassing the previous state-of-the-art by 1.8%. We hope
our findings provide valuable insights in developing reasoning-capable LVLMs
and can inform future research in this area.
68008583bee5b086995074fe
Hardy Chen
claimed_verified
2025-04-17T08:04:02.073Z
65112b6b8c4b535a971ad3cf
/avatars/1ef00b312d2b009c8c7aab21b4b3f258.svg
guiminghardychen
user
g-h-chen
68008583bee5b086995074ff
Haoqin Tu
claimed_verified
2025-04-30T07:57:03.666Z
604ae011caabafacfa48e3de
https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
Haoqin Tu
user
PahaII
68008583bee5b08699507500
Fali Wang
68008583bee5b08699507501
Hui Liu
68008583bee5b08699507502
Xianfeng Tang
68008583bee5b08699507503
Xinya Du
68008583bee5b08699507504
Yuyin Zhou
68008583bee5b08699507505
Cihang Xie
claimed_verified
2025-07-31T08:16:40.925Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
SFT or RL? An Early Investigation into Training R1-Like Reasoning Large
  Vision-Language Models
This work revisits the dominant supervised fine-tuning (SFT) then
reinforcement learning (RL) paradigm for training Large Vision-Language Models
(LVLMs), and reveals a key finding: SFT can significantly undermine subsequent
RL by inducing ``pseudo reasoning paths'' imitated from expert models. While
these paths may resemble the native reasoning paths of RL models, they often
involve prolonged, hesitant, less informative steps, and incorrect reasoning.
To systematically study this effect, we introduce VLAA-Thinking, a new
multimodal dataset designed to support reasoning in LVLMs. Constructed via a
six-step pipeline involving captioning, reasoning distillation, answer rewrite
and verification, VLAA-Thinking comprises high-quality, step-by-step visual
reasoning traces for SFT, along with a more challenging RL split from the same
data source. Using this dataset, we conduct extensive experiments comparing
SFT, RL and their combinations. Results show that while SFT helps models learn
reasoning formats, it often locks aligned models into imitative, rigid
reasoning modes that impede further learning. In contrast, building on the
Group Relative Policy Optimization (GRPO) with a novel mixed reward module
integrating both perception and cognition signals, our RL approach fosters more
genuine, adaptive reasoning behavior. Notably, our model VLAA-Thinker, based on
Qwen2.5VL 3B, achieves top-1 performance on Open LMM Reasoning Leaderboard
(https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard)
among 4B scale LVLMs, surpassing the previous state-of-the-art by 1.8%. We hope
our findings provide valuable insights in developing reasoning-capable LVLMs
and can inform future research in this area.
Supervised fine-tuning can hinder reinforcement learning in Large Vision-Language Models by inducing imitative reasoning, whereas a novel reinforcement learning approach improves genuine reasoning capabilities and outperforms existing models.
supervised fine-tuning reinforcement learning Large Vision-Language Models pseudo reasoning paths VLAA-Thinking reasoning distillation answer rewrite verification Group Relative Policy Optimization mixed reward module perception signals cognition signals VLAA-Thinker Qwen2.5VL 3B Open LMM Reasoning Leaderboard