RL makes MLLMs see better than SFT
A dominant assumption in Multimodal Language Model (MLLM) research is that
its performance is largely inherited from the LLM backbone, given its immense
parameter scale and remarkable capabilities. This has created a void in the
understanding of the vision encoder, which determines how MLLMs perceive
images. The recent shift in MLLM training paradigms, from Supervised Finetuning
(SFT) to Reinforcement Learning (RL), magnifies this oversight-namely, the
significant lack of analysis on how such training reshapes the vision encoder
as well as the MLLM. To address this, we first investigate the impact of
training strategies on MLLMs, where RL shows a clear advantage over SFT in
strongly vision-related VQA benchmarks. Motivated by this, we conduct a
critical yet under-explored analysis of the vision encoder of MLLMs through
diverse and in-depth experiments, ranging from ImageNet classification and
segmentation to gradient visualization. Our results demonstrate that MLLM's
post-training strategy (i.e., SFT or RL) not only leads to distinct outcomes on
MLLM downstream tasks, but also fundamentally reshapes MLLM's underlying visual
representations. Specifically, the key finding of our study is that RL produces
stronger and precisely localized visual representations compared to SFT,
boosting the ability of the vision encoder for MLLM. We then reframe our
findings into a simple recipe for building strong vision encoders for MLLMs,
Preference-Instructed Vision OpTimization (PIVOT). When integrated into MLLMs,
a PIVOT-trained vision encoder outperforms even larger and more heavily-trained
counterparts, despite requiring less than 1% of the computational cost of
standard vision pretraining. This result opens an effective and efficient path
for advancing the vision backbones of MLLMs. Project page available at
https://june-page.github.io/pivot/
68f6fb4824c4489363111942
Junha Song
claimed_verified
2025-10-22T15:45:52.251Z
65ffd913b1e509e1e45c81f5
/avatars/3a997f3a7874126596575c5bcd99d756.svg
JunhaSong
user
junha1125
68f6fb4824c4489363111943
Sangdoo Yun
68f6fb4824c4489363111944
Dongyoon Han
68f6fb4824c4489363111945
Jaegul Choo
68f6fb4824c4489363111946
Byeongho Heo
claimed_verified
2025-10-21T14:03:22.750Z
64b9feed96676e40d0fa89a7
/avatars/2154a6ceb87677ad2c9d9620de5b18ec.svg
Byeongho Heo
user
bhheo
RL makes MLLMs see better than SFT
A dominant assumption in Multimodal Language Model (MLLM) research is that
its performance is largely inherited from the LLM backbone, given its immense
parameter scale and remarkable capabilities. This has created a void in the
understanding of the vision encoder, which determines how MLLMs perceive
images. The recent shift in MLLM training paradigms, from Supervised Finetuning
(SFT) to Reinforcement Learning (RL), magnifies this oversight-namely, the
significant lack of analysis on how such training reshapes the vision encoder
as well as the MLLM. To address this, we first investigate the impact of
training strategies on MLLMs, where RL shows a clear advantage over SFT in
strongly vision-related VQA benchmarks. Motivated by this, we conduct a
critical yet under-explored analysis of the vision encoder of MLLMs through
diverse and in-depth experiments, ranging from ImageNet classification and
segmentation to gradient visualization. Our results demonstrate that MLLM's
post-training strategy (i.e., SFT or RL) not only leads to distinct outcomes on
MLLM downstream tasks, but also fundamentally reshapes MLLM's underlying visual
representations. Specifically, the key finding of our study is that RL produces
stronger and precisely localized visual representations compared to SFT,
boosting the ability of the vision encoder for MLLM. We then reframe our
findings into a simple recipe for building strong vision encoders for MLLMs,
Preference-Instructed Vision OpTimization (PIVOT). When integrated into MLLMs,
a PIVOT-trained vision encoder outperforms even larger and more heavily-trained
counterparts, despite requiring less than 1% of the computational cost of
standard vision pretraining. This result opens an effective and efficient path
for advancing the vision backbones of MLLMs. Project page available at
https://june-page.github.io/pivot/
Reinforcement Learning enhances vision encoders in Multimodal Language Models, leading to better visual representations and performance compared to Supervised Fine-tuning.
Multimodal Language Model LLM backbone vision encoder Supervised Finetuning Reinforcement Learning VQA benchmarks ImageNet classification segmentation gradient visualization visual representations Preference-Instructed Vision OpTimization PIVOT