Self-Rewarding Vision-Language Model via Reasoning Decomposition
Vision-Language Models (VLMs) often suffer from visual hallucinations, saying
things that are not actually in the image, and language shortcuts, where they
skip the visual part and just rely on text priors. These issues arise because
most post-training methods for VLMs rely on simple verifiable answer matching
and supervise only final outputs, leaving intermediate visual reasoning without
explicit guidance. As a result, VLMs receive sparse visual signals and often
learn to prioritize language-based reasoning over visual perception. To
mitigate this, some existing methods add visual supervision using human
annotations or distilled labels from external large models. However, human
annotations are labor-intensive and costly, and because external signals cannot
adapt to the evolving policy, they cause distributional shifts that can lead to
reward hacking. In this paper, we introduce Vision-SR1, a self-rewarding method
that improves visual reasoning without relying on external visual supervisions
via reinforcement learning. Vision-SR1 decomposes VLM reasoning into two
stages: visual perception and language reasoning. The model is first prompted
to produce self-contained visual perceptions that are sufficient to answer the
question without referring back the input image. To validate this
self-containment, the same VLM model is then re-prompted to perform language
reasoning using only the generated perception as input to compute reward. This
self-reward is combined with supervision on final outputs, providing a balanced
training signal that strengthens both visual perception and language reasoning.
Our experiments demonstrate that Vision-SR1 improves visual reasoning,
mitigates visual hallucinations, and reduces reliance on language shortcuts
across diverse vision-language tasks.
68afb491245176306494cbb0
Zongxia Li
admin_assigned
2025-08-28T18:55:36.789Z
65e643e0e999cde61f9221d0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/T10Y2F7sSgVrKGDmIzjNe.jpeg
Zongixan Ryan Li
user
Zongxian
68afb491245176306494cbb1
Wenhao Yu
68afb491245176306494cbb2
Chengsong Huang
claimed_verified
2025-08-28T08:55:08.595Z
62ea79dd01ed9b0e8f61ccd3
/avatars/70af83e0e267be39fcd5f23b85e2dafa.svg
Chengsong Huang
user
ChengsongHuang
68afb491245176306494cbb3
Rui Liu
claimed_verified
2025-09-12T16:21:58.369Z
6656bf615b203a05a1f0968c
/avatars/1ee0b0099c10dd76c8e3b7d312221b15.svg
Rui Liu
user
lr10260
68afb491245176306494cbb4
Zhenwen Liang
admin_assigned
2025-08-28T18:55:55.744Z
62ffa3f8311cad266f9af236
/avatars/203dac40bc546ee25a01d8715a4b3049.svg
Zhenwen Liang
user
invokerliang
68afb491245176306494cbb5
Fuxiao Liu
68afb491245176306494cbb6
Jingxi Che
68afb491245176306494cbb7
Dian Yu
68afb491245176306494cbb8
Jordan Boyd-Graber
68afb491245176306494cbb9
Haitao Mi
admin_assigned
2025-08-28T18:56:07.099Z
65147a1426fbd558dbd08f1b
/avatars/86574ee2d5c22e940be1c4e50be88675.svg
Haitao Mi
user
haitaominlp
68afb491245176306494cbba
Dong Yu
Self-Rewarding Vision-Language Model via Reasoning Decomposition
Vision-Language Models (VLMs) often suffer from visual hallucinations, saying
things that are not actually in the image, and language shortcuts, where they
skip the visual part and just rely on text priors. These issues arise because
most post-training methods for VLMs rely on simple verifiable answer matching
and supervise only final outputs, leaving intermediate visual reasoning without
explicit guidance. As a result, VLMs receive sparse visual signals and often
learn to prioritize language-based reasoning over visual perception. To
mitigate this, some existing methods add visual supervision using human
annotations or distilled labels from external large models. However, human
annotations are labor-intensive and costly, and because external signals cannot
adapt to the evolving policy, they cause distributional shifts that can lead to
reward hacking. In this paper, we introduce Vision-SR1, a self-rewarding method
that improves visual reasoning without relying on external visual supervisions
via reinforcement learning. Vision-SR1 decomposes VLM reasoning into two
stages: visual perception and language reasoning. The model is first prompted
to produce self-contained visual perceptions that are sufficient to answer the
question without referring back the input image. To validate this
self-containment, the same VLM model is then re-prompted to perform language
reasoning using only the generated perception as input to compute reward. This
self-reward is combined with supervision on final outputs, providing a balanced
training signal that strengthens both visual perception and language reasoning.
Our experiments demonstrate that Vision-SR1 improves visual reasoning,
mitigates visual hallucinations, and reduces reliance on language shortcuts
across diverse vision-language tasks.
Vision-SR1 uses reinforcement learning to enhance visual reasoning in vision-language models by decomposing the process into visual perception and language reasoning stages, improving accuracy and reducing hallucinations.
vision-language models visual hallucinations language shortcuts visual reasoning reinforcement learning self-rewarding method visual perception language reasoning self-containment reward hacking