CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement
  Learning
Image captioning is a fundamental task that bridges the visual and linguistic
domains, playing a critical role in pre-training Large Vision-Language Models
(LVLMs). Current state-of-the-art captioning models are typically trained with
Supervised Fine-Tuning (SFT), a paradigm that relies on expensive, non-scalable
data annotated by humans or proprietary models. This approach often leads to
models that memorize specific ground-truth answers, limiting their generality
and ability to generate diverse, creative descriptions. To overcome the
limitation of SFT, we propose applying the Reinforcement Learning with
Verifiable Rewards (RLVR) paradigm to the open-ended task of image captioning.
A primary challenge, however, is designing an objective reward function for the
inherently subjective nature of what constitutes a "good" caption. We introduce
Captioning Reinforcement Learning (CapRL), a novel training framework that
redefines caption quality through its utility: a high-quality caption should
enable a non-visual language model to accurately answer questions about the
corresponding image. CapRL employs a decoupled two-stage pipeline where an LVLM
generates a caption, and the objective reward is derived from the accuracy of a
separate, vision-free LLM answering Multiple-Choice Questions based solely on
that caption. As the first study to apply RLVR to the subjective image
captioning task, we demonstrate that CapRL significantly enhances multiple
settings. Pretraining on the CapRL-5M caption dataset annotated by CapRL-3B
results in substantial gains across 12 benchmarks. Moreover, within the Prism
Framework for caption quality evaluation, CapRL achieves performance comparable
to Qwen2.5-VL-72B, while exceeding the baseline by an average margin of 8.4%.
Code is available here: https://github.com/InternLM/CapRL.
68d9f33e0177a6054b013afc
Long Xing
68d9f33e0177a6054b013afd
Xiaoyi Dong
68d9f33e0177a6054b013afe
Yuhang Zang
claimed_verified
2025-09-29T05:58:18.935Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
68d9f33e0177a6054b013aff
Yuhang Cao
68d9f33e0177a6054b013b00
Jianze Liang
68d9f33e0177a6054b013b01
Qidong Huang
68d9f33e0177a6054b013b02
Jiaqi Wang
claimed_verified
2025-09-29T05:58:16.337Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
68d9f33e0177a6054b013b03
Feng Wu
68d9f33e0177a6054b013b04
Dahua Lin
CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement
  Learning
Image captioning is a fundamental task that bridges the visual and linguistic
domains, playing a critical role in pre-training Large Vision-Language Models
(LVLMs). Current state-of-the-art captioning models are typically trained with
Supervised Fine-Tuning (SFT), a paradigm that relies on expensive, non-scalable
data annotated by humans or proprietary models. This approach often leads to
models that memorize specific ground-truth answers, limiting their generality
and ability to generate diverse, creative descriptions. To overcome the
limitation of SFT, we propose applying the Reinforcement Learning with
Verifiable Rewards (RLVR) paradigm to the open-ended task of image captioning.
A primary challenge, however, is designing an objective reward function for the
inherently subjective nature of what constitutes a "good" caption. We introduce
Captioning Reinforcement Learning (CapRL), a novel training framework that
redefines caption quality through its utility: a high-quality caption should
enable a non-visual language model to accurately answer questions about the
corresponding image. CapRL employs a decoupled two-stage pipeline where an LVLM
generates a caption, and the objective reward is derived from the accuracy of a
separate, vision-free LLM answering Multiple-Choice Questions based solely on
that caption. As the first study to apply RLVR to the subjective image
captioning task, we demonstrate that CapRL significantly enhances multiple
settings. Pretraining on the CapRL-5M caption dataset annotated by CapRL-3B
results in substantial gains across 12 benchmarks. Moreover, within the Prism
Framework for caption quality evaluation, CapRL achieves performance comparable
to Qwen2.5-VL-72B, while exceeding the baseline by an average margin of 8.4%.
Code is available here: https://github.com/InternLM/CapRL.
CapRL, a novel reinforcement learning framework, enhances image captioning by using a vision-free language model to evaluate caption quality through multiple-choice questions, leading to improved performance across benchmarks.
Large Vision-Language Models Supervised Fine-Tuning Reinforcement Learning with Verifiable Rewards Captioning Reinforcement Learning decoupled two-stage pipeline vision-free language model Multiple-Choice Questions Prism Framework