Visual-RFT: Visual Reinforcement Fine-Tuning
Reinforcement Fine-Tuning (RFT) in Large Reasoning Models like OpenAI o1
learns from feedback on its answers, which is especially useful in applications
when fine-tuning data is scarce. Recent open-source work like DeepSeek-R1
demonstrates that reinforcement learning with verifiable reward is one key
direction in reproducing o1. While the R1-style model has demonstrated success
in language models, its application in multi-modal domains remains
under-explored. This work introduces Visual Reinforcement Fine-Tuning
(Visual-RFT), which further extends the application areas of RFT on visual
tasks. Specifically, Visual-RFT first uses Large Vision-Language Models (LVLMs)
to generate multiple responses containing reasoning tokens and final answers
for each input, and then uses our proposed visual perception verifiable reward
functions to update the model via the policy optimization algorithm such as
Group Relative Policy Optimization (GRPO). We design different verifiable
reward functions for different perception tasks, such as the Intersection over
Union (IoU) reward for object detection. Experimental results on fine-grained
image classification, few-shot object detection, reasoning grounding, as well
as open-vocabulary object detection benchmarks show the competitive performance
and advanced generalization ability of Visual-RFT compared with Supervised
Fine-tuning (SFT). For example, Visual-RFT improves accuracy by 24.3% over
the baseline in one-shot fine-grained image classification with around 100
samples. In few-shot object detection, Visual-RFT also exceeds the baseline by
21.9 on COCO's two-shot setting and 15.4 on LVIS. Our Visual-RFT represents
a paradigm shift in fine-tuning LVLMs, offering a data-efficient, reward-driven
approach that enhances reasoning and adaptability for domain-specific tasks.
67c6816614a1bf9855188b8b
Ziyu Liu
admin_assigned
2025-03-04T09:12:57.481Z
66fe1334ff3ee1f7569fab6d
/avatars/6868b1a545028a9b8bbded52490dc093.svg
ziyuliu
user
ziyuliu
67c6816614a1bf9855188b8c
Zeyi Sun
claimed_verified
2025-03-04T08:35:03.275Z
63fda3fced9eead590ff6918
https://cdn-avatars.huggingface.co/v1/production/uploads/1677566802735-noauth.jpeg
Zeyi Sun
user
Zery
67c6816614a1bf9855188b8d
Yuhang Zang
admin_assigned
2025-03-04T09:12:32.723Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
67c6816614a1bf9855188b8e
Xiaoyi Dong
admin_assigned
2025-03-04T09:12:25.627Z
67c0849ee08c178ef8d4e05c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mQ6VdnjZnRhb0H_waPclo.png
Xiaoyi Dong
user
sweetFruit
67c6816614a1bf9855188b8f
Yuhang Cao
admin_assigned
2025-03-04T09:12:19.177Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
67c6816614a1bf9855188b90
Haodong Duan
admin_assigned
2025-03-04T09:12:05.281Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
67c6816614a1bf9855188b91
Dahua Lin
admin_assigned
2025-03-04T09:11:57.087Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
67c6816614a1bf9855188b92
Jiaqi Wang
admin_assigned
2025-03-04T09:11:48.889Z
64638c4d51fa6e63060521b5
/avatars/c863ace5b1dc788a341bcf4ddbdfaec1.svg
JIaqi
user
Jiaqiwang
Visual-RFT: Visual Reinforcement Fine-Tuning
Reinforcement Fine-Tuning (RFT) in Large Reasoning Models like OpenAI o1
learns from feedback on its answers, which is especially useful in applications
when fine-tuning data is scarce. Recent open-source work like DeepSeek-R1
demonstrates that reinforcement learning with verifiable reward is one key
direction in reproducing o1. While the R1-style model has demonstrated success
in language models, its application in multi-modal domains remains
under-explored. This work introduces Visual Reinforcement Fine-Tuning
(Visual-RFT), which further extends the application areas of RFT on visual
tasks. Specifically, Visual-RFT first uses Large Vision-Language Models (LVLMs)
to generate multiple responses containing reasoning tokens and final answers
for each input, and then uses our proposed visual perception verifiable reward
functions to update the model via the policy optimization algorithm such as
Group Relative Policy Optimization (GRPO). We design different verifiable
reward functions for different perception tasks, such as the Intersection over
Union (IoU) reward for object detection. Experimental results on fine-grained
image classification, few-shot object detection, reasoning grounding, as well
as open-vocabulary object detection benchmarks show the competitive performance
and advanced generalization ability of Visual-RFT compared with Supervised
Fine-tuning (SFT). For example, Visual-RFT improves accuracy by 24.3% over
the baseline in one-shot fine-grained image classification with around 100
samples. In few-shot object detection, Visual-RFT also exceeds the baseline by
21.9 on COCO's two-shot setting and 15.4 on LVIS. Our Visual-RFT represents
a paradigm shift in fine-tuning LVLMs, offering a data-efficient, reward-driven
approach that enhances reasoning and adaptability for domain-specific tasks.
Visual Reinforcement Fine-Tuning (Visual-RFT) enhances Large Vision-Language Models (LVLMs) through reinforcement learning with visual perception verifiable rewards, achieving competitive performance in various visual and reasoning tasks with limited data.
Reinforcement Fine-Tuning (RFT) Visual Reinforcement Fine-Tuning (Visual-RFT) Large Vision-Language Models (LVLMs) reasoning tokens policy optimization algorithm Group Relative Policy Optimization (GRPO) Intersection over Union (IoU) fine-grained image classification few-shot object detection reasoning grounding open-vocabulary object detection Supervised Fine-tuning (SFT)