Visual Agentic Reinforcement Fine-Tuning
A key trend in Large Reasoning Models (e.g., OpenAI's o3) is the native
agentic ability to use external tools such as web browsers for searching and
writing/executing code for image manipulation to think with images. In the
open-source research community, while significant progress has been made in
language-only agentic abilities such as function calling and tool integration,
the development of multi-modal agentic capabilities that involve truly thinking
with images, and their corresponding benchmarks, are still less explored. This
work highlights the effectiveness of Visual Agentic Reinforcement Fine-Tuning
(Visual-ARFT) for enabling flexible and adaptive reasoning abilities for Large
Vision-Language Models (LVLMs). With Visual-ARFT, open-source LVLMs gain the
ability to browse websites for real-time information updates and write code to
manipulate and analyze input images through cropping, rotation, and other image
processing techniques. We also present a Multi-modal Agentic Tool Bench (MAT)
with two settings (MAT-Search and MAT-Coding) designed to evaluate LVLMs'
agentic search and coding abilities. Our experimental results demonstrate that
Visual-ARFT outperforms its baseline by +18.6% F1 / +13.0% EM on MAT-Coding and
+10.3% F1 / +8.7% EM on MAT-Search, ultimately surpassing GPT-4o. Visual-ARFT
also achieves +29.3 F1% / +25.9% EM gains on existing multi-hop QA benchmarks
such as 2Wiki and HotpotQA, demonstrating strong generalization capabilities.
Our findings suggest that Visual-ARFT offers a promising path toward building
robust and generalizable multimodal agents.
682d7a2340a42d1538fada76
Ziyu Liu
admin_assigned
2025-05-21T10:08:39.517Z
66fe1334ff3ee1f7569fab6d
/avatars/6868b1a545028a9b8bbded52490dc093.svg
ziyuliu
user
ziyuliu
682d7a2340a42d1538fada77
Yuhang Zang
claimed_verified
2025-05-21T08:39:42.962Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
682d7a2340a42d1538fada78
Yushan Zou
admin_assigned
2025-05-21T10:08:45.972Z
6671341b4eee852a8b25888f
/avatars/635d1821bbc960b4ea845e606883eb16.svg
yushan zou
user
zyshan
682d7a2340a42d1538fada79
Zijian Liang
admin_assigned
2025-05-21T10:08:51.938Z
652768eeb723b49e8c8865da
/avatars/491e02da9ec81e439ccda8a181634bca.svg
Zijian Liang
user
steins1096
682d7a2340a42d1538fada7a
Xiaoyi Dong
admin_assigned
2025-05-21T10:09:00.912Z
67c0849ee08c178ef8d4e05c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mQ6VdnjZnRhb0H_waPclo.png
Xiaoyi Dong
user
sweetFruit
682d7a2340a42d1538fada7b
Yuhang Cao
admin_assigned
2025-05-21T10:09:22.645Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
682d7a2340a42d1538fada7c
Haodong Duan
admin_assigned
2025-05-21T10:09:31.442Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
682d7a2340a42d1538fada7d
Dahua Lin
admin_assigned
2025-05-21T10:09:37.103Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
682d7a2340a42d1538fada7e
Jiaqi Wang
claimed_verified
2025-05-21T08:39:45.391Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
Visual Agentic Reinforcement Fine-Tuning
A key trend in Large Reasoning Models (e.g., OpenAI's o3) is the native
agentic ability to use external tools such as web browsers for searching and
writing/executing code for image manipulation to think with images. In the
open-source research community, while significant progress has been made in
language-only agentic abilities such as function calling and tool integration,
the development of multi-modal agentic capabilities that involve truly thinking
with images, and their corresponding benchmarks, are still less explored. This
work highlights the effectiveness of Visual Agentic Reinforcement Fine-Tuning
(Visual-ARFT) for enabling flexible and adaptive reasoning abilities for Large
Vision-Language Models (LVLMs). With Visual-ARFT, open-source LVLMs gain the
ability to browse websites for real-time information updates and write code to
manipulate and analyze input images through cropping, rotation, and other image
processing techniques. We also present a Multi-modal Agentic Tool Bench (MAT)
with two settings (MAT-Search and MAT-Coding) designed to evaluate LVLMs'
agentic search and coding abilities. Our experimental results demonstrate that
Visual-ARFT outperforms its baseline by +18.6% F1 / +13.0% EM on MAT-Coding and
+10.3% F1 / +8.7% EM on MAT-Search, ultimately surpassing GPT-4o. Visual-ARFT
also achieves +29.3 F1% / +25.9% EM gains on existing multi-hop QA benchmarks
such as 2Wiki and HotpotQA, demonstrating strong generalization capabilities.
Our findings suggest that Visual-ARFT offers a promising path toward building
robust and generalizable multimodal agents.
Visual Agentic Reinforcement Fine-Tuning enhances Large Vision-Language Models for flexible image manipulation and web-based reasoning, outperforming existing models on multi-modal benchmarks.
Visual Agentic Reinforcement Fine-Tuning Large Vision-Language Models Multi-modal Agentic Tool Bench MAT-Search MAT-Coding multi-hop QA benchmarks