R1-VL: Learning to Reason with Multimodal Large Language Models via
  Step-wise Group Relative Policy Optimization
Recent studies generally enhance MLLMs' reasoning capabilities via supervised
fine-tuning on high-quality chain-of-thought reasoning data, which often leads
models to merely imitate successful reasoning paths without understanding what
the wrong reasoning paths are. In this work, we aim to enhance the MLLMs'
reasoning ability beyond passively imitating positive reasoning paths. To this
end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new
online reinforcement learning framework that enables MLLMs to self-improve
reasoning ability via simple, effective and dense step-wise rewarding.
Specifically, StepGRPO introduces two novel rule-based reasoning rewards:
Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity
Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary
intermediate reasoning steps via a soft key-step matching technique, while
StepRAR rewards reasoning paths that follow a well-structured and logically
consistent reasoning process through a reasoning completeness and logic
evaluation strategy. With the proposed StepGRPO, we introduce R1-VL, a series
of MLLMs with outstanding capabilities in step-by-step reasoning. Extensive
experiments over 8 benchmarks demonstrate the superiority of our methods.
67d8eb0c18de6ef86c4eb457
Jingyi Zhang
claimed_verified
2025-03-24T08:54:57.692Z
67d7d1bc1414fcb61968daed
/avatars/df3d10511a73ab6370d6df90f2759489.svg
Jingyi Zhang
user
jingyiZ00
67d8eb0c18de6ef86c4eb458
Jiaxing Huang
admin_assigned
2025-03-18T09:31:01.731Z
65237910b80dc49ba03a96d9
/avatars/9d81c4c8fb2d597079e8dd9d9b79a8d8.svg
jiaxing
user
huangjiaxing
67d8eb0c18de6ef86c4eb459
Huanjin Yao
admin_assigned
2025-03-18T09:31:08.069Z
6590e03454f8826173ed5ee6
/avatars/f5e59d3e58c28a99f2ff39267ca51cdb.svg
Huanjin Yao
user
HuanjinYao
67d8eb0c18de6ef86c4eb45a
Shunyu Liu
admin_assigned
2025-03-18T09:31:15.587Z
6713afea187a20dc579e121b
https://cdn-avatars.huggingface.co/v1/production/uploads/6713afea187a20dc579e121b/ELxVQLVF9ifuT-TCWPK22.jpeg
Shunyu Liu
user
liushunyu
67d8eb0c18de6ef86c4eb45b
Xikun Zhang
admin_assigned
2025-03-18T09:31:21.950Z
6274a9620d11b4f675085fbc
https://cdn-avatars.huggingface.co/v1/production/uploads/1651812606924-noauth.jpeg
Xikun Zhang
user
Xikun
67d8eb0c18de6ef86c4eb45c
Shijian Lu
67d8eb0c18de6ef86c4eb45d
Dacheng Tao
R1-VL: Learning to Reason with Multimodal Large Language Models via
  Step-wise Group Relative Policy Optimization
Recent studies generally enhance MLLMs' reasoning capabilities via supervised
fine-tuning on high-quality chain-of-thought reasoning data, which often leads
models to merely imitate successful reasoning paths without understanding what
the wrong reasoning paths are. In this work, we aim to enhance the MLLMs'
reasoning ability beyond passively imitating positive reasoning paths. To this
end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new
online reinforcement learning framework that enables MLLMs to self-improve
reasoning ability via simple, effective and dense step-wise rewarding.
Specifically, StepGRPO introduces two novel rule-based reasoning rewards:
Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity
Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary
intermediate reasoning steps via a soft key-step matching technique, while
StepRAR rewards reasoning paths that follow a well-structured and logically
consistent reasoning process through a reasoning completeness and logic
evaluation strategy. With the proposed StepGRPO, we introduce R1-VL, a series
of MLLMs with outstanding capabilities in step-by-step reasoning. Extensive
experiments over 8 benchmarks demonstrate the superiority of our methods.
A reinforcement learning framework enhances MLLMs' reasoning by rewarding step-wise accurate and valid reasoning paths, leading to improved performance on reasoning benchmarks.
MLLMs reasoning capabilities supervised fine-tuning reinforcement learning Step-wise Group Relative Policy Optimization (StepGRPO) Step-wise Reasoning Accuracy Reward (StepRAR) Step-wise Reasoning Validity Reward (StepRVR) soft key-step matching reasoning completeness logic evaluation