Diving into Self-Evolving Training for Multimodal Reasoning
Reasoning ability is essential for Large Multimodal Models (LMMs). In the
absence of multimodal chain-of-thought annotated data, self-evolving training,
where the model learns from its own outputs, has emerged as an effective and
scalable approach for enhancing reasoning abilities. Despite its growing usage,
a comprehensive understanding of self-evolving training, particularly in the
context of multimodal reasoning, remains limited. In this paper, we delve into
the intricacies of self-evolving training for multimodal reasoning, pinpointing
three key factors: Training Method, Reward Model, and Prompt Variation. We
systematically examine each factor and explore how various configurations
affect the training's effectiveness. Our analysis leads to a set of best
practices for each factor, aimed at optimizing multimodal reasoning.
Furthermore, we explore the Self-Evolution Dynamics during training and the
impact of automatic balancing mechanisms in boosting performance. After all the
investigations, we present a final recipe for self-evolving training in
multimodal reasoning, encapsulating these design choices into a framework we
call MSTaR (Multimodal Self-evolving Training for Reasoning), which is
universally effective for models with different sizes on various benchmarks,
e.g., surpassing the pre-evolved model significantly on 5 multimodal reasoning
benchmarks without using additional human annotations, as demonstrated on
MiniCPM-V-2.5 (8B), Phi-3.5-Vision (4B) and InternVL2 (2B). We believe this
study fills a significant gap in the understanding of self-evolving training
for multimodal reasoning and offers a robust framework for future research. Our
policy and reward models, as well as the collected data, is released to
facilitate further investigation in multimodal reasoning.
676a25e38ffab02f2c91a99e
Wei Liu
claimed_verified
2025-05-22T07:19:21.443Z
6458af46f4d212d780bd7c68
/avatars/832fd34bcc041b0b7b551873a459fc3c.svg
Wei Liu
user
PeterV09
676a25e38ffab02f2c91a99f
Junlong Li
claimed_verified
2024-12-30T19:35:34.958Z
621e40ac944c7e36aaec2369
https://cdn-avatars.huggingface.co/v1/production/uploads/621e40ac944c7e36aaec2369/Yj-FJRWps3rvsS_B2bnKo.jpeg
Junlong Li
user
lockon
676a25e38ffab02f2c91a9a0
Xiwen Zhang
676a25e38ffab02f2c91a9a1
Fan Zhou
claimed_verified
2024-12-30T19:35:32.979Z
628f6e5ab90dde28ef57d293
https://cdn-avatars.huggingface.co/v1/production/uploads/628f6e5ab90dde28ef57d293/AxNzR2nvrND6Rf3RPkYMk.jpeg
Fan Zhou
user
koalazf99
676a25e38ffab02f2c91a9a2
Yu Cheng
claimed_verified
2025-01-23T15:05:17.014Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
676a25e38ffab02f2c91a9a3
Junxian He
Diving into Self-Evolving Training for Multimodal Reasoning
Reasoning ability is essential for Large Multimodal Models (LMMs). In the
absence of multimodal chain-of-thought annotated data, self-evolving training,
where the model learns from its own outputs, has emerged as an effective and
scalable approach for enhancing reasoning abilities. Despite its growing usage,
a comprehensive understanding of self-evolving training, particularly in the
context of multimodal reasoning, remains limited. In this paper, we delve into
the intricacies of self-evolving training for multimodal reasoning, pinpointing
three key factors: Training Method, Reward Model, and Prompt Variation. We
systematically examine each factor and explore how various configurations
affect the training's effectiveness. Our analysis leads to a set of best
practices for each factor, aimed at optimizing multimodal reasoning.
Furthermore, we explore the Self-Evolution Dynamics during training and the
impact of automatic balancing mechanisms in boosting performance. After all the
investigations, we present a final recipe for self-evolving training in
multimodal reasoning, encapsulating these design choices into a framework we
call MSTaR (Multimodal Self-evolving Training for Reasoning), which is
universally effective for models with different sizes on various benchmarks,
e.g., surpassing the pre-evolved model significantly on 5 multimodal reasoning
benchmarks without using additional human annotations, as demonstrated on
MiniCPM-V-2.5 (8B), Phi-3.5-Vision (4B) and InternVL2 (2B). We believe this
study fills a significant gap in the understanding of self-evolving training
for multimodal reasoning and offers a robust framework for future research. Our
policy and reward models, as well as the collected data, is released to
facilitate further investigation in multimodal reasoning.
A study on self-evolving training for multimodal reasoning identifies key factors and configurations to optimize reasoning abilities without additional human annotations, introducing MSTaR as an effective training framework.
Large Multimodal Models self-evolving training multimodal reasoning Training Method Reward Model Prompt Variation Self-Evolution Dynamics automatic balancing mechanisms MSTaR MiniCPM-V-2.5 Phi-3.5-Vision InternVL2