Diving into Self-Evolving Training for Multimodal Reasoning
Reasoning ability is essential for Large Multimodal Models (LMMs). In the
absence of multimodal chain-of-thought annotated data, self-evolving training,
where the model learns from its own outputs, has emerged as an effective and
scalable approach for enhancing reasoning abilities. Despite its growing usage,
a comprehensive understanding of self-evolving training, particularly in the
context of multimodal reasoning, remains limited. In this paper, we delve into
the intricacies of self-evolving training for multimodal reasoning, pinpointing
three key factors: Training Method, Reward Model, and Prompt Variation. We
systematically examine each factor and explore how various configurations
affect the training's effectiveness. Our analysis leads to a set of best
practices for each factor, aimed at optimizing multimodal reasoning.
Furthermore, we explore the Self-Evolution Dynamics during training and the
impact of automatic balancing mechanisms in boosting performance. After all the
investigations, we present a final recipe for self-evolving training in
multimodal reasoning, encapsulating these design choices into a framework we
call MSTaR (Multimodal Self-evolving Training for Reasoning), which is
universally effective for models with different sizes on various benchmarks,
e.g., surpassing the pre-evolved model significantly on 5 multimodal reasoning
benchmarks without using additional human annotations, as demonstrated on
MiniCPM-V-2.5 (8B), Phi-3.5-Vision (4B) and InternVL2 (2B). We believe this
study fills a significant gap in the understanding of self-evolving training
for multimodal reasoning and offers a robust framework for future research. Our
policy and reward models, as well as the collected data, is released to
facilitate further investigation in multimodal reasoning.
676a25e38ffab02f2c91a99e
Wei Liu
676a25e38ffab02f2c91a99f
Junlong Li
claimed_verified
2024-12-30T19:35:34.958Z
621e40ac944c7e36aaec2369
https://cdn-avatars.huggingface.co/v1/production/uploads/621e40ac944c7e36aaec2369/Yj-FJRWps3rvsS_B2bnKo.jpeg
Junlong Li
user
lockon
676a25e38ffab02f2c91a9a0
Xiwen Zhang
676a25e38ffab02f2c91a9a1
Fan Zhou
claimed_verified
2024-12-30T19:35:32.979Z
628f6e5ab90dde28ef57d293
https://cdn-avatars.huggingface.co/v1/production/uploads/628f6e5ab90dde28ef57d293/AxNzR2nvrND6Rf3RPkYMk.jpeg
Fan Zhou
user
koalazf99
676a25e38ffab02f2c91a9a2
Yu Cheng
claimed_verified
2025-01-23T15:05:17.014Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
676a25e38ffab02f2c91a9a3
Junxian He
Diving into Self-Evolving Training for Multimodal Reasoning
Reasoning ability is essential for Large Multimodal Models (LMMs). In the
absence of multimodal chain-of-thought annotated data, self-evolving training,
where the model learns from its own outputs, has emerged as an effective and
scalable approach for enhancing reasoning abilities. Despite its growing usage,
a comprehensive understanding of self-evolving training, particularly in the
context of multimodal reasoning, remains limited. In this paper, we delve into
the intricacies of self-evolving training for multimodal reasoning, pinpointing
three key factors: Training Method, Reward Model, and Prompt Variation. We
systematically examine each factor and explore how various configurations
affect the training's effectiveness. Our analysis leads to a set of best
practices for each factor, aimed at optimizing multimodal reasoning.
Furthermore, we explore the Self-Evolution Dynamics during training and the
impact of automatic balancing mechanisms in boosting performance. After all the
investigations, we present a final recipe for self-evolving training in
multimodal reasoning, encapsulating these design choices into a framework we
call MSTaR (Multimodal Self-evolving Training for Reasoning), which is
universally effective for models with different sizes on various benchmarks,
e.g., surpassing the pre-evolved model significantly on 5 multimodal reasoning
benchmarks without using additional human annotations, as demonstrated on
MiniCPM-V-2.5 (8B), Phi-3.5-Vision (4B) and InternVL2 (2B). We believe this
study fills a significant gap in the understanding of self-evolving training
for multimodal reasoning and offers a robust framework for future research. Our
policy and reward models, as well as the collected data, is released to
facilitate further investigation in multimodal reasoning.
Large Multimodal Models (LMMs) self-evolving training multimodal reasoning Training Method Reward Model Prompt Variation Self-Evolution Dynamics automatic balancing mechanisms MSTaR (Multimodal Self-evolving Training for Reasoning) MiniCPM-V-2.5 (8B) Phi-3.5-Vision (4B) InternVL2 (2B)