MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement
  Learning
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
a powerful paradigm for post-training large language models (LLMs), achieving
state-of-the-art performance on tasks with structured, verifiable answers.
Applying RLVR to Multimodal LLMs (MLLMs) presents significant opportunities but
is complicated by the broader, heterogeneous nature of vision-language tasks
that demand nuanced visual, logical, and spatial capabilities. As such,
training MLLMs using RLVR on multiple datasets could be beneficial but creates
challenges with conflicting objectives from interaction among diverse datasets,
highlighting the need for optimal dataset mixture strategies to improve
generalization and reasoning. We introduce a systematic post-training framework
for Multimodal LLM RLVR, featuring a rigorous data mixture problem formulation
and benchmark implementation. Specifically, (1) We developed a multimodal RLVR
framework for multi-dataset post-training by curating a dataset that contains
different verifiable vision-language problems and enabling multi-domain online
RL learning with different verifiable rewards; (2) We proposed a data mixture
strategy that learns to predict the RL fine-tuning outcome from the data
mixture distribution, and consequently optimizes the best mixture.
Comprehensive experiments showcase that multi-domain RLVR training, when
combined with mixture prediction strategies, can significantly boost MLLM
general reasoning capacities. Our best mixture improves the post-trained
model's accuracy on out-of-distribution benchmarks by an average of 5.24%
compared to the same model post-trained with uniform data mixture, and by a
total of 20.74% compared to the pre-finetuning baseline.
683dcb779187578a8ae24286
Yiqing Liang
claimed_verified
2025-06-03T08:48:57.342Z
6581d57b283fdb0c8dcb9851
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/elZhAKdRizAiAKtiAWBr3.png
Yiqing Liang
user
yiqingliang
683dcb779187578a8ae24287
Jielin Qiu
claimed_verified
2025-07-23T08:37:43.685Z
63294eeace50f125375243c1
/avatars/1a5e3b5552e9127d4e14d85880b122bd.svg
Jielin Qiu
user
jasonqiu
683dcb779187578a8ae24288
Wenhao Ding
683dcb779187578a8ae24289
Zuxin Liu
683dcb779187578a8ae2428a
James Tompkin
683dcb779187578a8ae2428b
Mengdi Xu
683dcb779187578a8ae2428c
Mengzhou Xia
683dcb779187578a8ae2428d
Zhengzhong Tu
claimed_verified
2025-06-03T08:48:54.428Z
62548d5fef3debb2ddf91217
/avatars/14975b45568f9c399c92c3986b6ce83e.svg
Zhengzhong Tu
user
vztu
683dcb779187578a8ae2428e
Laixi Shi
683dcb779187578a8ae2428f
Jiacheng Zhu
MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement
  Learning
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
a powerful paradigm for post-training large language models (LLMs), achieving
state-of-the-art performance on tasks with structured, verifiable answers.
Applying RLVR to Multimodal LLMs (MLLMs) presents significant opportunities but
is complicated by the broader, heterogeneous nature of vision-language tasks
that demand nuanced visual, logical, and spatial capabilities. As such,
training MLLMs using RLVR on multiple datasets could be beneficial but creates
challenges with conflicting objectives from interaction among diverse datasets,
highlighting the need for optimal dataset mixture strategies to improve
generalization and reasoning. We introduce a systematic post-training framework
for Multimodal LLM RLVR, featuring a rigorous data mixture problem formulation
and benchmark implementation. Specifically, (1) We developed a multimodal RLVR
framework for multi-dataset post-training by curating a dataset that contains
different verifiable vision-language problems and enabling multi-domain online
RL learning with different verifiable rewards; (2) We proposed a data mixture
strategy that learns to predict the RL fine-tuning outcome from the data
mixture distribution, and consequently optimizes the best mixture.
Comprehensive experiments showcase that multi-domain RLVR training, when
combined with mixture prediction strategies, can significantly boost MLLM
general reasoning capacities. Our best mixture improves the post-trained
model's accuracy on out-of-distribution benchmarks by an average of 5.24%
compared to the same model post-trained with uniform data mixture, and by a
total of 20.74% compared to the pre-finetuning baseline.
A framework for post-training multimodal large language models using reinforcement learning with verifiable rewards introduces a data mixture strategy to enhance general reasoning abilities and benchmark performance.
reinforcement learning with verifiable rewards multimodal large language models multi-dataset post-training data mixture problem online RL learning RL fine-tuning outcome mixture prediction strategies