SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis
Vision-language models (VLMs) trained via reinforcement learning with
verifiable reward (RLVR) have shown notable progress in scaling test-time
compute effectively. In this work, we investigate how synthesized RL data can
further improve RLVR. To this end, we propose SynthRL-a scalable and
guaranteed pipeline for automatic data scaling in reasoning-oriented RL
training. SynthRL comprises three key stages: (1) selecting seed questions with
appropriate distribution, (2) augmenting them into more challenging variants
while preserving the original answers, and (3) a guaranteed verification stage
that ensures near-perfect correctness and difficulty enhancement. Our empirical
experiments demonstrate SynthRL's scalability and effectiveness. When applied
to the MMK12 dataset, SynthRL synthesizes over 3.3K additional verifiable,
challenging questions from approximately 8K seed samples. Models trained with
our synthesized data achieve consistent gains across five out-of-domain visual
math reasoning benchmarks, with a significant improvement over baseline models
trained on seed data alone. Notably, detailed analysis reveals that the gains
are more pronounced on the most challenging evaluation samples, highlighting
SynthRL's effectiveness in eliciting deeper and more complex reasoning
patterns.
683fa36f7ed99d0040761114
Zijian Wu
claimed_verified
2025-06-04T08:57:09.765Z
626d268d5f7327906f05cad1
/avatars/18bda74612a3ee63a17f991bcc695106.svg
Zijian Wu
user
Jakumetsu
683fa36f7ed99d0040761115
Jinjie Ni
683fa36f7ed99d0040761116
Xiangyan Liu
683fa36f7ed99d0040761117
Zichen Liu
683fa36f7ed99d0040761118
Hang Yan
683fa36f7ed99d0040761119
Michael Qizhe Shieh
SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis
Vision-language models (VLMs) trained via reinforcement learning with
verifiable reward (RLVR) have shown notable progress in scaling test-time
compute effectively. In this work, we investigate how synthesized RL data can
further improve RLVR. To this end, we propose SynthRL-a scalable and
guaranteed pipeline for automatic data scaling in reasoning-oriented RL
training. SynthRL comprises three key stages: (1) selecting seed questions with
appropriate distribution, (2) augmenting them into more challenging variants
while preserving the original answers, and (3) a guaranteed verification stage
that ensures near-perfect correctness and difficulty enhancement. Our empirical
experiments demonstrate SynthRL's scalability and effectiveness. When applied
to the MMK12 dataset, SynthRL synthesizes over 3.3K additional verifiable,
challenging questions from approximately 8K seed samples. Models trained with
our synthesized data achieve consistent gains across five out-of-domain visual
math reasoning benchmarks, with a significant improvement over baseline models
trained on seed data alone. Notably, detailed analysis reveals that the gains
are more pronounced on the most challenging evaluation samples, highlighting
SynthRL's effectiveness in eliciting deeper and more complex reasoning
patterns.
SynthRL, a scalable pipeline for data synthesis in reinforcement learning with verifiable rewards, enhances visual math reasoning VLMs by generating challenging, verifiable questions.
reinforcement learning verifiable reward RLVR SynthRL seed questions data augmentation verification stage MMK12 dataset visual math reasoning benchmarks