Training Vision-Language Process Reward Models for Test-Time Scaling in
  Multimodal Reasoning: Key Insights and Lessons Learned
Process Reward Models (PRMs) provide step-level supervision that improves the
reliability of reasoning in large language models. While PRMs have been
extensively studied in text-based domains, their extension to Vision Language
Models (VLMs) remains limited. Existing Vision-Language PRMs (VL-PRMs) rely on
Monte Carlo Tree Search (MCTS) for data construction, which can often produce
noisy supervision signals and limit generalization across tasks. In this work,
we aim to elucidate the design space of VL-PRMs by exploring diverse strategies
for dataset construction, training, and test-time scaling. First, we introduce
a hybrid data synthesis framework that combines MCTS with judgments from a
strong VLM, producing more accurate step-level labels. Second, we propose
perception-focused supervision, enabling our PRM to explicitly detect errors at
the visual grounding stage of reasoning. Third, we systematically evaluate
multiple test-time scaling strategies, showing that our PRMs can reliably guide
VLMs toward more accurate solutions. Our experiments covering five diverse
multimodal benchmarks (MMMU, PuzzleVQA, AlgoPuzzleVQA, MathVista, and
MathVision) reveal several key insights: (i) VL-PRMs when used as Outcome
Reward Models (ORMs) during test-time scaling (TTS) can outperform VL-PRM
guided process step selection, (ii) smaller VL-PRMs can match or even surpass
larger ones in detecting process errors, (iii) VL-PRMs uncover latent reasoning
abilities in stronger VLM backbones, (iv) perception-level supervision leads to
significant gains in test-time scaling, and (v) TTS performance of different
policies improve on advanced math reasoning datasets despite not training
VL-PRMs on such datasets. We hope our work will motivate further research and
support the advancement of VLMs.
68db7e0ad2bf1f4b15ec77ae
Brandon Ong
68db7e0ad2bf1f4b15ec77af
Tej Deep Pala
68db7e0ad2bf1f4b15ec77b0
Vernon Toh
68db7e0ad2bf1f4b15ec77b1
William Chandra Tjhi
68db7e0ad2bf1f4b15ec77b2
Soujanya Poria
Training Vision-Language Process Reward Models for Test-Time Scaling in
  Multimodal Reasoning: Key Insights and Lessons Learned
Process Reward Models (PRMs) provide step-level supervision that improves the
reliability of reasoning in large language models. While PRMs have been
extensively studied in text-based domains, their extension to Vision Language
Models (VLMs) remains limited. Existing Vision-Language PRMs (VL-PRMs) rely on
Monte Carlo Tree Search (MCTS) for data construction, which can often produce
noisy supervision signals and limit generalization across tasks. In this work,
we aim to elucidate the design space of VL-PRMs by exploring diverse strategies
for dataset construction, training, and test-time scaling. First, we introduce
a hybrid data synthesis framework that combines MCTS with judgments from a
strong VLM, producing more accurate step-level labels. Second, we propose
perception-focused supervision, enabling our PRM to explicitly detect errors at
the visual grounding stage of reasoning. Third, we systematically evaluate
multiple test-time scaling strategies, showing that our PRMs can reliably guide
VLMs toward more accurate solutions. Our experiments covering five diverse
multimodal benchmarks (MMMU, PuzzleVQA, AlgoPuzzleVQA, MathVista, and
MathVision) reveal several key insights: (i) VL-PRMs when used as Outcome
Reward Models (ORMs) during test-time scaling (TTS) can outperform VL-PRM
guided process step selection, (ii) smaller VL-PRMs can match or even surpass
larger ones in detecting process errors, (iii) VL-PRMs uncover latent reasoning
abilities in stronger VLM backbones, (iv) perception-level supervision leads to
significant gains in test-time scaling, and (v) TTS performance of different
policies improve on advanced math reasoning datasets despite not training
VL-PRMs on such datasets. We hope our work will motivate further research and
support the advancement of VLMs.
Hybrid data synthesis and perception-focused supervision improve the reliability of Vision-Language Process Reward Models (VL-PRMs) in guiding VLMs across diverse multimodal benchmarks.
Process Reward Models Vision Language Models Monte Carlo Tree Search hybrid data synthesis perception-focused supervision Outcome Reward Models test-time scaling multimodal benchmarks visual grounding process errors latent reasoning abilities